T1/3 - Automated Mail Server Testing by Martin Boßlet (@_emboss_)

good morning my name is Martin and I want to talk to you about email server testing and let me start with a simple example so what is the first thing basically everyone does when they put a web server online these days well many times what you do after it is deployed and online you check the score on SSL labs which is a fine thing because we can save the claim that SSL Epps has made the internet a safer place because it's bad for your reputation if you have a bad score there so people will tell you about it and you will fix it until you have an A or your otherwise satisfied with the result and so this is

good in the end because it keeps the web safer it keeps bad configurations out of the picture the drawback is mess-ups only works with HTTPS websites now the question is what would you do when you put a different server online that uses TLS somehow what you do if you put a web email server online for example SMTP IMAP pop whatever and the answer is there's no good right now that would do a comprehensive report for you like SSS does and that's a pity because it's not so different as you can imagine TLS stays TLS wherever it is used and so the question is why don't we have a tool like that wouldn't it be great to have a

tool that not only allows you to test web servers but also allows you to test smtp I might pop or other protocol to claim that uses TLS and yeah it would be great and that's why I'm here today because the answer is we are working on something like this we want to give you a tool that allows you to do all of these things we haven't came up with a good yet so that's all work in progress or sometimes we just call it the project and that is to be discussed that's a bit lengthy here but I guess more important for you is what does this thing do and it's pretty simple well it's very

similar to what you do with SSL apps you have a HTML interface there's a form where you put your email host in there you hit OK and what you get back is a report it's not just about TLS but it's also about DNS records because it's also important for email and yeah that's pretty much it and how can you use that while we see too typical use cases well first of all there are private persons so for them it might be interesting to see if their email provider well if they're let's say you have an account with Gmail or something then you want to see if they do things the right way but I think a more important use case would

be similar to how SSL Apps is also used these days you use this as a tool to validate your setup let's say you put an email server online then you can immediately check if things are going ok for you and we believe this is a good idea because email security is important because it's still is one of the major places that an attacker looks for these days but note that we just care about email servers we do not have a focus on things like PGP or other client-side technologies which of course we welcome and endorse but that's not the focus of our tool let's talk about the history of the project bit how this all came to be and

like any good project first of all there was a proof of concept and this was sponsored by the Ministry of Economy and security made in Luxembourg thanks to them and the goal was as always for proof of concept get it running as quickly as possible and that we did what were the features of the proof-of-concept we tried to show a lot for what was possible in a short time while we did the DNS records so it fetched DNS records like TLS a CA d mark and SPF and we also did a basic TLS analysis and because that's the meat of the checks and validations to implement we took an existing tool called SSL decoder in

order to bootstrap the process so this gave us a lot of the tests that we could also display to users and it already offered a JSON interface so it was easy for us to use this in our own DNS record checks and this was the architecture of the proof-of-concept still pretty simple the main important thing is there in this gap here where we forward the TLS analysis to SSL decoder we do DNS record analysis in our own application and then we combine the two and show them to the user so the user fills in a form hit center rails app does DNS record lookup forwards TLS analysis deal ssl dakota combines initials so pretty simple and

here are some images of what it was able to do so here you can see the DNS record displayed for Gmail here's another example of the class features so that's very similar to what you know from something like SSL apps it shows the protocols cipher Suites certain features if it's vulnerable against things are not and finally the service certificate so any service sends a certificate chain and then the chain is listed each certificate is analyzed with its details but as you can already see all the values are in black so we didn't do any assessment in the proof of concept we just listed the details but didn't do any evaluation we ran into interesting problem since we wanted to

focus on email security and we wanted to cover all the DNS records that are important for email security of course we also wanted to include dekum but the problem is you cannot analyze dekum without having an actual email and let's look at a reason for this if you look up the DNS record in DNS for dekum then the string you need to look up is composed of two fixed parts in the middle you have to fix string underscore domain key and at the end you have the domain that you're examining but the problem is this part here in the front there's a selector and if you look at the specification this selector can be anything basically and it's even advice

to rotate this and to keep it unguessable so that means we in our tool cannot know this value are in advance and so that means basically we cannot look up the record so how can we solve this problem well we need an actual email sent from that domain that contains a decom header and once we have that email we can look in the dica matter and see what the selector value actually is once we know the selector value we can look up the dns record and this drawback let into phase 2 there was another big feature which I'm going to talk afterwards but this was really one of the issues we wanted to solve in

a second phase we wanted to solve this decommission and how do we get an email first we thought about solutions for this and I mean if we need an email from somewhere then let's just send an email and we pretty quickly came up with the idea of doing old-school web service style sent an email to a service and get an email back and at first we thought this was an excuse but if you think about it we're talking about email security here so in the end I think it really feels natural so if you want to talk about email security why not send an actual email and this we thought it was a good idea and phase 2 was again

sponsored by the same sponsors so we could implement this and it led us to a new problem if we do send emails do we really want to go through the whole effort of taking this HTML report that we had and put this in an email well not really it would happen I mean it's doable but it would have taken a lot of time so what we wanted to have is not sending a report but we would just send back a reply telling people ok your report is ready here at this URL go there and view it and then we would just show them the same report that they would see if they go to the

HTML web forum the problem with this is if we want to do it that way if you think it through it's a synchronous so we have a request we produced a result but the request comes later so we need to temporarily at least store the risk on somewhere we can't put it in a database because that would mean we could collect data and sell this data or whatever at least we don't want to collect data about our customers we just want to give you a good tool and so that's not an option to put it in a database but I mentioned we just need it temporarily and so the solution is we cache the result for a given time right

now it's an hour and after an hour the result is gone again and this has also other benefits we can cache a lot of things if you think about it a lot of times you will download the same C or else the same intermediate certificates and all of that stuff this could also be cached in the future and for phase two we actually had to expand our architecture a bit mostly because of the email feature so if you use the web form everything works as before as you can see we integrated the SSL decoder parts into the application as a cell decoders are no longer in there but we have three new components so let's walk it through

every user decides to send an email it's received by node.js front end which is using the note mailer API which is a perfect tool for our use case because it can act as an SMTP server parse the email and then just create some JSON to forward it using a REST API to the application itself so to the user it will look just like any SMTP server parses the email sends us the important features including dekum whatever we need and now this trick is a background job in the application background Java produces the result serializes it into JSON stores it in the cache once done an email is generated containing a link with an unnecessary can inner set peer

results it's just for you and we sent this email using postfix soco's fixes just for outgoing SMTP here now user receives the email with the link we'll decide to click on the link open it in a browser or something and this will trigger a new request to the app and the app will fetch the cached result displayed as usually now besides these synchronous features our other big feature that we wanted to include in the second phase is actually doing some parts of evaluation in the proof-of-concept we just have black text no judgement but here in Phase two we also wanted to give you some indication whether something is good or bad or if it's fine just as it is and we added a

new face so in the proof-of-concept we just had these three things so it was a linear process incoming requests from the request we collect all the data and then we present that data to the user now in Phase two we added a second step after the data is collected we create an internal data structure and then this internal data structure is evaluated so it contains all the all the data that is needed to evaluate it and we produce another data structure containing the evaluation and a combination of this is then presented to the user and this helps us to show our different reports I'm gonna just show you some quick examples of what it would look like

these are just saved pages that I did on some test runs so first of all this is a HTTP test so not actually email which is for our own website to show you that we're doing things mostly fine so many green values no red values that's good and now I want to show you two quick examples for emails for example yes yahoo mail that's interesting Yahoo's website is pretty much top-notch if you look at it on as a salads for example or with our tool but if you look at the email you will notice that here and I'm gonna zoom in so you can see that better if you look at their email setup they still I mean it's

debatable if you still want to support our c4 with sha-1 because some old-school clients might need it but our c4 with md5 should definitely be banned and this is this is the point that we want to make with this tool their web server is top-notch because of tools like SSL apps because there's public public analysis possible and so this would help also to expose glitches like this and the question is now that phase 2 is implemented where do we want to go in the future and we have a lot of ideas first of all telling you whether it's something is green yellow or red is good it helps you to fix things but it is not

really well to compare like you can't tell is this site any better than this or are these flaws really that important to do such are to present such a result we need some form of scoring like SSL apps has it's grades we need to know score too and so what we want to act in phase 3 let's call it is another phase that takes the evaluation and then puts Waits and scores to that and then outputs these grades or scores to the user so that's one thing but if we start doing that at least if we want to be want to be understandable to people and then we need to explain ourselves we need to give a reasoning for why we give

certain evaluations because it's useless if you just tell them this is bad you have to have a reason for why it's bad but if we need to do this then it's automatically also means we need to provide documentation and if we provide documentation why not make it useful for everybody we can you also use this for educational purposes like having a wiki or something that tells you about what is important in TLS and what are the mistakes or things to watch out for if you want to use this in a company context then I could imagine you could use this in automated processes for example monitoring checking every week if my email server is still ok

things like that and then it's awkward to just have an HTML form as your primary interface so it would be much better for a company to have a real Web API so you could directly access the results you have a JSON response which is documented and you can evaluate that response and act upon it so that's certainly something for the future - another big thing that I know from personal experience a lot of companies are a little bit uneasy about using something like us to sell apps because they don't know what they do on their service with the results and that's one one thing against using it another one maybe even more important you can only

test once the service online so if it's bad to set up if it's bad it's already been published online and that's awkward you you would much rather want to do something where you test this stuff internally in your own network and only if it's good then you're gonna publish it and to do such a thing you can't use a web application because the web application itself needs to have public access to that server so we're thinking about a command-line interface because you could use this anywhere in your private networks get a report in your private network and then once you're satisfied with the results only then you could publish the server and put it online

and yeah TLS is at the core of every validation in all of the email protocols also for web servers but there are a lot more TLS based protocols FTP XMPP whatever you can imagine that uses TLS it could all be a part of this and of course there are specific things that are only important in specific context but a lot of things and validations and checks are common to all different protocols so we could use this as a basis if you look at the tool I will publish the URL right after the talk here then you will see it's compared to SSL apps there are a lot of things missing and of course we want to catch

up and since we are also concentrating on DNS records we want to also want to catch up with our good tools in that sector maybe you've heard of MX toolbox and so we want to be on par with all of them the tool itself is open source and to make it useful our ultimate goal is to create a community around it because only then we can evolve it to the point where we see it in the future but as you know billing community is hard it needs sponsors partnerships we don't want to be the only ones who decide about things because we have a domain knowledge but we can't cover all of the aspects of all possible protocol so it

would be great if we have outside experts who could tell us about other protocols and who can give us their insights and so all together we can come up with a better tool with more tests with relevant tests and ultimately the perfect tool and that's really the vision that we have for it so we can build a perfect SSL apps for email if everybody comes together if we share our knowledge and if we can set up the tests that everybody is ok with and the cool thing is since this is all about TLS mostly we can not only build a perfect tool for email but we can really built a perfect tool for TLS as a whole in

general and that's the ultimate vision yeah so thank you this is the email if you want to test it it hasn't been battle tested so please forgive us if anything goes wrong and we would totally appreciate it if you can send us any are things that you noticed or improvements your thoughts it's currently secured against crawlers and robots so we have basic authentication on there we have a totally secure combination so username is email password is secure and we would really appreciate your feedback if you want to please send us an email to contact that email mate and Luxemburg thank you questions remarks yes so that was one as well okay so we go in order I

am I think this is great I would really love to have this I can I am going to spread this in some communities that I know that care about this friend of mine actually made a service that tests Dane records it's on have Dane net and he actually made he was struggling with the idea that you did you mentioned in sending email because he you need to receive an email he generates email addresses and that you have to send him so each sends imaged makes three email addresses and you send an email to all three and the website keeps polling and once they receive all of the email addresses or not he can generate the

results whether you check Dane or not so that's a that's another way you could do this it's open source as well so if you need any sure yes of course yeah then it's thanks for this I thank you for your talk first I can share this picture on Twitter so everyone can see it or not no I mean not that slide okay and second question because I did share as Something About You I get question on Twitter did he spoke about de M L say D mark okay well we're responding to questions on Twitter any questions here yeah did you have any plans to also add something like history or be able to compare reports I mean

it's a bit against what you said about privacy as don't want to keep track of information but it would make sense to see when something happens when you improve for

something changes right that's the intent of it that that's actually a pretty good idea thank you for that so yeah but it's like you said it's kind of works against the idea of not keeping data maybe I could imagine something like having an account so you can create an account and that means your you want to keep data and then we could keep the data for people who who like to have their data being kept in than accessing history or in the command line version yeah i mean if you store it on your server because it's publicly available you can either encrypt it with the user's password or accounts password with the sort of course or you can hash

it so even if there is a data Rick only hashes are out and not the real data you have to think about can I in advance connect create a similar response and then just check by comparing two hash values like for example if you if you keep hashes of a social security number then there are not enough values you can just iterate through the values create the hashes and then look if one of those is in there then you know that it's the original data there should be enough enough randomness in there yeah but also the encrypting idea that's also good if we have an account we can encrypt the data by using a password derived key

okay yeah I try to keep in mind who's who's next so first thing thank you because that's a very good interesting tool so thank you for this and the other thing is until this tool the way I used to check the security of an email server was to rely on check TLS calm which is unfortunately not open-source but which which is checking the SSL setting and the certificate chain of the makes our so it's but it's not answering the damask stuff and all the email security setting so it's partially answering to the question and there's a very a tricky way to do it is just to send an email from your mail server to a

gmail account and see if you have the little icon and then you can also be sure that you have TLS enabled or not on your on your mail server and then I getting the ID for the comparison what you can also do is to to save the result as a JSON file and then just to provide an interface to compare to GS and file from a previous test and a new test and then you can easily compare and you don't have to save the result on your on your back-end but that you just need to save the result we talked about what if users because we can imagine use cases of this where somebody creates a report

and then prints this to upper management and they want to have a PDF for example or they don't want to look at websites so it would be interesting to have also an export feature for PDF or you said son and then you can create own tool based on that that would be a good idea I think so too Thanks thank you very much fantastic work what I was thinking now during your presentation is that email servers usually have many supportive services for example anti-spam engine we know some of them are bad some are good some are core open source of our closed source some might give information about the signature that they're using the same applies for

anti viruses and you already have that information when you receive an email so you could make a careful for example assessment of what kind of spam engine they're using if they're not using that's an issue I believe if anybody has set up here Nemo's server knows very well that not using a spam engine is like going out totally naked your your food for the spammers and antivirus there are many UTM solutions that actually go in front of the email server and put a bunch of signature so take advantage of that by all means but because that's email security as well yeah actually that that was also my thinking that you know with with SSL I mean whoa your SSL sucks okay

well that's not really the end of the world but with email we that's that's a very good vector in order I mean sometimes I don't even bother doing anything just fish in the engagement if we do red teaming so you probably want to respond there and then you're next I love this feedback because that is my background is more crypto and SSL focused so those are the DEA's ideas that we definitely need to make this really all around perfect solution because I would tend to always focus on the cypher stuff and so on and those are the ideas that I like and that's why it's so important for us to have outsiders who bring us these ideas and

who want to get engaged so thank you for the ideas and also does the idea to have this discussion part I think Klaus you were next I may have missed it but did you also check the server permission settings I guess if the mail server was open for an open mail relay

yeah and I was I was thinking like basic stuff like I have to admit I had to run around but something like verify so when you can you know just enumerate things I don't know if you do such a check when you just want to collect but that is easy to check because I mean if you allow those commands them yeah that's bad yeah I know just I have just a few comments about the presentation as marking told you that the project it is supported by the Ministry of Economy and security made in Lu I will present them home for the moment and I see I'm not a tech guy so I didn't understand all the

questions but I see that there's a really an interest from you and yes I just like to propose that if you want to support a project we could talk together and see the better way to involve you to make it to improve it and perhaps to help to fund it and yes to Martin show that the the product was working at the moment and pretty well and but there's of course possibility to add some features of we had some proposed today and so I'd like to invite you or to talk together or to send an email to the address of the yeah yeah yeah yeah indeed okay if you're okay yeah sorry mikemonkey you go hello I've also

been using the the check TLS website actually the information there is much less granular that what you showed us so it's it's very nice on the other end most of the well as Lisa as far as I know and I checked last time most of the SMTP communication on the Internet today is using opportunity of mystic encryption so basically when you have two smtp servers that connect to each other they are doing a start TLS and up to now people find it more well better to actually receive email even in if the TLS connection is broken so it is nice to have the information that indeed your TLS is working fine and you have nice eyes and whatever but the point is

if you do a TLS strip attack and you just remove the start TLS sequence anyway it will bypass the encryption so I think in the context of SMTP at least today knowing which ciphers you are using is not as important as on the website and what would be really nice is to know which servers SMTP servers are actually checking that what they were they was that they enforce the encryption and if that would be doable that would be nice that would also maybe push people to decide that maybe security is better than just usability aspects so first of all if you send us an email we check if that last hop comes in encrypted and if

it does not then we flagged us as a as a bad thing which is something that Gmail also does but this doesn't cover the entire chain and that's

it's also to him you can just capture this one point and that's true but and we're also thinking about how can we because that's another important issue even if the last hop is encrypted that doesn't mean that the entire Jane was encrypted and if only one link is not encrypted then the whole thing is moot and yeah we're trying to imagine ways how we could ensure because that would be the ultimate nice thing to have yeah I'm it's this is hard to do to give a definite answer to to this but you can at least we can try to detect some things that might point there another thing is so when they send email if we

we can try to connect it to their SMTP server try to not use start TLS and to do it and then that's a check that I've actually been working on but it's not in integrated yet to connect to them try to send email unencrypted and see if they accept it because there are some configurations that would not accept behavior like this and and yeah it's that's that's the important thing to take away here I I believe as we saw in the discussion everybody has has good ideas that I never came up with and so it would be so awesome for us all to work together and at least share our ideas

all the all the people that have a sin teepee set up today they still want to be able to receive emails from sources that do not support the LS encryption so you need to make sure that at the point where you decide to do that of course you can still receive significant amount of emails because if you do that today I think you will block everything and so yeah the initiatives like this are nice because of course it pushed people to have good configuration in the first place and then in the future you can you can in insight indeed decide to turn it on turn it on and that it's no longer opportunistic but it's enforced

encryption actually has accomplished more or less for HTTPS because of the it's I mean it's a lot of people hate it because it's used for finger-pointing but also true but I think seen as a whole it helps to raise the bar and like you said once the bar is raised high enough we can start to disable stuff that's no longer beautiful so yeah it's definitely the goal now maybe for one more question if you have it if not deny you would like to thank you for the presentation Martine if you around applause please very very cool project