Angie Shue - Defense in Depth for Email Security

all right so our next talk uh angie shu is presenting defense defense i misspelled it on my sheet defense in depth for email security good morning [Music] i just had a co-worker give me a good luck fish as i'm talking about fishing today so i might hand that out later i'm not sure but i really i like my new little fish here i'd like to set the stage for this presentation with a quote from cisco customers of all sizes face the same daunting challenge email is simultaneously the most important business communication tool and the leading attack vector for security breaches so that kind of defines my job i'm supposed to let through 100 of the good email and

stop 100 of the bad email that's definitely a huge challenge and i haven't perfected it yet but i will tell you some of the things that were today that we're doing to try to meet that goal a little bit about me i am a cyber security engineer at team health i've been working there for about a year and a half and prior to becoming a cyber security engineer i was a sys admin in a lot of different companies doing a lot of different things everything from custom access databases accounting software support exchange admin active directory router switches firewall rules so all kinds of stuff a broad range of things so this is the first job that i've had

where i've actually gotten to focus on something so specifically um a relatively small area of email security so even though it's a very big responsibility i'm really enjoying being able to hone in and become the sme on this topic i do get to work from home which i love that's a picture of my family there those are my boys remington and magnum they are great danes and our little mop beretta and my very patient husband adam i mentioned i work for team health so in case you aren't familiar with that they they do operate nationwide and they are in very much in the healthcare industry they do a lot of different things they are expanding

their telehealth service line right now they one of their primary things is helping to supply doctors for hospitals especially ers and they also do nursing lots of lots of medical billing lots of different things in health care

and being in the health care industry you may or may not be aware that health care is a target a specific target for cyber criminals a big reason for that medical record sells for a lot more on the dark web than a social security number or credit card number all of the information in a medical record is very valuable and it's it they go for a lot more and so we've got a lot of stuff that we need to protect at team health and we take it very seriously a lot of personal identifiable information phi subject to lots hippo regulations that we need to keep an eye on all that stuff as well so as we are in health care and

and we have valuable information we are a target and we see that in the attacks that come against our network and of course via email so i'm going to show you some examples of emails that we've gotten in this first one you can see how they changed the from to look like it was coming from a legitimate team health source we see these fake delivery failure notices and lately i've been seeing a lot of fake voicemail notifications we've seen fake spam quarantine notifications and so there's there's a lot of emails in that category and the other one you can tell they even grabbed our logo they're trying so hard to make it look legit they even capitalize team health

correctly and that's actually one of the things that my filters look for is whether or not they have the correct capitalization for team health so they actually check a mark on that one unfortunately and this one we did actually stop at a filter but they they tried really hard they also you can see that little black dough area they addressed to the recipient by name so they did everything they they could to try to get someone to click that malicious link another couple examples we see a ton of vip impersonation of course it's really easy for just about any company to be able to scrape the names of the most important people from websites and

linkedin so we have we see lots of emails where they're trying to get direct deposit information accounts receivable lists and request gift card purchases on behalf of the ceo and then another example there just something just what might seem like really innocent and just social engineering attempt just trying to open that line of conversation what's your cell phone number so we see lots of these sort of things we've even seen links in emails that are specific to the recipient so we look at that malicious link so we block that website in the link if they were to click on it they would know exactly who that who the person was that clicked because of that very customized link

so i've talked a little bit about why team health is a target and why that company needs to take their security very seriously i'm going to talk through a couple of the email security systems that we have and then also other and other layers that we have just in case those two systems miss a bad email so how is team health dealing with the onslaught of malicious emails one thing that they did was they hired me i think that was a great decision on their part i am the first person that they've had on their secops team dedicated to email security and they one of these systems that i'm going to talk about they were on boarding as i

was onboarding the company so that was brand new to them and then the other system that i'm going to talk about was being admined by their exchange admin and he didn't really have a lot of time to to spend tweaking and tuning it so this we've taken email security to a whole another level in the last year and a half i am going to mention here we do still have on-prem exchange at team health so you'll hear me mention that and that's been a cost decision for us um up to this point and and i'll i'll point out why that would be the case a little bit later and i'm sure that 0365 is going to

be inevitable for us but right now we still have on-prem exchange and that does limit some of the options that we have in terms of email security because so many um of the email security systems now especially in the the space that i'm going to talk when i talk about armor blocks especially in that space they want to talk directly to o365 via apis and they're not really built to even be able to handle on-prem exchange so the first system i'm going to i'm going to talk about is the cisco secure email cloud gateway this is its latest name if you're familiar with cisco much you will know they love to rename things so this is just the latest name you may

have heard of it referred to as cisco iron port there's still they still refer to it as that in some areas esa which is email security appliance ces which is cloud email security and that's the term that i'm going to be using to refer to it today but this is its new official title is cisco secure email cloud gateways they've been doing a lot of rebranding of their names this year with cisco secure um did anyone does anyone hear have they have you ever managed one of these cisco secure email gateways or work for a company that has this as your seg all right very cool all right a couple um i'm not surprised that it isn't a large number of people i

feel that it's a very good product but it is so customizable that it can be more complex to administer than a lot of the other options so i really like it i think it does a good job but it definitely if you're going to tune it right it it takes a lot of time what other segs do people use in their companies proof point mimecast call out what your company uses for that

i guess okay there are there are lots of other segs out there um i know proof point in mimecast are a couple that i run into on a regular basis um fortinet is one um and a lot of them have encrypted email built in so that's that's part of how i get to see those with that email coming in secure email coming in from our partner organizations

so i will talk through that the cisco cs has a lot of different layers than email has to get through before it can be delivered through to on-prem exchange so i'll talk first about sender reputation you may have heard of cisco talos which is cisco's threat intelligence division and they are constantly looking to see what kind of things are out there and there are a few different places in the the ces layers that get information from talos on a regular basis so the first of those is definitely send a reputation and they telos rates all of the mail server ips out there that it sees on a scale of negative 10 to 10 and then in the cisco setup you can

say what you want to do based on those ratings for example if an email say a mail server is trying to send an email and it scores negative five you can have that in a bracket that says if the reputation is that bad just block the email just drop it don't even wanna look at it nothing else if the reputation's that bad just get rid of it then you can specify another range say negative two to negative five well this might be a little bit on the edge let's let's allow it but let's rate limit it so if it does start sending a ton of bad email that gives us time to react and block it all together

so you can have different you can define as many of those ranges as you want with different rate limits and then have your big block top like okay so if it's above a rating of five then we're just going to let this in at our normal normal speed our normal rate limiting for the email that is allowed through the sender reputation filters that email gets tagged with other information that it gets from talos one of the things that it gets tagged with is its geolocation so where is that mail server located another thing that happens in that first stage is it evaluates the email authentication records so by the time it gets to the the next stage of message

filters that message has already been tagged with whether or not it passed or failed the those email authentication records spf dkim and dmarc so if you're familiar with those and getting those set up in dns that's definitely something that is checked on the way in then we get to the admin defined message filters this is actually part of the process that you have to access the cli for with ces everything else is in the gui so if you're really nerdy and like cli this is a fun area to play in with this with ces all of these messaged all of the message filters are written in regex so that stands for regular expression if you're

not uh familiar with that this is an example of what one of those filters looks like so basically it's it's an if-then statement and you can specify a ton of different parameters um email comes in with a lot of different headers you can check all of those you can check the location the geolocation which is what this one does um so for example we allow we have a white list of mail server locations that we allow email in from and everything else is blocked by default so if you try to send an email to us from a mail server in russia it's just going to get dropped

the next filter that ces has is anti-spam most companies i think that put together anti-spam it's it's kind of their own proprietary way of determining that but what cisco does with their anti-spam filter is they again give a rating to the potential spamminess of the email on a rating of one to one hundred and then we can choose at what level we're going how we're going to treat those for example we can say that anything with a score of 65 and above is so spammy that we just want to drop it we don't even want to deal with it but then there might be a middle range say 35 to 65 where we're going to mark

it as potential spam and we can treat that differently farther on down the pipeline which currently today what's happening is that that email then get delivered to exchange but then automatically shunted to the user's junk email folder so there there's options there and you can tune it based on what type of email you seem to get in then there's an antivirus filter which probably doesn't come as a surprise to anybody ces has a couple different options for that and at team health we're actually using sophos for that anti-spam layer or the anti-virus layer and then cisco amp there's it's cisco's own advanced malware protection and this deals specifically with attachments so if an attachment comes in

that cs feels it needs to it needs some closer scrutiny ces can hold that email in quarantine well it uploads that file for analysis and that that email just stays there until that verdict comes back as to whether or not it's safe to release every once in a while amp does miss one it doesn't happen very often but sometimes amp lets an email through and then later it figures out that it made a mistake there's a method that it has to talk to our local our on-prem exchange that says yo dude that was an accident i didn't mean to do that could you please delete that for me and then let me know whether or not that

was successful so that's a really nice feature called auto remediation that's in ces luckily it doesn't get it wrong very often we don't have to use that very often but it's nice to have when that does occur so basically two layers of antivirus and this is a second layer of anti-spam this takes a look at all the email that's made through this far and it can categorize it as three different if it thinks it's gray mail it can categorize it as marketing social media or bulk and then this is yet another place where because of these tags we can treat them we can treat these emails differently if we want to further on down the pipeline

then we have the admin defined content filters this also is extremely flexible you can look at any characteristics of an email that's made this far including tags that it's received from other layers and decide what you want to do with it do you want to put a special banner on it do you want to remove all the links and attachments because it might be kind of on the edge do you want to send it to a quarantine either permanently or it temporarily until someone can take a look at it and see whether or not it was actually legit this is an example of a content filter and you can see it it also is regex

spaced and the big difference is for the message filters you do in the cli for the content filters you do in the gui but that does also mean that with the message filters being in the cli they're a little bit more flexible you can do a little bit more with not statements than you can with content filters um so there's there's a there's room for both of those specific reasons to use one or the other but this is an example of a content filter that we have

the last stop that a message has in ces before it can be released to our on-prem exchange is what cisco calls an outbreak filter this is another area where talos is constantly feeding information to ces so if there might be something going on with an email that has certain characteristics ces has the option at this point to hold that email in quarantine until it gets updated information from talos so five minutes later it might get information saying okay so we had we're holding all email with this broad criteria we found out that it only affects email with this smaller criteria so you can let go everything else that you had quarantined and just keep iterating through that and to try

to determine whether or not that email might be malicious based on the latest breaking things that talos is seeing out there i have had the question before doesn't anyone complain about all of the delays that can happen at various stages with your incoming email since ces does have the option in more than one place in the pipeline to hold on to that email for further analysis i've been very fortunate in the year and a half that i've been administering the system i haven't gotten any complaints related to that team health is very security conscious and there is a lot of support all the way up to leadership for the initiatives that we have on the in

the realm of security so up to this point i haven't had anybody notice anything being delayed that they've um felt the need to complain about for any infra nerds out there this is a a slide showing the infrastructure that we have with ces with ces being in the cloud and it's in two it's split between two cisco data centers so we've got six virtual esa's split across the two data centers all in one virtual cluster the esa part of ces is what does all of the actual work so that's what processes the email that's where all the configuration takes place with being able to look at evaluate all of those get all that updated information from

talos decide whether or not to send that email through to exchange and then we have the one virtual sma which stands for security management appliance there in the middle that's the centralized management piece so we that's for centralized reporting centralized message log tracking and that's also where all of the quarantines are located so whether it's a temporary quarantine from a layer in the process or a quarantine just in case somebody might want it again or a quarantine where i manual review them for false positives all of that gets stored on the sma so there's one place to go to look at all of that information there are a couple mentions here of ews api that is the path that ces can use to

talk to exchange for the auto remediation that i mentioned earlier so for security reasons we have it going through an aws web application firewall and so it kind of it kind of hairpins but that is two-way communication so that exchange can talk back to ces and say whether or not that remediation worked in addition to auto remediation that i mentioned we also have the option to do manual remediation so if an email got through maybe it was just a social engineering attempt um and so there's there's no attachment that ces might want to pull back but we do want to get it out of user inboxes we can go in and and do that search come up

with that list and request ces do a manual remediation as well so that's another way to try to keep bad emails away from our end users and we have to use that every once in a while another link here we've got b2b vpns between both of the cisco data centers and the team health data center the traffic that goes over those is syslog traffic we have several logs that come down to a local collector for our sim and also ldap traffic flows over that b2b vpn ces checks to make sure that a recipient is valid before it will accept the email so that's where all that traffic flows ces is constantly talking to rad to see

whether or not the email addresses that are being sent at it are valid or not and then the last part is the tls version 1.2 cisco does not give the option to also use that b2b vpn for email traffic so we and we require tls version 1.2 so all of that email in between ces and exchange is encrypted so that's what that last part is is just all the incoming and outgoing email i've done a heck of a lot of talking are there any questions up to this point anything about ces or how we have it configured

um i'm actually not 100 sure i had some very smart people in the organization put that part piece together for me

all i know is it works

there's a route in ces that points to an external ip address that's added to exchange and so yeah they for email delivery ces and exchange do talk directly yep that smtp handoff what's the footprint of team health that you can do all the geo-ip blocking is it just regional team health is mainly us based so that's definitely the most important traffic that we allow that we do get some email from other places and we have some contractors in india that do some work for us so that is another country that has to be on the allow list and i do have a long list of exceptions so even microsoft email sometimes gets sourced from europe even if it's for a

u.s company so there there definitely are a list of exceptions to those blocks just just in case we've we've got that capability just in case the mail server accidentally sends from somewhere that we're blocking any other questions so is this only for incoming email from outside the organization or is

very good question so the question was is ces only for incoming email or does it also see internal only email and outgoing email i'll start with outgoing email all of our outgoing email does also filter out through ces and so we have filters in place just in case somebody internally were accidentally trying to send out a malicious link or a virus we're going to catch that on its way out we also have a limit on how many recipients our people can send to because we're trying to very hard to protect our ip address we don't want to get blacklisted or labeled as spammers so that also is part of our outgoing checks if you're trying if you have too

many recipients on your email you get a message back denied try again a different way for internal only email that's actually an excellent question that lays right in armor blocks so i'll ces doesn't see internal only email which is a really good reason for us to have armor blocks which i'll talk about next

we do uh we we do have dlp enabled on ces and we don't have it completely locked down we're not blocking anything based on that but we are forcing encryption so if it sees a certain if it matches certain filters it will first check and make sure that that is going to be sent via tls and if it's not then it enforces envelope encryption any other questions that's cool thanks for all the great questions guys now i'll talk about armor blocks as our second layer and armor blocks is armor blocks doesn't even see the email until after it hits our exchange server but it talks to exchange via the ews apis and it can remediate very quickly within

a second or two so usually when it sees an email that it deems as a threat it can remediate that before a user even sees it in them in their inbox which is nice and this is a product that can see the internal only email so that is very cool for us it definitely looks at all incoming email anything that ces misses it evaluates it to see if it's a threat and it also can look at that internally only email so that if we ever did have an email account compromise armor blocks could alert us that something was not looking right so we could look at it more closely another big feature that we use armor

blocks for is what they call their abuse mailbox so when an email comes in we put a banner on it and we tell our users if this looks suspicious at all forward it to report phishing at teamhealth.com and that shows up the armor blocks abuse mailbox where we can take a closer look at it it actually helps us analyze that email it gives us information about the sender domain it can show a little screenshot of what a link would look like if we were to click on it and then we can make the evaluation there as to whether or not it's safe market is safe and let the user know that or mark it for delete

and it is possible there the person who evaluates most of these is our security analyst he has made a mistake before he doesn't happen very often he's awesome but for example if he were to mark an email for delete and i look at it later like that's actually a company that we do business with we we do need that email we can just swap it back to mark as safe and armor blocks moves all those emails back from deleted items to the inbox so it's really easy to change your mind if you need to after you make that initial evaluation of those potentially malicious emails we do get a lot of reports to that mailbox that are

actually legitimate emails and some of our end users are hyper sensitive to that sort of thing and we don't mind at all i would way better spend five minutes looking at an email and then being able to say no that's actually okay that's safe to interact with then have a user take a chance and interact with an email that would be potentially bad so no matter whether it's it was a safe email or a bad email we always thank the users for reporting those because we want we want to see those we want to know what's going on in our environment also so say it was a bad email that somebody reported armor blocks will tell us that

it matched on so many other emails that looked exactly like that and when we set it to mark as delete it deletes all of them at once so it's not only deleting the one email that was reported it's deleting all the matching emails

they're they're still working on that they're doing some of that but they're they're still building out that but right now you can get in there and see if they forwarded an email and we have had people that decide to not afford it only to report phishing but forward it to all of their co-workers as well just to get a wealth of input on the situation so so we are able to see that in armor blocks where all did that email end up

okay all right i've got one more slide here for infra nerds what does armor blocks look like in our environment i referred before that we had a kind of a somewhat unique situation with exchange we actually have two separate exchange domains in our environment one is for the admin side of team health and the other is for the clinician side of team health the clinician number of mailboxes is way bigger but way lower volume because they often have their own personal email addresses that they're using or the email addresses assigned to them by the hospital that they're working with so that's one of the reasons why it's cheaper for us to stay on prem at this

point is because the clinician is so low volume that it doesn't make sense to pay for those mailboxes in o365 though like i said before i'm sure we're going to be heading there at some point armor blocks doesn't currently have a way to look at both of those exchange domains at once so we have two separate portals for armor blocks so we do all the administration all the looking at it in the armor blocks cloud in their portal but on-prem we've got three node kubernetes clusters that ac that stands for armor blocks exchange connector so those are virtual appliances that we have on our vmware servers along with load balance pair that talks directly to

exchange so that's what that looks like we've got times two one in each of the exchange environments are there any questions about armor blocks i kind of covered that a little faster

armor blocks does some of that um and one way that we've um dealt with that as well is if ces if it's not on a known list when it comes to ces our email banner shows that envelope sender as well which outlook doesn't natively show to give our users another clue that that is not coming from team health that's not coming from a known address so ces can do some of that armor blocks can do some of that and then we also try to give that information to the end user as well to help them out

oh okay all right so i do not know what just happened to my powerpoint presentation so the next thing i wanted to address quickly though is why both why do we have both of those couldn't one or the other do a good enough job and in my opinion i think they both have a really good place in our environment ces while it's really good and it does get up-to-date information tell us on a regular basis it's still a seg it's still a relatively static rule-based product but it can filter a ton of bad email right off the top that then exchanges armor blocks never have to deal with they don't have to spend cycles on

some hundreds of thousands of emails that ces just blocks but armor blocks has that advanced signal analysis that they are constantly developing and rolling out updates for on a regular basis in order to be able to detect more zero days and more things that we can't catch with the static rules that we have in place on ces so i consider both of them to be extremely important to our environment let me see if i can get my powerpoint back

and then just to show you some numbers so that you can see what that looks like okay this is a report that i generate for our leadership i actually generate a weekly version of this for the cio and then generate a monthly one that my manager can show at the the leadership team meetings um so this this gives you some indication of the volume of email that we get to see so in the month of april over 3.7 million emails were sent at us ces decided that about 2 million of those were probably clean and good enough to let through about 800 000 were spamming rated spam and grain mail and then it blocked over

900 000 emails right off the bat that never that never had to be looked at by exchange just stopped cold up here that is armor blocks and the threats that it caught so that again it's advanced signal analysis where it looks at every email that flows through exchange inboxes to try to figure out whether or not they're bad most of them it categorized as gray mail but you can see during the month of april it did also see things that met its other policies credential phishing payroll fraud social engineering vip impersonation and internal payment fraud so those are some of the policies that armor blocks has specific things that it's looking for specific indicators and signals that it's looking

for in emails this down here is a report about the abuse mailbox that i mentioned so there's a there's a difference between email and instance and i hope you guys can see that better than i can that that writing looks very faint but the difference between emails and incidents is an incident is one thing that was caught whereas emails is how many how many things did it match in the environment so there might have been one incident of vip impersonation but the sender sent in 12 of those bad emails so that counts as one incident and 12 emails so that also ties into this report to abuse we had 122 emails reported by our end users and

that matched over 31 000 emails that we were then able to mark as either safe or mark for delete and you can see um between the two of those at least the way we crunch the numbers for this report the misses by both scanners is half a percent so it's we're looking at a very large number of emails um and and that's a pretty good percent but as i said you know the goal is zero i want i want zero misses and that's what i'm working toward every day with filter tweaks and then one other number i was going to mention so the emails analyzed there for armor blocks obviously that's much bigger than the two million emails that

ces let in so that does include incoming email outgoing email and internal only email whereas the ces numbers only include the incoming email

so what happens if both cisco and armor blocks miss the bad email they try really really hard but it still gets through we have more cisco um at team health we really like cisco so we have lots of their different products we have secure endpoint formerly known as amp so that's on our end points keeping an eye out for any malicious attachments that might have come in cisco umbrella that is a dns product it has categories that we have marked to block by default it also blocks newly seen domains and there's a custom block list that we maintain so if an email comes in but it made it through with a bad link we've got that option of blocking it

with umbrella and like i said some of those are blocked automatically and then we also have cisco firepower and the main thing that we're using that for related to this is again geo blocking that operates with a blacklist uh block list versus an allow list but for example russia again if if that web server is located in russia you're not going to be able to get to it and then just other best practices which you know this isn't rocket science or anything mind-blowing for you but the this is also a consideration if that bad email gets through it is the end user going to be able to do something with it so remove the local

admin permission just in case there's a bad attachment or a bad download especially with ransomware being all over the place we've had a lot of partner organizations get hit with ransomware in the past year so limit your east to west traffic in your network with micro segmentation multi-factor authentication if a user clicks the link and gives their username and password to the wrong website don't make that be an automatic open door for the cyber criminal and then security awareness training you need to help give your users the information and confidence to be able to make those calls as to whether or not that email may or may not be bad

and just to mention this isn't specifically email security but this is my favorite book that i've run across so far related to cyber security it's written for a layperson but it inspired me to improve my cyber hygiene there's lots of good information in there after i got done reading it i lent it to my dad and he incorporated some of the suggestions as well which made my mother very unhappy because now she can't figure out what any of the passwords are but they are safer so if you have people in your life that aren't don't seem very concerned about cyber hygiene and they are readers this is a great present for them or you can

read it and try to slip some of the information to them in conversation for example my uncle mark completely unconcerned about cyber hygiene he doesn't care if anybody gets access to his email because he doesn't have any important information in there so i tried to explain to him but what about all of your contacts if a bad guy gets access to your email account and starts sending out emails as you those people are going to trust you and click so for the benefit of everyone that you love uncle mark please take this a little bit more seriously so this is this is a great read in regards to that recommended for all levels so i've talked through why

team health is a target and then ces and armor blocks as our main tools for that and all of our other layers of defense that we have in place at team health are there any other questions

okay great thank you so much for your time this morning [Applause]