A year in the wild: fighting malware at the corporate level

related to what we do at the ALP when things go a bit not the way they should go with some of our employees machines and so on so a couple of words about me i joined the job security team in july 2014 since then I've been mostly involved in malware incident response or also working on automation some of our security processes tools related to that prior I was working at the research security and Trust group in Sapa in sophia antipolis in france and before that i was the university doing masters in computer science at university of science and technology in krakow poland were emergently from and then I moved to France for for another yeah mmm a couple

of words about Yelp I guess probably must have you know about it if you're around here for a while so we do connect people with great local businesses that's our main goal and when it comes to some statistics from the last quarter that means that we have 86 unique million unique visitors on our mobile application a 95 unique million 95 million unique reviews and seventy percent of our search has come up really from mobile so that's very interesting like to see the group there and there in 32 countries so it's growing and it's very big already what it means is we as a corporation we have more than 3,000 people and more than 300 of them are

engineers and yeah they from time to time they catch something interesting on their machines yeah either be it something relative that what they do privately or what they do for their work so the way we try to fight it is this sort of three pillars of our model response process so it comes to detection where we try basically to see as fast as we can that something went wrong we have to stop bleeding immediately so probably take the Machine up the network try to collect some forensics and basically do things like risk credentials all different sort of things that will let's say prevent any further possible things that attacker could use if he discovers vulnerable machine then we moved to

analysis so what they collected forensics we try to figure out if this is for real or not so we have lots of false positives I'll talk about the later so we have to figure out if this machine is really infected or not and the last stage is sort of trying to remediated so making sure that something like that and not occur in the future and a bit of errors like having less less impact on on our workforce so detection this is something that comes to us from values of different sources so we have sex among the viruses on our employees machines with around other endpoint monitoring software like our square which is open source tool from

Facebook we also have a couple of third-party proprietary network traffic monitoring solutions that also alerts from time to time and we build our own security and even management system which also collects a bunch of different information from us from either this third-party network monitoring tools or other things like OS query into one big data storage and couple of alerts are coming just randomly from the from the employees either true email or someone who's opening hub this ticket they're usually relative to things like fishing some employees noticing some random browser toolbars and pop-ups coming on and think why so at that point of time you'll just come to us and say like hey yeah there is something we are going with my

machine can you see what's there and obviously they probably have some malware infection so the way we deal with all this information coming in is through our automated Incident Response tool this is basically trying to make sure that when whenever antivirus spots something we have a actionable means to provide some reasons for that so what this automated response tool will do is basically that kind of way to process the information from the central machine that collects all the information from from our antiviruses filter out potential potholes positive so we will usually curate something like white list of the things that we don't really care about and they really mean that something is not that interesting and we

should rather concentrate on one why did it alerted in the first place don't try to match a couple of information like IP addresses to employees office so we know in each in which location the user actually is and later we can have a helpless personal come and grab the machine we see the credentials for the users and study basically start the first response and also for our theme for our mauer incident response team to cut the ticket and make sure think with all the information that we can have there to provide some actionable means to fight a fight with what what went wrong there so this is typically how this antivirus other would look like a

couple of pieces of information there like file path which is probably the most important for us because you resent me see something like executable on Mac machine this is not something we we really care about in the first place so we try to leave that out throughout the first response process our square is probably some of you heard about it is very interesting project when you can basically query your operating system this is our passion like you would pray a database and ask for for instance kernel extensions installed on the machine user logins changes in config file hashes browser extensions started items lunch demons and so on we typically have a central instance where we collect all the information from our

square e clients installed on different employees machines and we try to analyze incoming data to provide information if something is okay with this machine up something is not okay basically to alert us whenever they see some potentially changes that could potentially indicate infection on poor machine being honed so we paired it up with another tool that we open sourced which is called last dirt this is basically to alert out of elastic search data so basically all of our our square information goes into central elasticsearch cluster and then we try to using the last alert and certain rules there to other does whenever there is some match so for instance here in this alert you can see

that there was a match with our blacklist and there is a there is information like path to to the to the file and the the user host identifier which is service query wait to say which machine is it so this is typically what we what we do to collect the information from our resources and alert on that so a lot alert is open source if you're more interested in checking out that there is our github project are you can also pretty much work on your own rules like relates to their frequency in your date has spikes similar related flatline which might be very cool when you want to just check if your system is still

alive or not and it also loves to see changes around certain time frames so if you have differences between events of events occurring in certain time frame it's good levels on that another sort of information that we collect is from our DNS resolvers this is very cool because you can basically tell us if employees hitting too much in for too much websites that we actually block it's probable that he's not doing that manually but rather he has something in salon his machine which is trying to basically I request all those websites and this probably indication there is something not run going on with the machine we also use lost lurtz for that so basically we query a lot slower would

query our DNS locks collected in the elastic search instance so yeah I'm going to to the next step analysis as i previously certainly drop this basically a way to figure out whether this infection is actually for real there and whether where is it actually coming from in the first place so we may have a couple of just few simple ways to figure out if this is really something interested interesting for us in the first place by weaving out the false positives basically trying to figure out this is Windows frat on the Mac machine which we won't really care about or Mac threat on the Windows machine these are usually way way less than and the other

way one and also try to figure out who is it because for from couple of the alerts like basically DNS spikes we may not be entirely sure this is the right person we may not have the indication and alert itself that this is the right the right machine to take a look at so we also have to know which machine is it exactly so going back to our example of the spikes in the Nunez blog you may see employee heating values different websites but well this there's like I guess around 20 of them here put 20 different lookups but but well some of them are okay they they're not necessarily you know we relate like

4chan for instance but they do not necessarily indicate there is already something going wrong with the machine so we also have to take a closer look at the view of the domains popping up there and yeah we can figure out it for example oh yeah they are two of them which actually have some hits on either virustotal things like opendns investigator and thanks to that we figure like actually yeah there is the dark certain websites this machine is coming in with that there aren't they're not really something we want to be singing our network traffic so we try to correlate that with other information sources that we have at our hand so we do collect as a crew be summation data

promo s query also into our elasticsearch instance and basically we installed querying around the machine that that cost the DNS spike alert so we'd usually request those query data from from the particular house and then we'll have like list of the extensions all different browser extensions kernel extensions bundles installed on this machine I think like changes in the configuration there is lots of lots of them so it's quite painful to go to them manually we usually try to automate it as much as we can thanks to either having our own blacklist whitelist to make analyst is faster but eventually it comes down to okay you have potential field of a couple of things that are not

looking that great they're not really something typical and yeah you may see like oh yeah there is the zip cloud which is in the in the other staff and if you go Google around you can see that yeah the cloud is some malicious Safari extension that serves you lots of arts and yeah this is not something we usually like our employees to see on their machines so most of our employees use Mac works and because of that we we focus really on our end zone on macbooks and mac OS 10 and basically a couple months after i joined we open source the tool called OSX collector which is a tool to basically collect the forensics from the machine and also a

couple of extensions which we call output filters is trying to do some automatic analysis out of this collect forensics to highlight what's going around there but typically the original size x collector script is just one python file that you can just drop on infected machine run it there and it will collect the forensics for you pachyderm into third easy you can get that back and start the analysis on the machine that's not infected and basically the animals will use to figure out what's what's wrong with the with the infected machine potentially infected machine so the way that OSX collector govers all this information is basically my typically during similar thing to what r square is also doing so

we will collect a couple of things like kernel extensions browser extensions things that are in the downloads directory of the machine also things like browser history to figure out basically what what was the different chain of operations that user has done prior to to getting mullah rounder box bit to be sure that we can in the future prevent further outside that all the information that OSX collector gabbers is stored in JSON format so basically each item like your next entry will have their own Jason object which will have a couple of common fields like file hashes file path also for for executables it will be signature change so it can be also useful for for figuring out whether this

is a summary that's coming from the trusted source light like Apple in this example or something that's not signed so typically being well you called white those things like that yeah and then we start to do automated analysis so basically we'll run this output through to a couple of the the filters which will collect threat intelligence for us like we will either check it against our internal blacklist then we'll run it to shadow server which is sort of way of white listing application so it basically has its source information about all expected hashes on the healthy operating system so we can weave out already the information that's not so much interesting for that further investigation then we will go and

collect further forensics further information from api's like virus thoughtful this is giving us information basically about things that are not that great so it will let us know how many different antivirus is collected picked up this this particular file hash or or a domain and the last one is open dns which we also use for figuring out what's what's wrong with with with couple of different domains and also we can automatically try to sort of filter their browser history to see like what happens directly prior to the incident to be more knowledgeable about the websites he or she visited or different actions the user to and after that we have basically the same Jason file that

came and at the input but documented with all this information so this is typically still lots of information so there is a way in I was x factor to basically just provide you with a shorter summary of what happened and this will basically summarized you all the really bad points so you will highlight the applications that were picked by virus also it will highlight some domain from the history so you can directly see what actually caused potentially caused the infection yeah oh six cutter is also open source so if you if you want to go check it out for for yourself there is our github project and they're also part of OSX collector that's directly communicating with this

different API Slyke shadow server virustotal and open dns is a separate project that we use in all sex collector as a library and we call it Fred int'l it's basically a way to call all this different IP is from Python so also if you're applying to extend your tools with this information you can do it as well we also see a lot of things reported directly by our employees related to fishing so this is typically where the user gets emailed it looks like Dropbox file sure or someone sharing Google Drive or yeah one of our x.x getting like information about tell you should transfer me five million dollars and can you do it so typically

things that usually don't look that secure for us but well you never know some of them are so elaborate that even let's say very experienced user will fall to them so it's also good to have some other protection measures against that but what we would do and typically we'll get things slide user reporting to us the the the email that they get which they didn't really expect it sang them to open the document is we try to investigate what what where it came from so for the users that we that actually report it to us which is really great it's already i guess like ninety percent of the job done it's someone reported in the face place we

try to reorder them so we have this Chompy stickers which is our mascot to basically thank them for their effort to keeping day up secure but the important part of that actually is to have a good employee education program in the first place so make sure you communicate stuff like what fishing is what all this different social engineering practices are so people at all levels in your your company don't fall victim to them what we also did is we basically set up this one email address that all employees know about that they can afford suspicious emails to this is quite important because it's good to have this like one source where where we know we

can respond directly to if employees record it to help desk is always like taking us a while it's taking some time till it reaches us and yeah also something like some of the companies are starting doing certain offering products like automated email attachment scanning this is interesting idea it has some drawbacks it has some some some positive sides as well but this is something you can do as well on your own if you if you want to have more let's say automated way of reporting phishing emails rather than relying on employees but I feel like it's really important to to also give power to the users and then make sure that everyone is on the same page

when it comes to reporting different things that may go around in your in your company so typically for for email we received into this reported to us by by employees to this email address we'll try to analyze a message headers if there is some attachment if your email client didn't pick it up in the first place is a virus we will try to detonate in our vm we try to figure out the past user interaction to see if for instance it was from from their business contact if it was from their business contacts we will be a bit more careful about not blocking this email sender because it was important business contact if you block it they won't be able to combine

some some important business for for example so it's also good important to trying to understand what's the context there and a important part is also to figure out who has received it because if there is one employee reporting something but no one else usually the case is that is that usually a group of people receive the same email but only one person reported it another thing that you can do is go to fish tank on this is website is run by opendns that they basically provide a way to serve if have crowdsourcing database of all different fishing campaigns so we can also submit your own you can search what you received if it's already there and

they basically done all the senses for you so you don't need to go to all the steps yourself so yeah go check it out as well and remediation so what we usually do is well if something is really factor with the swipe at yeah there is so much it can do about that but this is sort of like that the last the last possible way to do it we try to do some other stuff before them as well and so we'll try to do things like blog things at dns resolution update our firewall rules also trying to go through our indicators of compromises so all this different blacklists white list that we curate that OSX collector is

using to make sure that it's always up to dated and there is they're very some information there which is not stale for fishing campaigns we usually plight try to block someone send the email address also good idea is to maybe block the way way back so not make it possible for the employees to send the message back to that to the Fisher because yeah they receive something like hey can you provide me all this secret information like the list of your social security number if your employees the guys are good to prevent the channel back to that to the vision and the important thing as well is to communicate back to your employees especially if it's like larger

fishing campaign you may have some coming I would go in communication from your team to say yeah you should you should be aware that there is something like going on it also makes a stronger message during process the message that they get previously from from things like anti-phishing education so sort of going back to all this three major steps that we do like detection analysis and remediation in the first phase we try to cover well first collect the alerts from from various sources make them actionable make sure it's fact is happening fast we don't waste time on things like false positives or something like or wrong go as analysis will basically after collecting the forensics

will try to correlate the information from various sources that we receive and also run some automated analysis to make sure that things are repeatable rather than going one by one with each of the cases that we see and the remediation will will be there yeah wiping the machine if it's really infected and yeah taking all the results from the analysis and making sure that our tools are getting better and better along the way so also updating our blacklists blocking educating users things like that so all the steps that we do there they're sort of not separate or not like set in stone they're not something that we just tried to follow you know from oh yeah hi so we've heard

a couple people in the background and a hard time hearing if you guys are talking a lot and want to talk to your neighbors I'd be great if you could move to the lounge over there something because the people in here really want to hear what's going on so the people in here ah well that's better well can everyone hear me now good good so the people in here want to hear what's going on here so we've heard that some people in the back can't really hear what's going on so if you're talking a lot and want to talk i recommend going to the lounge above or somewhere else because this is the only place that we can hear

this young gentleman here give his presentation but you can talk about the people anywhere else thank you ok I should just pick a lather also how so yeah as I was mentioning this is this is not the process that we sort of try to go from from from you know step 8 is to be it's not set in stone it's not something we just like follow without ever giving it another thought so we'll also try to make it better along the way and the way we do it is basically we yet our goal is to try to make the response faster and more accurate so we aim at reducing that the number of false positives is happening around either

trying to have better tools basically updating our tools keeping them fresh up to date and whenever we see some repetition also trying to make sure that it can be automated so if we're doing something more than twice three times will try to maybe run script around that if we are we see that we use this script a lot we'll try to make it included into into our response process along the way so for instance combine it together with our ticketing system and make sure that everyone is using it so also important part of the whole process is is education whenever there are any changes along the way you need to make sure that everyone in your organization is

knowledgeable about them knows how to do it so also make sure that everyone's that there is a and always exchange going on around the different different parts of the of the investigation process to make sure that everyone is familiar with the fools everyone knows knows how to use them and everyone knows how to update them if necessary I me all these different pieces are coming on around to fine tune our hour of our response so this is pretty much it will I have for today if you have any questions I'll be more than happy to answer them thanks for listening

so the question is like at the beginning how the organization how big was the organization and how the process changes change along the way so what the Elvis is quite particular that okay we are growing like year by year since i joined like there where there is like rapid grove of number of employees but especially it comes to more like personal like sales for instance which are not necessarily tech savvy people but we still want to make sure that they they're able to use their machines and they're not getting impacted so it's it's basically trying to to make sure that you know whenever you have that look like this bunch of alerts coming in to try to response as fast as you can

and if you see some patterns in the in the responses so if you see like this one large campaign affecting most of the users you should have maybe some targeted response rather than just following this step-by-step what what what i mentioned before also what helped us a lot was at the very beginning we didn't have dns resolution we were not blocking traffic italians resolution level this helped us a lot is basically like cut our commodity malware alerts like more than a half i guess so so that was very helpful so i would say like you're not doing that yet so it's probably consider it some other question i think the question is what was our

network intrusion detection system i mean we used a bunch of things that i've mentioned the the combination of the tools I've mentioned we from the third party proprietary software yeah that's so comfortable dropping the names but it can have you take it offline if you really want to hear the let's say this if feels like I guess the important message we are not very you know like we are not married to them that this is my take on that always so so we see something like you know certain third-party Eve and there's not giving us what we want and not really over they give us in the first place is not really helpful for us and cost a lot but we

will just try to evaluate it and we could possibly change it so it's all so I guess important aspect to make sure that your tools are quite interoperable if you need to change them from from one third party vendor to another like from one network monitoring solution to another make sure that you keep the sort of interfaces in between that if any of your tools below will change all the rest of your of your stack can still work any other questions there is wrong

so the question is how lot is our security team I think we have around 15 people right now just in the core security team some of them are around here so if you if you catch any of them to say hi and we also have like generally some tech security savvy people on our team in other teams like operations yeah pretty much like we try to make sure we have connection in each of our teams yeah so yeah it's it's like there is core security team of 15 people but also we are communicating well with other with other departments some other questions madam thanks a lot and the SC around you