Tony Drake Incident Response for the Overwhelmed, Understaffed, and Unprepared

hey everybody um thanks for coming to our um last talk for today tony tony has over 25 years of experience in infosec and systems administration he's working roles ranging from systems design to incident response tactical intelligence and managing pen tests today his talk is instant response from the overwhelmed understaffed and underprepared above the human factors in this instant response in the russians came to the end today anyways um thanks a lot for being here tony and you go ahead well good afternoon everybody and uh yes last talk of the day uh so my name is tony drake and this is incident response for the overwhelmed understaffed and unprepared quick outline of what we're going to do

today we're going to start with an introduction i'm going to introduce the concept of guerrilla incident response talk about the steps of gorilla incident response tools you can use we'll pause for q a and then depending on how that goes we have story time so who am i i have nearly 25 years of what i used to call playing with malware i shudder when i look at that and see that because it has been a long road and i have seen a lot of changes over the years have about 18 years uh doing system administration and security roles ranging the full gambit as you can imagine i hold certifications of the cissp sans web app pen testing network pen

testing forensics threat intelligence and open source intel lots of battle scars and lots of hard-won knowledge i hope to share with you today so a quick disclaimer i'm going to talk about things i'm going to tell stories i don't want you to assume that anything i say or anything in these slides refers to any particular employer past present future real imaginary you'll get the idea also i am a first person storyteller i grew up in the south there is great debate about whether florida actually is in the south i say it is and i will tell stories in the first person i me we our again that does not mean that i necessarily did it or it happened

to a particular employer i am telling this story to show in and demonstrate a technique or an idea so story starts with a phone call in the afternoon well actually it was a link message it was a chat message when the director of operations said hey can you look for something in the sim i happened to run the sims so this wasn't easy of course i can what would you like me to find that the next statement was the only statement i needed really to hear he said look for suspicious stuff now any of you who've got the pho gotten a phone call or a chat message like this knows that that warrants a deeper discussion

a deeper discussion i made a phone call and i said what exactly is going on in the contents of that conversation which i will not share essentially had me picking up my stuff at work grabbing my bags and going 500 miles to a remote location to meet the director of operations and the forensics team to start doing forensics and as i drove down i thought about the fact that what i was about to do was unlike anything that i had ever done i was going to do gorilla incident response and i'm not talking about the first kind of gorilla and i'm not talking about the second i'm talking about the third i'll give bonus points to

anybody who puts in discord without googling or image searching who the second gorilla is so thing is that incident response has rules we have rules we have policies we have procedures we have things that we have to do that the corporate says we do we develop playbooks and orchestration but there's another set of rules that most people forget about and that is the rules to guide you as a responder rather than to guide an incident response these rules allow you to be effective and efficient in dealing with an incident and this talk is going to be about the second type of rule if you cannot cope with the stresses strains challenges and emotions that you run

into in an incident response you cannot respond you have to cope in order to respond so rule number one is don't panic adrenaline rushes are real and keeping calm is important i can tell you that there is a real physical reaction and it doesn't matter whether you're talking about a sock incident or an ops incident when you get the phone call when you see the alert when you look at the screen and you know that something big is about to happen you get the chills down your spine your the shivers in your arms and the pit in your stomach you need to get past that in order to respond keeping calm is important rule number two write it down i'm going

to introduce a concept that's going to be new to many of you especially those of you who are younger it's called a pen and a paper and they come in these little things we call notebooks and this is what i mean when i mean write it down the biggest asset to an investigation is your notes your notes will be real time and later they will help you with the report and they will help you with the response i like to buy a separate notebook for every single incident some people use the same one some companies have specific ones it just kind of goes with whatever works but here's what happens you'll be working on something

and you will pardon me you will find an idea you'll be working on thing a and say oh i've got to do thing b well you have to write that down because if you stop working on thing a and move to thing b you will never finish thing a if you write it down you can rest easy that you will get to it later and when you finish with thing a you will now know what you're going to do next you're going to do thing b you can use the voice recorder on your phone if you happen to be out for a run out for a drive at the grocery store doing some work around the house and all

of a sudden you get a brilliant idea record it i'm not talking about siri or google voice dictation your level of paranoia may vary but you know that information does go to the cloud to be translated into text so i just record my own voice on what used to be called a tape recorder now it's a digital voice recorder or a cell phone and then when the time comes i go back and i write my dictation into the notebook rule number three now is not the time for kuda woulda shoulda this quote is from ernest hemingway and he said it best he said now is no time to think of what you do not have

think of what you can do with that there is what this means in our terms is that during the response you are going to think of every single project that isn't done everything you wanted to do and they said no every budget item that was rejected that's for the post-mortem right now you've got to get to work well number four you have to be flexible now we all have playbooks plans procedures workbooks orchestration you know whatever you want to call it and it has been built for every possible situation except for the one that you are dealing with right at this very moment in time in military terms no plan ever survive contact with the enemy

so you need to always maintain compliance with all of those things that you have to maintain compliance with but just know you're gonna have to be flexible you're gonna have to go off book you're gonna have to get manager approval to do things that they wouldn't necessarily tell you to do if you weren't doing this investigation rule number five this is probably the hardest one you're going to screw up and now you want to push and that's perfectly fine making mistakes is part of incident response it is what happens during a response you're going to do something and say man i coulda woulda should have done something better the most important thing is to let

yourself off the hook at the beginning of the incident before you start and keep moving don't second guess rule number six just the facts and that's probabilities you know nobody will ever be as interested in who you are or what you do as when you start working on an incident response you are going to be asked for information updates status and all of that fun stuff by people who you may not have even known exist and who you probably didn't even think you think knew you existed the problem is during an incident response the situation on the ground is fluent is fluid and rapidly changing as they would say in the military so i use the language of estimative

probability to express what i know and how sure i am that i know it you've probably seen these in reports high confidence medium confidence low confidence things like that so we know with high confidence that at 12 midnight on sunday mary's machine began communicating with a country with which we do not do business and with which we do not have any customer relations we know with high confidence that mary is not in the building because mary is on a cruise ship and she the badges logs say she's not in the office and we have pictures of her on the boat we think with medium confidence that mary might have gotten infected in the recent travel related web

compromise that we read about here why is that medium confidence because on monday when you go through the email logs and you find the delta plane ticket themed attachment that she actually clicked on that you found on the laptop you will now change that item to say we now know with high confidence this is how she got infected rule number seven always know your aimed goal as satisfying as it would be for our investigations to end with a perp walk with someone in handcuffs that's not the world we live in the sandbox we play in if you are doing one of those kinds of investigations insider threat hr legal etc there's a whole bunch of

stuff that's beyond the scope of this talk that you need to manage and monitor if you are not doing one of those investigations you have now changed you are not matlock or perry mason you are a cross between a commando and a janitor your job is to get in get the bad guys out clean up the mess and make sure they don't come back rule number eight voyeurism is bad you do forensics you will have access to all of the data on systems sometimes these systems belong to very high level people and have a lot of things on them there are four types of data that you will find in an investigation there is data you need to know this is the

malware samples and the emails and the files that are related to the compromise there are things you do not need to know basically there's everything else there are things you do not want to know these are things like browser histories of people who may not always surf only to the corporate website and there's information that is salacious and interesting and you are going to treat all of this other information like it is radioactive medical waste in a landfill you're going to avoid it like the plague if it does not get you to containment and eradication move on forget about it rule number nine eat walk sleep and avoid burnout we are dedicated to our jobs we are but

we are also human and humans have physical needs that they need to manage you need to eat you need to sleep you need to exercise you if you have a routine where you exercise every monday wednesday and friday you need to keep doing that if you don't have such a routine i suggest you start you have to get up from your screen and walk away this means eating lunch somewhere other than in front of the computer this means walking down the hall or walking outside in a circle in the park or something you have to get away you will find you are much more effective when we look at evidence from breached companies we see that long

and detailed incident responses especially the high visibility ones burn out the sock in the words of smokey the bear to paraphrase only you can prevent you from burning out if you make sure that you take care of yourself you're going to be sharper you're going to be more effective more efficient and you're going to get a better result so now that we've talked about incident response and how we're going to be good incident responders and how we're going to take care of ourselves what exactly do we do so the first thing we have to do is we have to start with an incident if you don't have an incident you aren't doing incident response i know this sounds like a no-brainer and

thank you captain obvious but all alerts are not incidents and all investigations are do not turn into these types of incidents once you know that you're dealing with an incident of this type you've got to evaluate your options you now know that something's going on and something could be big you have to make decisions about whether you're going to work at yourself or you're going to call in external help you have to understand what kind of incident are you dealing with and how big it might be and how sure you are of how big it is once you've evaluated those options you've decided you're going to work it yourself now you need to look at what

you have and how you can work this incident with what you have have you ever done a response like this before what data is available and what data is missing i can tell you from experience that no matter how much equipment and fancy instrumentation you have the incidents never happen on those well instrumented endpoints so if all if you have data missing which i've never been in an investigation where i said man if only there was less data to go through can you deploy something to help i talk about this a little bit later but if you are going to deploy something to help how long is it going to take who's going to do it and what approvals

do you need if you're stuck phone a friend this is really an important one and conferences like this usually in person we meet up with people we exchange business cards or email or linkedin but when you're at an investigation you're not going to know everything and there's a lot of situations where if you call somebody or talk to somebody who's done it before they can usually get you over the hump if you're in an industry that has uh you know an isac or an isal or an industry organization that you're a member of maybe you can contact people through that if you're a member of the local chapter issa ic squared college cyber security programs

maybe you know somebody there who may be able to give you some advice trusted friends or colleagues from conferences and certain types of organizations can enlist help from the us government the cisa the cyber security and infrastructure security agency and the us cert please assume you are being wired depending on who the attacker is in the situation they could be in your emails they could be in your voicemails they could be in your chats this is especially true if you have a cloud breach and you're dealing with a breach of a cloud system if this is not the adversary in your stuff you also have to remember that nowadays everything is legally discoverable if you work in a large company your

emails and chat logs and all that are being archived and saved and people send requests for discovery and so there are certain words that you may use that may not mean what you think they mean it may be discovered and then you will end up later on trying to explain what you said how you said it what you did it is much better not to type it than to have to explain and defend it later if you are getting outside help and or using your outside counsel you have to discuss it with your legal department and or your external legal department to decide exactly how communications are going to take place observe your evidence all of it so

long ago and far away when forensics started as a discipline we had rules of thumb and it was find the bad thing and unplug it take it offline well that's great but you lose a lot of data when you do that an unsaved artifact is an artifact that you are sure to read or want later so there are a couple things that go into preservation the first is you never work on the original you have to make copies of everything you have a golden copy on an external hard drive or something like that share drive that you don't touch and then you work on a copy of the original if you screw up the copy you're working

on you make a new copy if you are uh dealing with a user endpoint give the user a loaner and keep their machine it may help you later to be able to go back and get something directly and while i'm talking about preserving evidence i want to talk a little bit about memory because memory is one of the most useful artifacts and it's also one of the least understood in order to capture memory on any windows system is trivial excuse me instead of shutting the system down what you do is you hibernate the system if it's not a laptop where the hibernate option doesn't appear in the options menu there is an administrator level command

that you can execute which will allow you to hibernate the system this will write a file called hyperfile.cis which can be analyzed as a full memory dump of the system if this is a more advanced group you may be able to look and see if there's an existing hyperfile.cis and you can copy that to hyperfile.cis.old and now you have two copies of memory if you are dealing with a linux system memory capture is a lot an analysis is a lot more complicated i'm not going to go in the details here but you can google avml and there are some great blogs that explain the intricacies even if you can't get all of the things that you need to process

linux memory save a copy of the raw memory you can analyze it with strings you can carve binaries out of it and do a lot of things even without being able to use tools like volatility so now that i've talked a little bit about how to do a response let's talk about tools this is incident response for the unprepared which means that you probably don't have a fancy tool kit waiting for your use so here are things that you can get and grab and use immediately first thing you're going to do is gather artifacts in order to analyze data you've got to capture it and one of my favorite old school low-tech tools is called tr3 secure

you see it's from 2013. it uses a couple of tools raw copy and robo-copy to copy windows artifacts and put them into a zip file on the infected machine so that you can then take those files and analyze them on something else redline redline used to be a mandiant tool then it became a fireeye tool as fireeye and mandiant have now split i'm not sure whose tool it's going to be going forward but redline is kind of an all-in-one memory and system timeline and triage tool if you don't have any other tools and you don't have a lot of expertise using redline can basically tell you where to start it will grab a full copy of memory which can be

analyzed with other tools as well as doing timelining and registry analysis and sometimes it will even tell you which uh binaries are suspicious hanzo and power forensics these are two powershell tools if you are a powershell person you probably think you can do just about anything in in powershell well power power forensics and console allow you to prove it you can download these frameworks you can look at the scripts you can improve them change them you can do whatever you like with them and pull artifacts and do an incident response last query if you are dealing with non-windows machines os query from facebook and their security people is a great tool it essentially gives you

a kind of an edr capability but it's open source it is very good for macintosh and linux systems for which a lot of the tools that i talk about previously do not work security onion if you're going to be looking at any kind of network data security onion is your tool you can deploy a security onion sensor on a network with just a uh basically any old pc or any old server with two nics one to listen in promiscuous mode and one to connect to the box it has everything you need to do packet capture logging ids you name it it's in there so now we've gathered the artifacts we have to analyze them and again since this is incident

response for the unprepared how exactly are we going to do that well my first tool is a sift vm sift is put out by sands and they spend a lot of time and effort to make a fully functional virtual machine that you can download log into and start analyzing artifacts uh currently it is on ubuntu 20.04 there's an iso it's downloadable as a vm if you want to build your own ubuntu vm you can basically install sift onto that vm all of the tools that are on it have been tested and set up to do incident response tasks and analyze artifacts you have collected in with tools in the previous slides sometimes it doesn't have exactly

everything you want but it's very easy to install or update tools from github if you want something if you are going to install another tool on an existing sift box you should use a virtual environment so if you're using python they're vms if you're using ruby i believe they're the same but you do not want to install new tools new libraries new language uh libraries etc onto a sift box because you don't want to upset the delicate balance that sans has spent so much time trying to preserve easy tools so along with sift uh sans paid a instructor eric zimmerman to create a set of tools to essentially take care of processing every single windows artifact that you

could need in an investigation these are powershell tools you can download them you can look at them you can change them but they're they're very effective if you're doing analysis on a windows system bro so bro is part of security onion i like to think of bro as a packet decomposition and logging tool that's how i describe brill what it will do is take a pcap file so you have managed to get packet capture maybe your organization has it maybe you've gotten it some other way maybe you've gone on a box and done tcp dump or wireshark do not have a p cap and it's huge and you want to get information out of it you can run bro using the

replay option sometimes it takes a while to figure that one out but on that p cap and what it will do is it will take that pcap and it will split it into logs for each level of the osi so for example if you want to look at http request there's an http.log if you want to look at dns there's a dns.log there's a connection log which has essentially netflow this allows you to very quickly identify traffic you're interested in if there are files involved if for example you think that there was a malicious file downloaded and you happen to have a pcap of it that is pure gold and you can use bro to carve that file out and you now have

the original file which was sent down from the network it's awful if you do not have a sim and you do not have a way to process logs the windows uh your log viewers are are definitely subpar for that task so the easiest way to do this is using a vm called soft elk it was developed by phil hagan another sans instructor for his sim class this basically is a sim on the fly on a vm you run the vm and you can immediately import logs you can syslog to it all of that fun stuff it is free and downloadable from the internet so now we move into the the futuristic phase of the talk where i

say what if you're not doing an incident right now what if you actually say you know what this really sounds like a lot of work i want to do stuff ahead of time so that i am not running around trying to do all these things during an incident now first thing i tell you is use sysmon on every windows system if you are not running sysmon my only question for you is why and after you give me an explanation i'm going to say yeah but why and then after that explanation it's also going to be yeah but why sysmon is a no-brainer it is a tool for microsoft sys internals it generates what windows logging should

have been but never has been it has a config file and if you don't want to mess around with creating your own config file and figuring out the things you want and don't want that's okay all you have to do is go to swift on security's github page download the sysmon config swift on security has generated a file that will work for almost all circumstances and give you the data that you need in a log to be able to find evil on the system if you are dealing with microsoft security event logs on your systems it pays to be very diligent there are a lot of things that the microsoft event logs especially the security event log

can log there are a lot of those things that it does not log by default there are registry keys that need to be set gpos that need to be enabled scheduled tasks things like that there are sometimes patches that need to be applied so if you need to go through and look at what you are logging in your security event log and if you want to log things other than what is in there you need to look at how to do it and make sure you test and implement the am cache so starting in windows 8 we added a new registry hive to windows called the am cache they back ported it to windows 7

it requires a patch and a scheduled task in order to work but why do you care about it the amp cache is forensics gold it keeps track of file execution on a windows system complete with the hash of the file the date the time the command line information like that it can be a goldmine of information when you are trying to sort out an incident and don't have a lot of other information to go on remember that logs can be stomped but am cash cannot be so now i'm gonna pause uh for some q a if uh we have some q a we can do it now yeah um if anybody has any questions uh

please post them in general uh we haven't gotten anything yet all right well in that case i'm going to go ahead and move on to uh kind of story time these are scenarios and situations i want to remind everybody that my disclaimer from the start of the talk still applies again don't assume this necessarily happened to me don't

so here is my contact information i'll uh ask one more time for questions

all right all right um thanks tony for your talk today uh nobody always asks any questions yeah come on you all you're being very silent well i guess i i guess i explained everything perfectly wow yeah i guess so yeah so it's it's also the end of beyonce so i don't blame folks if they're feeling feeling tired i i get get give me all our things well um all right thanks again tony um i don't know if you'll be sticking around in the chat for a few minutes um you know if you are people might you know have come with some questions and um poke you in the chat but uh yeah again thanks so thank you so much tony and um

for everybody who's watching the stream we are about to switch over like almost instantaneously after i'm finished speaking to um aaron and dan who's uh representing rule four today and um getting ready for the countdown