How to Orchestrate a Cyber Security Incident Tabletop Exercise

all right everybody next up we have Mel give her a round of applause okay a quick show of hands how many people are not from here in the Bay Area wow that's a lot okay so welcome to sunny California okay so my journey into InfoSec if you watch Rachel's keynote I loved her nonlinear path and in post suck and mine is a little more linear although I didn't realize what it was or that it was I was a Systems Administrator specializing in Windows operating system and the threat response manager at the time I had a really good working relationship with InfoSec and he asked me to go to happy hour and I said sure let's go to happy

hour and we got there and he said you realize this is a recruiting meeting right and I was like what what kind of Systems Administrator possibly have to contribute to information security like what what are you talking about I had no idea that it was absolutely the right skill set my name is Mel Masterson I currently work at Airbnb on the threat intelligence and response team I was pre-op Lending Club doing the same thing nothing I say here today about any incidents is has happened at my previous or current employer I'm also the events lead for women and security and privacy wisp and yeah and our mission is to support and advance the careers of women in information

security and data privacy a colleague so today's presentation is on security incident response tabletop scenarios but I want to spend the first few minutes talking about security incident response before we dive into the tabletop exercises and we're gonna do them together and I want it to be interactive it'll be fun and a colleague once told me that when he joined incident response he observed how it operates and it struck him that it operates much like the human immune system in that it's responsibility isn't just to respond when there is a foreign invader which is whether it's bacterial virus our other infectious agents it is also responsible for immunizing against future attacks and that is in a nutshell

what we do when we're not responding to incidents or better where we are making better minister automation capabilities and of course tabletop exercises help improve our response process so what before we go over this answer I want to give you an example of what a security incident is not so a colleague of mine works at a tech company here in San Francisco and it's a household name and I can't tell you the name of this company because it's an embarrassing story he would kill me and not be my friend anymore if I said the company name okay so this this company has multiple offices around the world and they're in one office it's a company

office where they share the building with other companies on different floors and the physical security guard is in the parking lot and he hears two people talking about how someone was infected with ransomware physical security guard goes and reports a security incident in all caps subject was eminent cyber cyber attack happening now that is not a security incident so the definition is a security incident is when an event or a series of events has been confirmed to be of malicious or fraudulent intent and is in violation of your security policy which is deemed unlawful unauthorized or unacceptable involving a computer system network device or any device with an operating system when we use the word

incident it implies harm or intent to harm an event simply refers to an observable occurrence okay so here is an example of a real incident that happened I've condensed it into a short paragraph the case study is actually in an incident response book that is 12 pages long we read that we read that book okay so here it goes an attacker exploits a sequel injection vulnerability on a publicly available web server there's a security Miss configuration in the DMZ firewall that allowed the attacker to execute commands on a database server running inside the corporate network the attacker spends the next few months doing extensive reconnaissance evading antivirus installing backdoors cracking password hashes establishing persistence and ultimately commanding control

because they had established persistence they would go in every couple months or every couple weeks and dump new cardholder data about ten months later a systems administrator notices some weird activity realized it was malicious and an incident was declared but think about that for a second it took ten months before detection so the Incident Response process as you do tabletop scenarios and get your incident response team to a really solid point you could potentially have detected that instantly if they had the necessary log files feeding into a central aggregator they may have been able to set to detect the initial attack vector so being able to detect and respond efficiently depends partially on the fidelity of your

logging and there are many different log sources and this list is by no means comprehensive but it gives you an idea and it's imperative that your logs your log files are centralized into a log collector or sim security information Event Manager more acronyms so I wanted to ask you what are some other log sources that are important anybody active directory logs mmm cloud trail logs administrative access would you want to know would you want to be able to detect if administrators account was compromised and added additional users to the domain admins group you wouldn't know that or you wouldn't be able to detect it if you didn't have these logs oh and speaking of bad ass

detections if you have not heard of the pyramid of pain I recommend that you write that down and read about it later I don't have time to talk about it today but if you want to build high fidelity detections pyramid of pain is where it's at okay so instant responders we rely on what are referred to as indicators of compromise or IOC s what are some examples of IOC s Cash's IP address ers figs what URL right so other examples include the IP address of the website visited to download the malicious content the IP address of the command and control servers the that the affected hosts are beginning out to or any other host based artifact it can

even be a modified registry registry entry it can even be a set of commands so if you very much like a yaris egg like if you can if you can determine if you determine that this malicious payload always uses these three commands then you can build detection off of that so when we make our decisions we rely on data and intelligence data is information a statistic or a fact intelligence is what you ascertain from a process of collecting different pieces of information and analyzing and disseminating it into something useful for your organization and for your investigation the difference between data and intelligence is analysis and when we do analysis we are doing things such as sandboxing so we detonate

malware to see what it's doing we reverse malware we correlate data to see hey this thing happened at this time well this thing other this other thing happened also at this time maybe they're related and if you have not heard of that who's heard of The MITRE attack framework okay and then intelligence so there is for example Oh sent open source intelligence mm-hmm which is your Google LinkedIn all of that good stuff and there are too many sources of intelligence to list here but we have for example the Homeland Security a Department of Homeland Security will share their indicators with you if you sign up for that the InfraGard is a partnership between the FBI and

private sector they will share their indicators with you and then we have the ice axe and that stands for information sharing Center and there are ice axe for lots of different industries health care aviation finance lots of others so you if you want to know about what affects your organization or in your industry there are six phases of Incident Response and it is a coordinated and structured approach to take an incident from detection to resolution and first you confirm whether you have an incident and declare it and you know when you're in a major incident in a war room the tension is very thick who's been in a war room before yeah super tense and

energy it's very serious you know I've seen people's faces go blank people forget to eat and drink and there are people there are multiple stakeholders in there including the executive team which makes it even more uncomfortable and the incident commander should remain calm establish leadership and control that person that person should be prepared to delegate roles and responsibilities whether that is to ask someone to take notes or even to go get food trust me it's really hard to remember to eat okay so you determined and document the scope of the incident and remember that your goal is to reach containment and minimize disruption the stages of IRR preparation this is done by doing tabletop exercise and really

getting your training your Incident Response Team so that when the incident inevitably happens we know what to do the identification phase is only achievable well maybe not only achievable but the best way to be really in a good spot for this there.this is by collecting your logs that way you can build detections and identify when you have an incident when you have an incident the next thing you want to do is contain which is stop the bleeding so you haven't actually resolved the problem but you have stopped the attacker from being able to continue doing any more damage and then is the eradication phase and that's when you have put the controls in place that make

this type of attack no longer possible but you want to trust but verify so when this happens let's say you've contained something you put a control in place like two-factor for example and someone says ok this is done you want to be able to go and double check and make sure that try that attack again and make sure that it doesn't happen and recovery and lessons learned are the phases where you the the pain is over and you're really getting together and documenting everything that happens talking about what we what you can do to get all the security controls in place that you need so that it cannot happen again and sometimes the recovery and lessons

learned will not sometimes I'd say who would agree that the recovery and lessons learned takes the longest right it can take weeks and months and okay tabletop exercises are they give you a way to practice what you what your response would be given a hypothetical incident scenario the purpose is to get better and faster at responding to a security incident so in many breaches the initial attack vector is a phishing email so you can just use that let that be your first tabletop exercise what would you do if an employee of yours successfully got fished what would you do if several of your employees got fished do you have a do so do if someone

clicked on a phishing email in your company do they know how to report it do you have a security reporting policy write that down as a betterment if you don't have that here's another hypothetical scenario let's say an attacker gets ahold of a recent credential dump from one of the many recent breaches they take those usernames and passwords and just systematically run a script and try all those combinations against your platform is it likely that some of those will be successful yeah so they're automatically year now so now you have some customer accounts that are compromised you can use that as a tabletop exercise what what do we do would be would we be able to detect that so how do you

formally conduct a tabletop exercise you think of a scenario that is relevant to your organization that's the hard part I don't people say to me I don't know what scenario would work for my company and so what you do is you come up with an idea and you work with maybe someone in IT or someone in engineering and say hey would this scenario work on us and if they say yes maybe or I don't know then that used that as a tabletop exercise so you have your tabletop exercise scenario written out you put it on a slide and then you begin the tabletop exercise you invite your IR team information security team and anyone that this scenario would impact

so for example if it would impact information security I'm sorry information technology folks invite the help desk team and if it would if it's something that maybe would impact your platform invite the engineering team you want to also make sure you invite their leadership it's very important that leadership is there so that they see exactly what a [ __ ] it would be if you didn't have these answers okay now this is really good this is a really good tip here if you're having trouble coming up with relevant scenarios you can also some tips are read some case studies of reaches that have happened look you can take the Equifax struts example if you want to

like you could say hey Equifax was compromised by this stressful of our ability then ask your engineers hey do we use struts yep okay could this happen to us yep what was the last time we patched two years ago okay let's let's use that as a tabletop exercise also there is a person his name is Ryan McGuire of information security and coinbase and he has a Twitter page solely with whose sole purpose it is for Abell top scenarios so remember this to write it down it's called at bad things daily bad things daily and two or three times a week he'll post scenarios and you can just and the purpose of it is to just take

that whip your chair around and talk about it on your team say would this work another good source for coming up with tabletop scenarios is if you have how many people have a bug bounty program in their company okay so about half it's probably a good idea for the rest of you so go to go get your most recent bug bounty report look at the findings and say oh this is what they found okay so this is possible let's use this as a tabletop exercise okay okay so we're gonna do a tabletop exercise next and I'm gonna you're gonna see a scenario on the slide and we'll all read it and then we're going to all of us

we're going to interactively talk about what would happen next and the other thing is to remember and when you're doing your tabletop exercise that your company is encouraged people say it's okay if you don't know the answer that's the whole point we want to know the answer now before the incident occurs and we don't know what to do then so it's okay if you're like I don't have no idea what we would do okay here we go a disgruntled employee writes a quick script to brute force all LDAP accounts now all the clients including service accounts and super users are locked out this affects production systems and the code that relies on those service accounts to run the platform is now

offline how many people use active directory here yeah okay so um active directory people people usually have a password policy and an account lockout policy so let's say the account lockout policy was if you type your password in wrong five times lock the account lock the account for 30 minutes okay everyone in active directory every user has read-only access to view the account lockout policy it's very easy for an employee who wants to [ __ ] to go and view the account lockout policy very easy to write a script and lock out every single account what would happen next well yeah because this person wrote the script what let's say okay let's say the script omitted his

his account okay so let's say and then what I would do in this in this exercise I you you insert what's called an inject and you inject more information so in a real scenario I would click the next slide and the next slide would say it's 8 p.m. on a Wednesday night hmm so most people are not working but there are some people that are working how would this be detected what would happen say louder right okay yep yep yes you said there would be alerts firing for things failing the system should be yelling at you yep right so maybe like your DevOps team would start to be like wow something's wrong okay someone else

said something up here help desk calls right so someone that's working on the VPN all of a sudden which loses access and they're like what the f so they might call helpdesk and be like I lost I lost I lost VPN oh okay what would the helpdesk you try to log in right and it wouldn't work because the council would be locked okay so then the helpdesk person would go whoa you're my accounts lock too and then all of a sudden now people are starting to get like really panicking right like people are starting to call each other account lockout policy expires in 30 minutes the disgruntled employee runs the script every 31 minutes what would happen next

what else would happen

he said he said have we determined that we are running the script from inside the network I don't know how would you do that you're you're you're an admin you're a Windows admin let's say how would you determine that he said we haven't necessarily determined that this is happening yet so yeah so basically what he's saying is shit's hitting the fan people are starting so yeah people are now starting to know something is really wrong because multiple phone calls this is now at a manager level at least director level at least and what would you bet that one of someone on the executive team is also effective at this point probably now we're now executives

know that the system is down and okay so the betterment set I've done this exercise before and one of the better minutes that came out of this is to have your domain admins make another account in another Active Directory domain that's higher than your Active Directory domain have those accounts be able to log in because these domain admins cannot log in their accounts are all locked you can have let's see you can have domain admins and another and another sorry in another domain login to the VPN and you can use an account lockout analyzer and see where the see where who what is triggering this account lockout okay time okay scenario two all of a sudden

you're fishing at company name com alias goes bonkers the pager is pinging like crazy hundreds of employees are forwarding this same exact email to the incident response team ir does initial analysis and determines the email contains a malicious payload what would you do next [Music] email everybody not to open it work with the email admins to do what identify who received the emails how would you do that who would you be doing that yep yeah so you're working with your email admin team too would you also ask the email admin team to delete the emails from you can ask them to purge the emails from everyone's inbox what what about the people that have already

clicked on it how do you detect that logs right logs so if you don't have your firewall logs or whatever log it is that's capturing this the event where the person clicks on this thing if you don't have that you would have no idea so assuming your company let's say has thousands of people 100 of them have forwarded this email to you what about the rest that didn't forward it so but you block the URL in the firewall yep what about people at home on their phones right you have no visibility into that what else would happen let's see do you have someone on your team analyzing the malicious payload to see what it's trying to do right let's say it's a

credential harvesting email what would you do force a password reset on the people that have clicked on it that you have we were able to detect because of the firewall logs right but then there's also the chance like you like we talked about where the person is at their house on their phone or somewhere else and you don't know if they've clicked on it so it is probably a good idea to force a password reset against everyone's accounts yes yeah sure he said can you force a password reset on people's accounts who have read the email if you had a way to I'm sure it's possible I guess through email logs to determine if the person

opened the email or read it then sure that would minimize disruption yeah but imagine yep I'm not I've no let me little can you say that rephrase okay following and password uh-huh sure uh-huh sure hmm okay did everyone hear that

if we force everyone's Password Reset we have now created a denial-of-service it's not really hacking me it's yeah the account just no compromise because the attacker now has the password so what is your point there sorry okay as in like these people fall for fishes sure absolutely she said make sure these people fall for fishes and make sure that they're not doing anything suspicious like lateral movement yeah good so what so monitoring their activity hmm yes right so he said we should warn those users who we reset their passwords that if they use that same password somewhere else to also change that password someone had a question over here would it leverage a password reset push sure we've now we

have force reset everyone's password uh-huh whom who might be asking for a reset why would the why would the payload want the user to reset their password Oh to capture it maybe

uh-huh sure right so that okay yeah good point okay let me see did we go over a good chunk of this yeah all right and let me tell you if you've not gone through a company-wide password reset it sucks people are on vacation or on VPN it I mean it's just it's really difficult so you know you might actually that's actually a good betterment write that down form a process on how you would execute a company-wide password reset how would you do it because you don't want to figure that out on the spot yes right he said it probably would be a good idea to enforce or to force everyone to enroll in mfa if they aren't

already yeah so your company might already have MFA but which is good in this case but if not yeah absolutely we don't have MFA write that down get a new project going alright next scenario the CFO sends an email to someone in finance urgently requesting all employee information the employee sends the information then sends a chat to the CFO to let her know that the task has been completed hey Jenny I sent you the file she then realizes the CFO didn't actually send the email and notify his IT support what happens next this one sucks cuz it's like this person sent the email so at this point you have you possibly have so what would you do you

would someone in IT would probably look in the sent items and see look at the file that was sent to see if it contains PII or how bad it really is let's say yep yeah oh absolutely I get let's so assuming that this this attachment contains PII you now have a data breach due to excellent raishin so now the incident response process formally is declared and you have to include you have to know involve legal privacy corporate communications I mean you name it and all of these scenarios that I'm reading off have actually happened that I've read about or like someone's at someone's company that I know that kind of secretly told me and so these all have happened so it is

definitely possible it's not just an idea that you came up with okay so this next slide is in 3d so if you didn't get your 3d glasses at the no one go okay I actually don't have any more slides that is the end of the presentation so if you have any questions how much more time do we have okay does anyone have any questions yes

what do I do about you know I actually don't do a chain of custody so you'd have to talk to someone in law yes tools for analyzing analyzing malicious payloads there is hybrid analysis there's virustotal there's one called Joe's sandbox hybrid analysis is great yep can you repeat the question do I see sure for preparation absolutely because you want your team to be on top of what the current threats are so yes absolutely like for example a couple of months ago there was an email going around that said that said it was a malicious email not real but it said I have a bomb in your building if you do not send this much Bitcoin in 24 hours I

will detonate the bomb what do you do that's a good tabletop exercise right there what would you do who would you send that to so yes it's definitely helpful and preparation recovery probably short yeah absolutely see what the see what the public sector is doing about it yeah okay presentation is over thank you very much [Applause]