Incident Response: A Scalable Methodology for Cyber Security Teams

um so all right i'm telecom or chris you're here for besides denver 2020 socially distant our first speaker this morning is uh benjamin edelen and he's going to talk about incident response if i remember correctly yep all right i'm going to go ahead and mute out and take it away everybody enjoy all right so my name is benjamin edelen and i am the chief information security officer at the city of boulder and as a ciso i am not a subject matter expert at any topic because i have to do a whole lot of different kinds of stuff for uh a city which is an organization that has a whole bunch of different kinds of businesses going on at the same

time things like airports and police departments and public libraries and rec centers and every one of those types of businesses are deploying technology very quickly and so it's my responsibility to respond to incidents that we have in those technology areas and i i have some folks who are helping me with that but one of the most powerful skills i've found uh in any domain is the ability to to organize people into a work effort that that uh stops damage from happening and that restores your organization to functionality and during my time at the city i've been through a variety of major incidents um including major major fires uh floods the floods of 2013 uh and this

pandemic response and so i've learned by hanging out with and watching our emergency management folks in boulder has an office of emergency management over time i began to see that they had a methodology that they used for responding to what they call all hazards incidents so hazmat fire flood and then in other parts of the country they have things like hurricanes and that methodology was it seemed really cool to me but it didn't seem super accessible for whatever reason it was usually happening inside an emergency operations center there were big screens up on the tv all kinds of people were working at the same time and you didn't get a lot of face time with the emergency managers to kind of

pick their brain or understand what was happening at their scale so eventually i got into a position within my organization where i could go meet our head emergency manager and ask for help and and ask for some uh training about how this works and actually the federal government does provide what's called ics training um and that training would educate you about sort of this methodology but it doesn't uh it isn't really immediately adapted to cyber security and i was interested in responding to cyber security incidents on behalf of my organization of all kinds and i wanted that skill set so over time i was able to learn this the broad strokes of the ics methodology

and adapt them and you'll have to forgive me as i go through this presentation because this isn't a perfect adaptation of ics i am a minimalist i like to cut out things that are not working or unnecessary in most situations and so there are many elements of incident handling that you will need to inject back in if you especially if you have a bigger incident i think that it's important that cyber security people realize that even though we may be able to respond to a cyber security incident and gather a group of people and work a problem we are not good at things like figuring out where people are going to sleep or making sure that they stay the

appropriate distance from each other while they're working through an in-person work engagement during a pandemic or even just making sure that pizzas are coming everybody's getting fed everyone's getting hydrated with safe to drink water things like that so incident handling people have that stuff on lockdown which is part of why i wanted their skill set so over time i developed this hybrid or this this sort of variant on ics that is specifically targeted for cyber security and i've worked many incidents with it and one of the things that had me really encouraged was a few years ago when the state had an incident at the colorado department of transportation i know that ics was used

that a methodology similar to my own was used to respond to that incident and that actually allowed and scaled all the way up to the colorado national guard responding to that incident and that was a historic response to a cyber security issue and it gave me a lot of confidence that my methodology was going to be useful and scalable all the way up to like truly mega sized response where a military could even take over and respond in turn on behalf of my organization and yeah i was going to say i did i have run dozens and dozens of incidents and the vast majority of those incidents were not major um there are you know very minor

incidents but i do think this this gets to the heart of uh getting getting through those sands phases of recovery and capturing what you need to know about your incident and it's it's kind of elegant and like any cool tool uh once you go through an incident with me you you just sort of learn what you need to know to do it yourself and i thought that was really neat i've see i've gone through this incident with this incident process with people before and they've been able to come out with the ability to run incidents themselves which i thought was a really neat outcome and one that i wanted to share with besides denver and i've also used this

methodology in other parts of my life so i will say that my my wife got breast cancer at about this time last year and i used a very similar methodology to this uh to help get her through her chemotherapy and surgery so this methodology can be used for a variety of types of major incidents in your personal life as well as as in your professional life but the goal here is that i'm going to run through an incident and if you kind of pay attention to the tools and if you want copies of these tools they'll be available at the end you will have some tools that will let you jump in and kick off an incident in your

own organization and effectively manage it through the entire incident the tools are well enough designed that just by putting these up the incident begins to sort of self-organize and handle itself as long as you get some people assigned into the roles which is pretty awesome and you'll you'll see what i mean here in a minute but i i want everyone watching this to envision first of all this methodology came out of conference room incident handling it's not an international tool um you know doug brush could be here to tell us about international incident handling i tend to pull technical people into conference rooms and get the right people in there and kick off an incident and so the first

phase once i know that something is an incident the first thing that i do is i go get those giant white uh contact sheets that you can stick up on the wall i get a big sheet of those and i get a box of markers and that is the main tool that i use for incident handling i have a set of windows stickies here which is nothing like my cool uh sheets but imagine a conference room any work conference room you want to put those big contact sheets up on the wall north south east and west and every single person who walks into that conference room can read the contact sheets on each wall and see where we at in this incident all

the information you need is on this growing set of contact sheets that are on the wall and when i manage incidents over microsoft teams for my organization i use mural which is an application that lets multiple people jump on and edit the equivalent of those contact sheets i'm going to share my screen here and throw up mine oh ho screen sharing is disabled by the host all right i'll have to ask that the host hooked me up with the ability to share my screen

and i so i keep in in my mic you should be good to go okay i keep a template for these contact sheets oh which one is ah okay i keep these templates around because they they make this work so the first contact sheet that i put up is the instruction sheet and i'm gonna open that up here and i'm gonna stick it and you're just gonna have to imagine that i've i've grabbed a marker and i've written this on a white contact sheet or like i said a tool where the whole team is able to write on these sheets as well i'm going to throw these up and then i'm going to explain what each

one of them is about and why and how these work and then i'm going to run through an incident that is relatively real unfortunately but the names have been changed to protect the innocent

so imagine that you just got sucked into a conference room and there were there's four big white sticky sheets on the wall and these are what they say

the first sheet is the instructions for how we handle incidents and this list of instructions has grown over time and it's it is important for me and my process that everybody take a moment to read the instructions before they join an incident handling team so i'm sitting alone in a conference room trying to kick off an incident um and i'm writing this up here and then obviously things are confidential you know obviously we don't want people speaking at the same time but some of these down here came out of more unique experience we will often find team members will be hesitant to add things to the worksheets on the walls themselves it this it is critical in this process that

people are comfortable participating and and writing things up on the sheets we we when communications external communications are being written a lot of times the chat is too noisy um or the people talking are too noisy and so it's okay to tell everyone to quiet down and to stop side conversations um and then you build out your team and write down who they are you'll need this information later so it's important to write it down i publish the links uh this is a new addition to this for when i'm managing these things online because i think that um sometimes people will want something they can copy and paste to get back into the chat quickly without having to look

for it uh anyway let me kick off a demo incident and i will show you exactly how these function i'm going to try to move my little window out of the way so you can see the other side okay so imagine that i am hanging out eating my lunch in my office or at home since we're all at home and i get a phone call saying that our senior system administrator has just gotten a barrage of file change warnings on a uh like a windows file share and they're super scary so and suspecting a cryptoware attack on those systems that cis had been shut the servers down just sort of proactively that was the best thing he could think to do he

didn't want any more files to be encrypted and he shut the files down and now he's worried that he has a couple shutdown windows file shares that might have been affected by cryptoware and he is worried that um there may be cryptoware happening elsewhere in the organization so he calls his boss and says this is what i think happened and these are the actions that i've taken his boss calls me and i immediately kick off an incident so the first thing that i do is i go either go to a conference room or i start a team not necessarily a microsoft team but you know some kind of team chat situation and we pull in some initial folks to be our

our initial response team i put these contact sheets up on the wall and we build out the incident from the contact sheets so you know i get a resource manager someone who can get me sysadmins network planners um other folks like that i pull in a communications person someone who sends email on behalf of our department and i pull in a scribe so i grab you know maybe a service desk member or someone from my team and i asked them to take notes on the incident as we go along and to help me keep these contact sheets up to date i publish these contact sheets to the team and in a way where they're able to

update where the anyone on the team is able to update them and then we set meeting cadences so um i typically tend to set one and a half hour meeting cadences for serious incidents i think that gives us enough time to work and then come back with something something valuable and we start to log the incident that the team meetings and we log when we send out our status messages uh the communications person is responsible for writing up something saying what is down so here's the incident that we have right now first of all the information we have and it i put timestamp here i think this is important for meetings but because um this is a demo incident

i'll just use the current time 8 17. notification of

notification file changes let's say mass notification a file changes on file server zero one and file server zero one shutdown this is just so everybody knows i love to break up information and confirmed information into two different pages because we hear tons of rumors whoa that's weird because i hear a ton of rumors during an incident a lot of information comes in that is kind of garbage and i ask that everybody update the information section and then when something is confirmed we move it down to the confirmed section so this has been confirmed by our sysadmin so i'm going to move it down into the confirm section um and i tend to put this information sheet

up by a door that's near the that's nearest to the service desk team and that lets the service desk team come in and write what what is down and so now that the file servers are down a service desk person might come in and say i think gis is down gis is like a geospatial platform

so now we're worried that gis is down we don't know why the file servers are down maybe it's a prerequisite maybe the file server is a prerequisite for the gis system um and next up we need to start listing the things that are down so we know file server zero one is down and that serves the a b [Music] and c drives so if those drives are down department drives or things like that obviously those files are inaccessible by the users and that is a serious issue and we begin now that we know what's down and what are the circumstances of this incident i begin pulling people in and we start working on the incident

so i pull in a resource manager and ask for sysadmins and we ask the systems to get to work tasks like um and i'm going to ignore time due here um i get a system named bill for example and i start and i say bill i need you to secure the backups this could be a cryptoware incident and the backups are lifeline go secure the backups and i have another system named ted and i say ted we need to take a look at the cpu of all of the systems in our environment that are centrally managed and we need to find for example if there are systems that have elevated cpu maybe there's a way that we can detect

uh which systems are which other systems are encrypting files so ted is going to do that

and then jim um let's have jim i had a good idea here for jim i'm just typically when you're running in this you have a team of people coming up with ideas right you've brought them together they're all talking in the context of these sheets that you've put up um and in a cryptoware incident you know people very quickly move on to what else could be wrong so we might have jim take a look at the active directory servers and see if there's been abuse of our users or issues like that

and now there's a lull i've given some assignments out we're probably getting more information from the service desk coming in about which systems are down who's frustrated that these drive letters are no longer accessible and we need to send something out to the organization saying folks were experiencing an issue and the issue has resulted in this kind of outage and here's what you can expect from us and so we send out a system status message so our communications person um and by the way i make these assignments these are automatic assignments so i i bring in people uh the communications person determines the communication cadence and schedules that those communications and i schedule a team meeting

and you send out actual meeting appointments for these things so our communications person sends out a system status message saying that drives a b and c are down we apologize for the inconvenience and we're working on this issue and you can expect another communication from us in one and a half hours at this time which will be you know whatever um 10 o'clock basically and i think it's important when you do those communications that you are fulfilling on your promises so we i try to set an expectation with communications people that we don't blow it just like i don't blow it when i do my meeting we have we make it happen we meet on

time if people blow off the meeting to get their tasks done that doesn't work so now uh bill i'm gonna have to i guess i'm not gonna look it up there is no good way to do no there is a good way to do straight through bill gets back to me and says the backup servers are bulletproof they are hardened such that you need a specific secure workstation to get on them those workstations have not been accessed the backup systems have not been accessed they're off site there's nothing that's going to get to our backups and that's obviously something you have to plan for ahead of time so we're going to strike the backups off of

the system or i mean off of the assignments list on ted he went out found no computers with a high cpu load and so we're going to mark that off as well oh i'm sorry i'm not going to do that yet jim goes to a.d and jim gets back something very strange so jim comes back and says why is that bold

tons of ad user activity and we immediately jumped to the conclusion that perhaps a bunch of our ad users are being affected and i said no cpu but i'm going to go with a little different tack here so let's say we also find out from ted that

we have elevated cpu on several desktops

and i'm going to mark ted's job off because he did it so now we're looking at a report from active directory that shows a ton of user activity and i wish i could pull this report and example this report up um but it's showing almost every user in the organization has a bunch of file change activity going on and everybody within the incident team is losing their minds it's really scary to see that kind of thing because if your a.d is compromised you are toast as an organization and even the best incident handling crew is going to struggle uh with a with a pwned a.d to get something back for your organization and that's where

something that i've learned over many many many incidents comes into play you really have to read those reports carefully so we go back and read through these reports and we do not move things down into confirmed information until they're confirmed so i we tasked somebody with really looking at this elevated cpu and really looking at this tons of ad user activity and what we find out and this is just a demo incident we find out that this report was run really fast and somebody pulled in way too broad of a scope so maybe the last 10 days instead of the last two hours for this report and so of course almost every user in the organization has

changed some files over the last 10 days but almost no one has changed any files over the last two hours so this is this turns out to be a false positive and i i can't stress enough that i have spent and wasted more time in incidents running down false positive stuff by treating information as correct and when you work an incident within your own group you really really really need to be circumspect of false positive or of the kinds of information that come in because if they turn out to be a false positive you're going to waste a ton of time on them i've spent more time working on false positive stuff than i have

working on the actual real issues which is ludicrous but it totally happens this on the other hand is real we're obviously worried about this um and these elevated cpus we double checked a different way we jumped on one of them and sure sure enough uh this cpu is elevated because that system is crapping out binary files and text files that contain ransom notes so this thing is true we're going to move it down to the confirmed information and maybe i should do this for this one that's false we're going to strike it we don't care that about the ad activity so now

going back to bill

and remember this is i mean we'd have hours and days to work through uh identification and containment but i'm i'm just trying to get through the whole process of this incident for folks today so you know keep in mind we're not going to do root cause analysis here or anything like that um but just imagine in broad strokes if you were running an incident and you were being traumatized by um having your file shares down having your users be upset uh you would you would be doing a whole constellation of more different team meetings and conversations and things like that but we're we're moving through this pretty fast just to get just to get everybody a glimpse of the

process uh and and why these four contact sheets are essential so we are now we've tasked bill with getting a list of pwn computers and ted what we don't want to do is get the list and sit on it in case it's spreading so oh i'm sorry i s sean move his webcam window to see sheet four oh okay you guys can't see sheet four or did i do it

i'm not sure give me one uh tell me what's up sean just the top right what if i move this down is that better this is fun i can't see my own stream so okay perfect thank you okay yeah sorry about that the um if if folks weren't able to see this these are this is my sort of template assignments and then everything below these are the assignments that i've been making for for people working the team okay i didn't so bill's going to get the list of phone computers um whether it's by cpu or by doing you know a powershell script that fi searches for the ransom notes or other things like that ted's gonna start shutting down phone

computers uh and we're gonna shut systems down until we stop seeing this activity and your you know if your mind goes straight to well other computers could be pwned but not generating you know cryptoware notices that's absolutely true um i'm i'm not saying that you'll have achieved containment through this but it is just for the purposes of this of working through this abbreviated engagement that we're doing here today so we assume for this engagement that as ted works through shutting down all these computers we're going to be in good shape and we know that our active directory is probably safe and so we want to start working on this file server and we sit down and we think about it do

we want to bring it up without a network interface maybe so we can see what's on it do we want to just blow it away and rebuild it um what is the plan for this file server so that we can get it back in the hands of the users so ultimately we decide that we want to rebuild it from backups i would say that you can get a little ahead of yourself during incident handling by trying to get into a recovery phase before you're done with the containment phase but for the purposes of this you know we're going to ask jim to rebuild fileserve01 from backups bill starts to knock out the things that are pwned

uh oh i'm gonna have to shrink this a little bit so we get word back from bill and ted that they are done shutting down phone computers we want them to wait an hour you know and make sure that no new uh high cpu events happen or things that we can use to detect those uh crypto weird computers in the meanwhile jim is cranking out a new file share we send out another system status this is the system status saying we have the issue contained we really appreciate your patience uh shares a b and c are going to be back available within whatever time frame whatever we feel like jim can get done with his rebuild

task and so jim finishes up that rebuild and fileserver01 returns to being online and so we're going to strike it from the down list

and then we're going to strike jim's assignment we're going to send a final system status saying that we've restored so at this point we were able to identify the incident pull in people create a process for working it together as a team pull in information confirm that information to make the choices that we need to make within the environment track which systems were up and down track what we all agreed we were going to do to move through the incident avoid false positives and we sent out high quality user communication throughout the whole process i submit that that is good enough for incident handling if you can run an incident like that you are doing a good enough job

at facilitating knocking out a major situation within your organization and these are really scary you know when you sit down and it's your systems that are damaged these are really scary so we finally are able to send the system status now final system status saying the issue is contained and the file server the file shares a b and c have been restored to everybody's pcs and we hope you have a great day there is uh one last step that i always do in every single incident which i leave on the contact sheet so that i can remember to do it which is to facilitate a detailed incident debrief and publish the lessons learned and the

fun part about facilitating an incident debrief if you've done this in a conference room with white contact sheets is that you can pull the team of people that you listed under team back together oh i'm sorry i should have done this

so i've got these guys listed i didn't want to lose them you can pull everyone back together and you can relive the incident together because you have the sheets and you have the record of how things went down so you can go through them by time stamp and say this is what happened here was everyone okay with how we handled it was there too much table talk and someone might come back and say yeah my the communications person might have said look everybody was talking at the same time and it was really hard for me to have the chance to write down uh the system status message and get it out and there were a ton of typos in it or

something because i was i was i didn't have the environment that i needed and that gives an incident handler like me valuable information that i can use to tune this process and add more instructions or manage the table talk better during the incident or things like that and i just want to draw everybody's attention to some of the details of the way these contact sheets lay all of this really really plain um so you can always you if you are handling an incident for long enough you have to go to bed you absolutely have to say to another human being i'm handing this off to you and i need you to work it and the cool thing about this process is

that i even though i'm the only person who really does incident handling work within the organization anyone you hand it off to will be able to look at these sheets and understand what am i doing i'm asking these people to get through these tasks that we've assigned i'm keeping up to date the list of things that are down and the information that comes in that's all we're asking for the 12-hour shift that you're doing you don't even have to be an expert at you know the sans phases or anything like that because anything that doesn't get done while you have assigned someone else to be the the incident handler you will see on the contact sheets isn't

done and when you get the incident back you can finish out those items so that's what makes this scalable is that you can hand this incident off for example you can call your cyber security insurance company and if they fly someone out to help you they that person will recognize exactly what's going on when they look at these sheets if you work for a city or a county and you ask for an emergency manager to come and help you the emergency manager will look at this and this will be very familiar to them and they will know exactly what's going on and that emergency manager for you or uh some that kind of professional will be able to

coordinate where are people gonna sleep how are you gonna get water because you keep in mind this is this kind of incident handling is um doesn't do a great job the way it's designed here of getting people drinking water or a place to sleep and that is some those things come up very very quickly if you're handling an incident 24 7 for three days or for six weeks or for something like that um so you need to be prepared to scale this up but it scales up so easily as you grow the information that you have available you put more contact sheets on the wall if the contact sheets are not appropriate you move it into an electronic format

but the fundamental structure behind it no matter how large the information set is no matter how long the term of the engagement is this is still that fundamental set of information that you have to have to work that incident front to back and so finally after you've worked through the whole incident and you've brought all your stuff back oh man here's doug finally after you worked through the whole range of issues um you want to publish some kind of lessons learned and obviously when i say publish i don't mean publish to the press or anything like that um although your communications person should be able to work together with a press liaison or things like that if that's necessary

that's absolutely built into this process um and it's it can be very important to manage the press if depending on what kind of organization you are but what i mean is publishing some kind of knowledge about the incident and a basic a basic expectation that i would have for publishing information coming back out of an incident is way beyond just the nature of the incident you have to have a full root cause analysis done before you're done with the incident before you're publishing you have to know what what hardening would have needed to be done within your organization in order to prevent this kind of incident in the future and you want to be able to say

that that hardening has been done you don't want to just publish that it's a good idea you want to say you were able to get it accomplished but most importantly for some of my purposes you want to be able to say the kinds of damage that the incident caused and so you want to keep track of how much money did you spend how much staff time did you spend how much um vendor time not necessarily their staff time but but um keep track of some some details about your vendors the the people that you worked with you know if you called mandy and tim to help you out with something or if microsoft's was uh engineers were

on the phone with you for 30 hours you want to you want to keep track of some of those materials and details uh let's see okay time money reputation damage that's not quantifiable typically actually there's probably somebody out there who has a cool way to quantify reputation damage but i don't know how to quantify reputation damage um so that's a little bit of a qualified measurement but if you keep track of your incidents and you keep track of how much they cost you how much staff time they use and and what happened to the reputation of your organization as a result over time you can show the effectiveness of your cyber security program and of

your incident handling techniques um because you're able to say at the end of the year here was how much we suffered um based on this kind of investment and so you can always tie that back to the investment that's being made in cyber security or back to suggestions that you have about how to do security well so if you're a security leader you want these metrics coming back out of your incident handling process and i assure you that if you don't collect them during the lessons learned phase they'll be lost and for city and county folks out there obviously you could be reimbursed for some of these things under certain circumstances if only by your cyber security insurance

company but sometimes even at the federal level you can be reimbursed for these kinds of incidents um so that is an overview of of how this process works and i will say one more time that i have worked this process dozens and dozens and dozens of times i mean i i literally had one where it was like the doctors and from like i said i used this for my wife i had like all the doctor contact information under my team and i had the appointments that we were going to do um and i had the medications and their schedules and you know that process it was that modified process actually worked extremely well for for my family as well

so i will make these templates available to anybody who's interested in them and i do have a few other fun incident handling uh materials that i will make available um but i will say that there are people like doug brush who have traveled all over the world you know who've been dropped into unique places and situations and worked incidents uh with nothing except like a dull spoon i mean some of those people can tell you more about some things you may need to know if you're interested in pursuing a career as an incident handler um for example you really may need to know where like how to pack a kit or things like that there may be critical things that you

need to have in a backpack if you're gonna like go somewhere and work an incident and i'm not an expert at that i am a defensive security person so i work incidents at my own organization for my own organization usually the team of people i'm working with is my people or people we've brought in as vendors so keep in mind that you'll have to heavily modify this process if you want to use it at your next engagement in a foreign country or things like that but i think no matter where you go you may well have uh you may well find that these materials are perfectly adequate for working through almost any kind of incident

or disaster that crops up within your environment so i'm gonna stop sharing i will get those materials put together and i am done

thank you very much benjamin that was that was fantastic um admittedly i was listening to it on three different audio streams slightly different offsets but it sounded really interesting um so if you want to move over to the discord i shall um and you can do i believe there are ways to do voice and video in there if you want to but if you just want to do text that's fine as well all right thank you very much for your time everybody have a great day thank you for volunteering to contribute today it was that means a lot to have i mean especially you know we're besides denver uh you're you know a brother city just

up the road considered part of the denver metro front range more or less as much as boulder doesn't like to admit that sometimes oh we totally are and i will say this our cyber security community that we have along the whole corridor of the front range is very tight through groups like colorado equals security and and this b-sides group you know it is um it's a huge honor to be a part of the all of the different people working shoulder to shoulder against the adversaries um to protect the front range and i do think we have one of the strongest cyber security communities in the world for that reason that's that's really good to hear um

i've actually been involved in the community for a while and i can remember a time where um there was a there was there was and is a group called the 303 it's mostly just a a drinking group with a hacker problem um and there was i would run into people like defcon for instance um who would who were you know full on full into security very knowledgeable very smart very you know relatively speaking for hackers social people who would say oh yeah we've heard of three or three we don't like those guys i would never hear from them again um so it's improved a lot over the years yeah i i was not i don't know that i've

been around long enough to see that i think uh like i said we have such great leadership um from the state level and cyber security uh brings a lot of diversity together into our area and then colorado equal security is unmatched in my opinion around major metropolitan areas for bringing our cyber security leaders and talent together to share all kinds of things so i'm proud to be here i'm going to jump over on the discord thank you everyone have a great day you too bye