AJ Leece: Workshop: Incident Response Tabletop

[Music]

[Music] we got enough to to roll with let's uh let's get started so good afternoon everybody happy first day of b-sides uh it is always so wonderful when this day comes i i do assist with some of the ctf development in fact one of the puzzles that you're probably interacting with is one that i built purely on my own so that one actually has some automated twitter functionality so if somebody is successful in breaching the business uh to some extent there may be some uh notifications going out publicly uh so today we're going to spend some time on what happens after the business gets breached so how does an organization clean up a breach uh and

then what we're going to do is we're going to play a game so this is a game that i built uh and the game is designed to uh teach you some incident response so whether you've had any time in incident response or not totally irrelevant i walk you through everything uh and we're just here to have a good time learn a few things uh and fortunately not actually lose out on the problems uh that we're about to face so uh we'll just cover a bit of uh incident response basics so that we're all playing from the same sheet of music i'll give you some insight into what it is that we're going to be spending our time doing today

and then we're going to play and we'll open it up to some q a at the tail end of it and we'll go from there so a little about me i've been in the business about 10-ish years uh this is my fourth b-sides calgary that uh fourth or fifth uh the years have all started to run together uh anyway i've been around since uh part of the beginning there uh and it's been quite a ride ever since so for whatever reason i'm very interested in incident response when i started my security career this wasn't something very appealing because you tend not to have a regular schedule it tends to be very panic inducing very high stress especially

when you are missing a lot of the core pieces that go into resolving an incident and the managers are going why don't we have this oh because i told you eight months ago this needed to be here and you said you'd rather buy new printers very common problem very uh defeating and unfortunate but nevertheless organizations that do lean into your information and all of your efforts do tend to benefit well from them when you show up to respond to an incident so uh i'm hoping today by the end of this you will have some new tools in your tool belt uh to help guide you in this journey if you are just getting started in incident response considering a

career into it uh feel free to reach out and let me know anytime uh you're interested and we can certainly chat offline anytime so there's two different definitions that we need to keep in the back of our heads as we go through it so an event in this case is simply an observable occurrence that is relevant to information security it's all it is it's just something that happens that we find relevant we don't really know whether or not if the intent behind it is good bad or ugly so there does usually need to be some follow-up actions some more investigation some more effort going into it because an event simply on its own without context really doesn't offer a

ton of value in the grand scheme of things but the other side of it is events are necessary for us as far as an incident response capacity goes uh without these events we can't really correlate what's happened inside of our environment to anything meaningful and all of this information does pose some some value in the incident response process these events can simply come from all different kinds of places you might get phone calls from your service desk saying hey we have reports of staff phoning in with this issue you might get some kind of event monitoring tool if you have like infrastructure monitoring or security monitoring you might actually get a notification that says hey we saw

something and it was evil so please act accordingly uh or you may actually experience some kind of outage uh so whether you are using a website for the business and it goes down suddenly uh or you can't reach something uh or for whatever reason the data that's showing up isn't the way it's supposed to there can be a whole series of events that go on around that so you don't want to discount one just because it might not be relevant it all tells a big part of the story so an incident uh is simply a security event that results in damage or risk to information security so when we talk about events we don't really know

the intent or anything behind them that is the case here we actually do know now for sure that we have conclusive proof there is some kind of uh incident in the works we do have somebody who has run a bunch of events that is uh designed to to elicit some harm you know break something to do with the information security of the network the secret ingredient here is crime that's the the long and short of it uh basically somebody is pursuing this criminal act on for whatever means and whatever measures they want to take with it but you're the unwilling victim unfortunately so let's talk incident versus event so we mentioned before that an event is

simply just some information of uh relevancy right so we have a failed logon attempt do we have an incident right yet or do we have uh just an event for the time being i'll put it out to the chat there

[Music] yeah an event absolutely because it's just simply somebody failed to log in right i fat finger my password all the time uh you know every three months we always run into this problem what about a malware outbreak

yeah yep we're 100 in incident territory at this point malware is malicious software that's just the portmanteau for it so really any unwanted or unwelcome software is usually another situation uh many organizations will have potentially unwanted programs they might not necessarily be malware but they might create an excessive permissions situation where you can't uh you know keep control of your data as well so unnecessary software may not necessarily elicit a an actual incident in and of itself but anything that is like confirmed malware is 100 yeah we're in incident territory what about an antivirus service that's been stopped on a workstation are we in incident territory or events still yeah event absolutely because uh various infrastructure management

will need to adjust antivirus tools they may need to stop them in order to install certain components they might need to uh pause them to introduce some bypasses that are supposed to be there so very real common problem what about a distributed denial of service attack that's pretty much in the name right there we have an actual incident on the way basically somebody is taking a bunch of computers and launching them in our direction yeah we absolutely have an incident now what we can and can't do about it that's open to interpretation uh sometimes your isp can get involved most of the time they just want to take a nap so at the end of the day you might be on

your own what about a large upload to an unknown location this is the last one i promise yeah simply an event right people back up stuff all the time right so whether or not we have uh an incident in the works uh we're not really sure so when we're talking incident response we we have a few very specific outcomes from each of the steps uh and very simply the main one is to get the business back online so chances are when you're in an incident response scenario uh you have a situation where you can't uh perform some layer of the business right whether the websites are down whether a database is inaccessible whether some part of the business is

unavailable for standard business activities you your business is offline so you have to be able to get it back online that's the big ultimate goal from incident response but there's a few subcategories that we follow here so our first step is identifying compromised assets we need to scope the size of the problem we need to understand all of the affected systems how they might interact and what they might be affected with before we can really move to the next one which is containing the spread so when we have identified and cataloged all of our assets that are compromised we want to be able to migrate them or otherwise isolate them from the network so that the damage doesn't continue to

spread so if we're talking something like a malware outbreak for instance we might go ahead and disconnect them from the network we might remove them take away some permissions physically move them whatever that looks like at that point oh the sorry the eradication needs to go here i keep meaning to change this and i for whatever reason can't get around to it um we want to eradicate the threat first before we recover the asset uh don't don't take that as the the steps in in order there eradicating the threat is simply we find all the attacker methods of persistence and we close them off so whatever those might be get rid of them we don't need them

now once we've eradicated the threat now we can recover the asset so recovering the asset is simply bringing the data back online into the systems back into the fold so if we're talking something like a malware outbreak we might reinstall any operating system tools reinstall any libraries bring anything back onto the system itself that is relevant and useful keeping in mind that we have to do a few steps after this recovery piece before we can close down the incident once we're there we want to minimize whatever fallout we can if it means getting involved with our insurance companies that's sort of an overarching thing we want to make sure that we're always in touch with them if they're

responsible to something in some capacity if we have legal for instance they might want a lot of information on the tail end of the incident they may work with you in between depending on if you are engaging law enforcement you might need to provide some additional evidence or some additional insight depending on where they come in the the pathway but the most important thing is we learn something from it what are the lessons learned from this that we can take into the business make some business decisions and uh carry on from there so our first big step and this is a constant one that that sort of happens before the incident is usually the best

time for it sometimes you'll find uh preparation is lacking in some incident response that's a very real problem that every organization faces but you won't rise to the occasion you'll fall to your lowest level of preparation so at this point uh if you if your teams aren't well prepared and they aren't well established in this business uh and within your own business to respond then you're going to have a bad time when it comes to responding to an incident uh the preparation is important so you want to keep your contact info updated so depending on your incident response process your plans your notification requirements you may actually have to get somebody on the phone at a really

uncomfortable time you don't want to have to be calling around a bunch of numbers that may or may not be the correct one for the person so phone trees are a good step for this make sure they're up to date check that usually once every quarter if people have a habit of changing their phone numbers often if they don't you can extend that out to six months but make sure that that update is or those updates go in accordingly when we talk out of band communications uh this is simply in the event that our email is compromised as in the attackers in our email what happens if the attacker has an account that has access

to slack or teams or whatever that looks like what are we going to do with that so you want to make sure you have a way to secure your communications during an incident until you can confirm that your traditional ones that you're using in the business are well and truly secured so uh out-of-band cell phones are always good individual personal cell phones i worked in a spot where they had what was called the bat phones where the they were just flip phones that were sitting around in case of emergencies so in the event the voip system went down we we had a plan in place to continue having discussions uh forensics media if you are fortunate

enough to have somebody internally who can run your forensics for an incident you want to make sure that any collection media that you're using to gather the evidence is sanitized ready to go and available so that goes into the next one your tools here if you are uh collecting this evidence and it is going to law enforcement or you are collecting it for an action internally as in we're going to you know let this person go as a result of this breach that they created themselves kind of deal whatever that is you want to make sure that the evidence gathering process and the evidence gathering tools as well as all the buckets that it's going into

are clean and available so you want to test that before the incident happens because figuring out that you have old evidence in the middle of collection probably not a good time uh when we talk documentation this is simply your like your standard operating procedures your your processes your procedures for getting in touch with different parts of the business notifying customers any of that documentation that's relevant to the business and by extension the incident you want to make sure that somewhere because it's never a good time to have an incident if sharepoint goes down as part of the incident and you're relying on the documentation that lives in sharepoint so you want to make sure that all of that information is readily

available in two places if you can some organizations will keep offline copies although that's incredibly difficult given our remote situation in most cases that's likely to stay but having it in two separate spots that you can access that are somewhat separated in a way that one being compromised won't affect the other you'll have to keep documentation up to date in both spots but fortunately you can just update in one and push it over to the other and away you go and finally a clean shirt and make sure you're well rested incidents take a long time uh in in a for the most part when you're dealing with incidents they are long and you're usually stuck in a

boardroom and the smell of stale pizza and stale people gets a bit old pretty quick so yeah definitely make sure you have a clean shirt and you've had some sleep uh the collection of evidence yeah grant we're we're gonna get to that um so the the process of identification here we can look at many different sources we have log data uh we have other kind of uh event information uh again the people who actually sent in the original request of this is broken or it isn't working that can be a really solid source of identification they might be the first line of identification that you have so it does come from many sources but once

we've declared an incident we proceed doing the following knowing what we know about our various layers of compromise we want to take those little bits so we know so for instance if we're looking at something like a ransomware outbreak and we have a desktop that has this confirmed ransomware we might want to know what that activity looks like on that desktop so part of the investigation might be very carefully going into that desktop looking through and seeing if we can find any kind of files that might be dropped if it's dropped in a specific pathway naming conventions file activity event activity we want to look for all of those as we identify assets that are

compromised we want to be able to catalog them in a way that we can quickly reference one of the things that you have to do during the containment phase is go through and actually contain all of the assets that you found that were having problems so once you've done so then in that case you can move on to the further steps so make sure these are all catalog nice and easy excel spreadsheets are available all over the place google sheets are another example of it just make sure that your permissions are well locked down and everything is kept away from unauthorized use from there one of the things you can do if you have a series of indicators of

compromise but you want to be able to to pick up if anything new shows up uh various monitoring tools go into this to help you with that so one idea uh that consistently gets a lot of uh praise in sort of a an incident response circle is this you you configure your firewall to sink whole known bad traffic and alert whenever it experiences a connection outbound to the known c2 address so if we're talking command and control uh systems in that case when we see a an asset reaching out to a malicious url we want to make sure that that gets put into our firewall so that a it can't talk out anything any further and grab

new updated information from the attacker and b cataloging that goes a long way to figuring out just how big the spread is so usually once you put in these blocks the attempts to bounce back off of it can be you know pretty easily caught from there anything else that comes up if you find any new files any new registry entries any new traffic just go ahead and include those in your various layers of monitoring and your notifications because you want to keep a note of all the observations as you go along lots of books related to wine yeah we uh we're wine connoisseurs around here that's for sure so once we've gone ahead and we've

identified all of the the affected assets we want to start the containment process so our best way to do that is just simply removing them from the network whatever that looks like so if we're talking about windows systems uh depending on the resources being used we can just simply remove them from the domain uh and they're no longer accessible but that doesn't quite contain the whole spread so if we're just removing from the domain we might want to do some additional network layer stuff like vlan segmentation or amending our firewall acl list so when i mentioned creating some configuration that would allow your systems to tell you if any new communications were happening outbound your acl list is where that

would happen that's a good place to put it so keep those in mind keep your firewall admins on speed dial whenever there's an incident in the works if you have a physical device simply remove it from the network if you can just walk over to the desk and pull the cord that is a really exceptional way to do that works pretty well as long as wi-fi isn't still enabled if wi-fi is still enabled on the system then yeah you're going to have a bad time so make sure if you are physically removing the ethernet cable that the wi-fi doesn't immediately kick into the corporate wi-fi as well so should we power off the affected assets if we can avoid it no um but i

say if we can avoid it there will be some times where a user might panic and simply power it down and uh and carry on that is a very real possibility there may be some other situations where we have to power down the device because we have to ship it that's a very common problem too for whatever reason if we have to power it down then we have to but if we can avoid it under any circumstance uh we shouldn't because that volatile data lives in ram so if there's any kind of suspicious connections going outbound uh any kind of weird process activity all of that is going to be well contained within the volatile memory if we can

preserve that for forensic analysis and evidence gathering that becomes very helpful so once we've gone ahead and identified the entire problem and contain the spread we can now start this eradication problem so one of the things i mentioned about uh the the containment and eradication efforts uh was removing all the attacker methods of persistence depending on what your attacker has done they might be a collection of a few things but at the very least you're going to want to look for infected files so when you were looking through the systems you want to make sure that you're aware of any files get dropped by the processes and anything else included with it because those binaries might create

additional connections that you might not be aware of and then the reinfection process happens as soon as everything comes back online uh any network backdoors if somebody opened netcat and started talking outbound to their c2 server you're going to want to close that down and make sure that it can't communicate outbound to those so be sure to keep an eye on those if you're dealing with something like an insider or a compromised account in general those accounts should have been disabled sometime in the containment phase now this is the time to change them so sometimes this can happen in parallel in the the scope of business you might go ahead and remove an account from the

environment you might disable it reset the password hoping that the user gets there first before the attacker does in most cases you can simply phone them and say i'm about to reset your password please immediately log in set a new one set it nice and strong no don't tell me what it is and then carry on so um you may go through the process of changing these passwords while you're containing the incident that's totally fine um any uh authorized sorry unauthorized scheduled tasks or cron jobs so uh for anybody who hasn't done a lot of sysadmin work the the the scheduled tasks and the cron jobs are basically just automated tools that system administrators have available to them at

large the really nice thing about it is they can streamline a lot of basic uh sys admin work the downside to it is attackers if they have the right layers of credentials can do the same things you can so if you're an administrator creating scheduled tasks so can the attacker so you want to have if you can beforehand in the preparation phase have a list of these hanging around if you can so that you know what good bad or ugly looks like if you don't have those sometimes you can take a look at where their located attackers might not go so far as to put them in the right files they might not be so wise to put

descriptions behind it any scheduled tasks or cron jobs in a linux environment that just don't smell right you can usually disable them and wait for the screen test keeping in mind that you might impact the business so it's it's a balancing act so we've gone ahead and identified all the problems and we've contained everything as best we can or as completely as possible now we've gone ahead and eradicated everything we're fairly confident that the attacker is no longer persistent in our network great let's go through the process of recovering the assets so in this case we're going to be restoring our affected systems which is essentially bringing them back online reintroducing the data whatever it is we

have to do so any applications that need to go back into business those get installed updated whatever need be that data gets loaded back into it once that data is restored successfully the data ownership has to confirm functionality so we talk two things you see data ownership and data custodianship data ownership is the person who's ultimately in charge of the data right so in something like a finance environment the payroll data might belong to the payroll department they are the data owners but the ones who actually grant access to the data custodians that could be somebody in the it department doing that work so just because you're granting the access doesn't mean you own the data

just because you own the data doesn't mean you're necessarily hands on keyboard provisioning the access once we've gone ahead and restored everything the business signs off says yeah all the data is exactly where it's supposed to be functionality is great looks good excellent go ahead bring it back online keep testing keep validating make sure that everything's working okay watch the traffic watch the environment make sure that there is no uh reinfection or reissue of uh malicious intent that's happening definitely spend some time analyzing the system activity before you go ahead and close it down that's where that additional testing piece comes in by the way consistently keeping an eye on the environment lessons learned so what did we learn

right uh most of the time business decisions will come as a result of an incident you'd be surprised how there's no budget for security then there's an incident and then suddenly money uh is raining down from the heavens so as unfortunate as that situation might be that's just the long and short of it so uh what did we learn because that is going to ultimately go into a business decision down the line so that post-mortem process a lot of meetings a lot of discussions a lot of conversations about what can we do to amend this next time we want to put together incident reports so most organizations will have a need for an incident report whether it's for

compliance whether it's for another audit situation down the line some of them just hang on to it for the sake of keeping record as to all the incidents that have happened regardless of your reasons keep those incident reports they do provide a lot of insight into what happened the actions that were taken and away we go i i like to state this i try to never advocate for dismissals so all of my consulting days i would be brought in i would have a lot of assessments and a lot of risk associated with it you know regardless of what happened i would never tell somebody that yeah hey okay this system has to go unless we

obviously had reason to suspect they were an insider threat which i'd never come across uh for the most part you know stuff's gonna happen right like accidents happen uh security mistakes happen i mean i was two weeks into an internship and i accidentally let a virus off the chain you know these these things happen so um i never really advocate for dismissal unless it's kind of one of those like okay well you really should have known better than to save your credentials in the dark web kind of deal but even then you know i i really try to never advocate for dismissal in the report itself uh i do like to present the facts of what happened so uh

at this time in this capacity this happened the who what when and where uh and i guess by how much if we're looking to force a 5w here um basically we're just there to present the facts and tell the story to anyone who wasn't there so most of the time managers might be aware there was an incident but they might not be directly involved totally common when we report this to our incident response or sorry when we conduct our incident reports we want to make sure that we have uh recommendations in them because again we're making business decisions as a result of this what are the recommendations to fix this problem next time that has to include an

ask from senior leadership are we short on resources to manage some of our systems let's find some money and hire some people uh do we need some security tooling to help us do this better at scale let's free up some capital and get this going most of the time it's about finding some money for resources that you didn't have so the ask usually needs to be clear has to be backed up by a strong business case and if possible align the ask to hey it would cost us a half a million bucks to this but it cost us a million bucks to clean up this incident suddenly the control doesn't seem so out of reach anymore

when we're forking out a million bucks in fines fees and overall costs never blame the people for process failure that's actually a really point adam i really like that uh you know process failures happen all the time processes are written by people though um so if we aren't testing processes if we aren't validating the processes in place um then you know we're we're in tough shape otherwise so yeah that's uh that's that's where we're at so i want to give everybody just a few minutes if you gotta run uh you know answer some emails uh go to the bathroom grab some coffee whatever that looks like uh let's do so yeah let's take ten we can carry

on we'll do it without the the camera everybody knows what i look like there isn't a whole lot there uh i promise you aren't missing anything uh okay so all right so let's uh let's talk about our company here uh that's in the pursuit of uh about to experience a breach so we represent flan crest enterprises uh yes that certainly is not mine um but i just found that whole idea hysterical and decided to roll with it so uh yeah thanks adam right i used to play in bands i know how this goes so uh we are the ever intrepid folks uh at uh flan crest enterprises uh we are an e-commerce based business here in

calgary alberta canada but obviously some of us will work remote that's totally uh acceptable and inbounds we're really cool about that we do sell an awful lot of products something to the tune of 13 million dollars annually from our e-commerce environment which is that's a lot of money that adds up so uh we are an enterprise uh albeit we might have a small team but we're still going to work well and it'll be good so we have an hourly revenue uh of fifteen hundred dollars per hour that's that's where we're at um so every hour that the business is online and awake 24 hours a day because it's e-commerce uh we're good for 1500 bucks coming in the

bank account we finished our first pci audit as in we successfully said yeah we're good go right ahead and attest that we're in bounds uh in january of 2021. so fairly recent uh but there's definitely some considerations here so there are some ground rules uh this is a simulation but this is real right these kind of situations do happen on a regular basis uh we wind up in this situation where people who have compliant footprints still do actually run into a situation where they might be out of compliance or they experience a breach so definitely don't block the simulation we're here to have a good time and learn and grow and develop and moreover just

you know really enjoy it so feel free to work together obviously i'll be putting you into some kind of a group dynamic here so standby for that uh but yeah most importantly have fun collaborate ask any questions are there any questions about the whole what we're about to do we're about to go play i'm going to show you the the game here in just a second but if there are any questions before we get to it i will happily answer those right now

excellent i'm going to take silence as golden alright flan crest team let's have an incident and for that there we go that's the one outrageous everybody can see this we are on teams tj oh good to see you yeah absolutely so this is a team-based approach right every time we talk uh incident response it's always team based uh even in small organizations you're still gonna be working with a litany of other people involved in this so um you are going to want to get used to working in teams the only two roles in this case that do not have a team focus are the ceo and the cso so they have some specific elements associated with

the incident that we have to to work through they are by and large responsible for some of the more strategic parts of it so our ceo is by and large responsible for managing some of the customers uh dealing with some of the service provider information whatever might be coming around of it there may be some element of insurance and compliance associated with it so we do have some of that as well as ceos having spent some time in a business of some capacity because nobody's ever born being in charge i guess depending on what part of the world you're in um but for the most part we come into a ceo role having spent some time in a business

part of how we spend that time is in the pursuit of building our network and growing our various staffing levels so we actually possess the ability as ceos to bring in some ad hoc bench strength we can actually just simply make a phone call to somebody who doesn't work uh in the space a gifted security professional will just simply arrive and present some information of use that may result in some of these power-ups i'll talk about those in a sec so our cso is a little bit more hands-on in this sense but again still very strategic so we are in a situation where our ceos are sorry our csos can do something called clearing a roadblock so when we

have an incident situation if there's a gap in some information or if there's some kind of situation that leads to a problem non-technical in nature as in a computer hasn't divided by zero and fallen down on itself if anything short of that if it's a people or a process situation something that another person can simply walk up and fix very quickly that's what the battering ram does another one that is very useful is this birds of a feather so csos do have a very vast security network so this might call down a more uh a different security professional who is from the the same kind of section as the ceo ability there but it takes less time

there's more of them csos usually spend more time recording more security professionals so they tend to have a wider network they can usually bring people in a little bit faster so same concept if you feel the incident installed or if i feel the incident installed i will go ahead and advise you to to click that accordingly and when i say click i mean i do the clicking because it's all running internally here so our department managers our department managers they possess again a little bit of strategy but a little bit of operational capacity that they need so uh they're often involved in some of the resource shifting to make it so that the incident can progress uh

so a good one here to keep in mind is this out-of-band communication strategy so remember how i talked about the out-of-band communication some of these things align uh the out-of-band communication strategy is effectively an ability that allows the rest of the team to carry on uh working the incident in exchange for taking themselves out of the game so one of the problems we have is if we don't communicate frequently in an incident we might get pulled into a call where we have to share what's going on we will by and large in most incident scenarios have to have phone calls we will have to work with different people we will have to engage them for you know

warm handovers hot handovers but we will still need to have some phone calls we want to limit the amount of phone calls because that definitely uh gets into the way and loses focus so out-of-band communication strategy you reserve the right to cancel an all-hands meeting if there hasn't been enough communication uh in exchange for taking yourself out of the game as a department manager similarly uh you know humans need to rest right we're not robots we have the capacity to program robots which is such a cool discipline i'm having so much fun doing that um but the the unfortunate nature of humans is the you know the spirit is willing but the flesh is weak

you as the department manager have the ability to go hey you know listen i just need a little bit more and then you can call it a day so when we get to turn number 12 that represents 12 in-game hours that's 12 hours that your incident has been rolling we don't have 24 by 7 coverage in flan crest part of our next step was to get that but for the short term we're just working with the team we have so when we get to 12 hours we have to take a rest everybody has to go to bed we have to just move the incident along but the business is still down this ability grants the opportunity for

new resources to just work it for a little longer and hopefully get to somewhere closer before we have to take a quote-unquote break now for what it's worth there won't be any more breaks until the end of the session which case you can uh take as long of a break as you need so when i mentioned that we have a pci compliance element to this i absolutely meant it uh so our unsung heroes that are kind of strange and wonderful i'm allowed to call them that i used to work in compliance it's okay we are very much about the paperwork in compliance unfortunately but it has a very specific requirement uh the other side of it is

you have to maintain some kind of communication with your compliance governing bodies in the event of an incident so for anybody in the pci space if you have a large-scale data breach one of the first phone calls might be to one of the major card brands and that usually has to come from somebody in your compliance program so uh you want to keep some good governance good compliance people involved most of the time the compliance people do have a role to play inside of an incident and that is absolutely the case here so uh when we're talking the compliance team we have a paper dragon's ability that is basically your interaction with your outside compliance environment so you do

want to make sure that you are as a compliance professional we're considering running that but it takes time we can't spam them so we have 15 turns before we can play this again so this is one of these abilities where the more you play it the more likely you are to achieve some kind of bell curve but unfortunately because of how long you can how long this rolls you you kind of want to hope for a better role up front rather than hoping that you can even it out the numbers but i'll leave it up to the compliance team to decide a lot of compliance professionals are also formerly technical they may have come up through operations they may have

started uh in some other capacity or another a lot of them have seen a command line and whether or not they actually enjoy it doesn't matter they do have some technical acumen uh it is incredibly valuable and in some cases uh in this case it's a bit like riding a bike you never forget but the longer it's been the higher the chances you'll break something so having that previous technical acumen inside of a compliance role is excellent it does grant you the compliance person the ability to play some containment roles i'll go through how that works in just a minute so my operations leads they're a little bit further down in into the weeds here so they are spending

a little bit more time hands on keyboard they might have some level of business acumen within the business itself and how it works uh they might have some client-facing interactions depending on the scott the size and scope of the business um in this case flan crest does have some pretty large scale enterprise clients that do leverage the e-commerce platform so you actually do have a bit of business to business as much as you have uh you know business to customer interactions here so the the operations leads they typically have a little bit more insight into the environment itself they understand the business a little bit better so you'll find their abilities kind of transcend a little bit of both

investigative actions here uh this purpose-built knowledge this is basically they've been coupled with the infrastructure for so long they know all the corners of it uh if somebody gets lost along the way quote unquote they actually reserve the ability to just go in and amend some of this if they need to so that one can be really helpful if you're trying to investigate in a large capacity uh similarly they have a customer interaction a lot of operations leads might actually work within a customer environment depending on how the situation is so for anybody from a managed services environment you might actually be that person who is working very tightly with the business itself as part of their uh

it strategy so you do actually provide some kind of customer interaction some kind of uh interference to some extent and last but certainly not least our uh very intrepid technical analyst team these ones uh this is a very wide scope this is just a way of putting a fine point on a wide scope uh this is basically anybody who you know your database submits your server admins your firewall admins you know network admins any of the admins in a business would fall under the technical analyst category here so they have a lot of really excellent technology interactions that they might need a lot of the hands-on keyboard work in an incident does come from the

technology based teams so you will find a lot of the activity maybe comes in this direction so for those of you on the technical analyst team yeah be ready to think through a couple of steps ahead um this ability is one that i will actually call out and i saw one name that popped out on here in particular if he's still on the call uh that one would work there's the guy right there mr david coombe he and i go back quite a ways uh he in this case will be the venerable gray genius so for what it's worth uh i built this ability uh in in recognition of my dad douglas uh who is doing a lot of the ctf work in

hacking 101 work uh yeah when it comes to technological abilities i have yet to find somebody who is even as capable as he is the guy is just seeing around corners all over the place so if you have a chance go pop by the ctf room tell me say hello thanks for the the contribution because he did an amazing job with that ctf and he does every year but every technology-based team member always has that one person uh you know gray-haired and uh their their skills are again arcane and mystical in this case dave uh is also a gray-haired somewhat spectacled gentleman from time to time uh with a very large background in forensics so with that

having been said your teams and by extension the whole of the game you now have in-house forensics exceedingly rare most organizations don't have the money it takes to bring that on but because dave's here and he has an amazing background in that space your whole team now has the ability to run forensics without having to outsource it to a non-player entity so that's awesome you don't need this forensic analysis work now it's already built in house in the hair color oh okay well dave's on the call so i'm going to keep everybody with the forensics anyway i can fill in the gaps so the way the game works is we have uh we have it's a turn-based system right so

each of these activities whether we're communicating with these various entities whether we're querying these security controls these power-ups show up on their own so it's not a big deal but the the these power-ups when they come i'll simply invoke them uh but any of these abilities constitutes one turn so if we were to go ahead and

having a lot of good conversation but it's taking a little bit to come to a consensus yeah i'm gonna go

we can see that going through and investigating everything single step takes a considerable amount of work is this better can you still hear me come on i have all the power now what's going on [Music] is that any better yeah okay all right i just got ahead of it okay i'm too fast i gotta slow down i think i gotta go to bed too um so the we can investigate each of these individually but as you can see we have an enormous amount of infrastructure and it's going to take a great deal of time to go through everything so maybe let's not investigate piecemeal because it can take a little while okay so we'll set this up again we'll leave

that back to back to blank there we go now every single team possesses the ability to communicate okay so the communication button here this is just simply a status update of whatever value to whomever so in the case of most incidents when we talk about uh communication usually it flows up because the various layers want to be aware of what's going on from the technical analyst team but any team can communicate at any time with no penalty it doesn't cost you a turn so you can just simply push this communicate button once within your turn whether it goes well or whether it goes poorly that's a random decision the only time that you're going to lose anything or suffer

any penalties is if we roll a one in which case then yes there might be some kind of penalty on the other side of it but it is a small chance because we have 20 options here anything uh greater than one and everybody gets to continue playing regardless of the outcomes however if we go too long without an incident communication from any team it doesn't matter which team any team in general if we go too long we get pulled into an all hands meeting where nobody gets to play for at least an hour maybe two depending on how long it's been so one of the things that's a good habit to get into through an incident work is

just making sure that you're communicating often whether that's inbound communications like teams or email if those aren't compromised or whether it's something like over the phone like a conference bridge hopefully with uh you know better mic fidelity last but not least is our security controls okay so you can access these security controls as the technology team by and large they're the ones who have the most interaction with it so we can go ahead and communicate with those anytime those can either return some very interesting information that'll help guide the incident in some direction or they might not work and we have to try again so it is a bit of a guessing game as to

whether or not we're going to get to where we need to go but the nature of it is we have these security controls they can provide insight they do have value although these are mostly detective controls at this point so all we're really doing is just looking in at log data so when the turn comes up uh i'll simply decide i will tell the the teams what we can and can't do i will guide you through everything i am i am google for all intents and purposes as well so if you're not sure about whether something is feasible uh or whether you can do something just simply pop it in the chat and ask uh it

is a team based decision making process so while i was going through this you know definitely i hope you saw some teams that stood out to you as something that you would want to have whatever that is right now please go ahead and just pop in the chat where you want to put yourself in these teams keeping in mind that the ceo and the cso are singular roles unless i have an enormous amount of people showing up uh in which case this can just be kind of a global kind of situation um so yes in the chat right now just go ahead and pop in the teams that you want and if i have to rebalance them i will

but anything that's missing i'll just go ahead and play that side of it so yeah pop that in right now

[Music] unless you're sending me direct messages i can't see those as easily

simon if you just want to observe go ahead and observe okay no worries vincent i have vincent as an operations lead okay sarah i have sarah as a technical analyst excellent how do we participate tj you uh you pick a team that you want to be on and we go from there thank you bruce our unsung compliance heroes anything technical let's put you on the on that roll as well okay is anybody else playing

so we can run with a small group for sure but the more people that want to play whether you've had any experience or not is irrelevant this is built for everybody i've run this successfully with departments that have never seen incident response and they come away having learned a whole ton all right simon i like that jaw jason jason can i put you in the department manager space yeah all right okay uh simon would you like to be my ceo the more the merrier that's right we're here to have fun it doesn't matter it i promise i promise there's nothing uh stick me in the executive role yeah okay sure adam adam you can be my cso

simon i'll put you in as the ceo now don't worry again yeah fun at all costs right that's the spirit tj um yeah at the end of the day the only way that we're ever going to learn and grow in this space is if we get to get our hands dirty with it this isn't a real business there is no real impact no matter what happens here we're just here to have a good time chances are whatever you encounter whether it's good bad or ugly all of the events that you're seeing here have all happened in real life with died in the wool incident response professionals i promise you uh we're just here to have a

good time let's learn let's enjoy okay so we have uh enough to run with and uh oh ben sure uh ben let's put you in [Music] yeah let's do there's a position with okay all right so okay we'll do that uh okay love a buddy tj's with you sarah you're not alone in this uh ben let's put you in the operations leads i need another i need another body in there okay excellent we all ready i'm gonna sue more already because the time it's gonna take for me to get the reddies in the chat everything's good to go all right so uh we are the intrepid team at flancrest enterprises it's first thing in the morning we've come in we've

seen a notification that says for some reason or another our systems are not functioning we're not sure what we just received a notification from our help desk that their customers have been calling in and complaining that they can't access the environment uh in whatever capacity it seems to be very intermittent very strange we're not really sure what's going on so i'm going to go to my technical team first my technical team because this is a wide discipline of knowledge uh this the one team to to rule them all you basically get two for one turns okay so you can use that as an opportunity to investigate a couple assets now i give you these two for free they're not

impacted i promise you so you don't have to waste time investigating your firewalls but in this case you get two actions for the price of one so in one single turn you can do a couple of queries you can use an ability or two or you can simply manually investigate something so one of the best things to do when we're not sure we have a problem but we're not quite sure uh we you know it's an all hands kind of situation our customers are not able to ring through any transactions we're running out of problems here to solve we're running out of hands to solve the security team gets involved so one of the first steps that i would always

recommend in this case wherever we can is if we have these controls available to us these detective controls maybe they can give us some insight so we know that ever since 8 a.m this morning the systems haven't been functional uh just for the sake of keeping everybody on the same page we'll say that it was some time around when we received the notifications that were in there so the timestamp versus the data that we have to look back in an incident scenario pretty limited so the incident manager who is me is going to recommend that we either query some of our detective controls uh or we can go ahead and try some of these other abilities but in this case because

we don't really know whether or not we are dealing with anything concrete yet it might be good to go ahead and run one of these as our first turn so i'll leave it up to tj and sarah to just think through it and chat through it if they want that you can certainly put that chat right in the window here if you'd like or you can direct message each other but yeah i'll just give you a minute to come up with what you want to do

[Music]

yeah query the logs good cool i like it let's have a look some suspect traffic shows up but it's not entirely clear since it has some similarity to a business process seen before investigation is effective and you get one additional investigative action if a senior director of the business is present in the game in this case because we have a department manager you actually get an additional freebie so what i mean by these additional free actions is in the scope of our one major turn your team now has one action it can run again so whether you want to uh do the the knowledge ingestion like sarah mentioned here whether you want to go investigate this

desktop individually and do a different uh security control whatever that is but you get one free uh investigate one free push of this investigate button without any penalty in the the turn piece so you have one more turn plus a free investigation ability you can hang on to it until later it does transcend through the rounds if you do want to use it for now so it's up to you if you want another uh query of the security controls that's not a bad idea because we have one device that looks kind of yeah but we're not really sure because it is similar to some business process that we've encountered in the past so it might be good to do another round of

this just to see but again i'll leave it up to you do we want another round of the logs or the sim data or anything in the ids

[Music] yeah another round of firewall logs sounds good or sim data so the benefit of querying the sim data is it can give you sometimes other correlating events outside of the firewall data so the firewall is just going to be connections in and out the sim data itself definitely can give you maybe some more endpoint like efforts uh or anything else that might have been seen across controls so sim data is going to be a good one i do i'm going to take the logs again and i'm going to use this as the sim day because that is going to be excellent so we have two devices we have a workstation 6 and we have a backup system

that are showing some suspicious activity repeated rdp failures were observed and a corresponding alert was found detailing these events as part of a possible brute force so we have one free investigative action that you can run this turn you can either hang on to it until later or you can use it now i would recommend either yeah either recommend uh investigating the backup server or investigating that workstation six um just to round out your turn which or you can simply hang on to it if you want to stack those up those do carry on yeah let's look at that workstation now i haven't had a chance to update this interface i just came out of beta with

all ah investigation complete no indicators of compromise found oh well okay so our technical team

we're still suspecting a breach in some capacity or another but we want some more investigation so my in my operations leads they have sprung into action everybody's interested i highly recommend this purpose-built knowledge because that'll give you some freebie ones there is a server outstanding yes we still have to investigate backup one but my inve uh but my operations leads here they do have the ability to um grant some increased investigative actions if we run this so i would very highly recommend this if the ops team is interested

let's do it grip it so we have another freebie investigation action so we can either delegate that down to the technology team to go and investigate the backup server but because you're the operations leads you can also leverage that this turn to go and investigate that server if you so choose so i'll leave it up to you to decide here and if in the absence of that we can just simply move to the next turn

and if we want again we can hang on to those investigative actions not a big deal we can certainly look at that backup server if we wish

oh and sorry that's on the ops team if the ops team wants to do that or not

all right let's take a look at that server anyway oh a device shows some indicators of compromise we found some encrypted files investigation is successful however compliance team loses one term while they take up the list of backups that are supposed to be encrypted yeah in in this case what we're seeing here uh is uh we have some encrypted files but pickup server we're not quite sure if the backup itself uh because we have a compliance requirement and we do store some of this information we do need to maintain some kind of security over the data at rest so the compliance team does need to go through and figure out whether or not that data is supposed to

be encrypted now i'm not going to miss i'm not going to uh like actually ask you to go and investigate that because i don't have any data for you to investigate compliance team it's going to be another turn before we come around to you because you're busy investigating whether or not this data is supposed to be encrypted perhaps i missed it but for curiosity what's the annual revenue of this company just trying to weigh the downtime costs oh the annual revenue we're at 13 million and change 1500 bucks an hour and you can actually see the current business downtime because the business we're not currently processing orders so we're losing up to three thousand dollars an hour right now

so we're it's it's gonna be expensive hopefully hopefully not too expensive uh okay so we have confirmed that we have something that's kind of fishy we're not really quite sure let's leverage our technical analysts again we want to dig in we want to find some more of this information it might be a good time now to leverage this accelerated knowledge ingestion this is representative of people you know having to read documentation pretty quickly and figure out if uh if we're actually in a position where we can do anything of a major consequence so um in this case uh the instant manager heavily recommends this uh accelerated knowledge ingestion uh as one of your turns or we

can simply run some more control queries uh as well so remember you get two turns uh because the technical teams get those by default so i'll leave it to you to decide

[Music] yeah let's build up some let's build up some investigations because we have to look additional two freebies so you still have one turn uh one one round that you can use within this before we go hunting did we want to do another round of our security controls because now that we we have some awareness as to some of the things we're looking for but we want to get further scoping so do we want to consider one of these now again maybe the ids the sim or the firewall log one of those three whichever keeping in mind you might get some similar events that you've already seen so you may want to go ahead and consider

the ids data if you're interested otherwise the sim data might be a good one yeah it can be yeah excellent so we have a whole litany of other devices now we're looking at something interesting so we have uh two database servers here that we are seeing as potentially problematic so let me just jot that down here sequel one and two uh and then we have some terminal servers that are also having some problems and potentially having issues we have our web servers jankies come on that's that's too funny that's too much that's too funny uh okay so prod server one two and we have workstation one two and five so we have a lot here we have two free

investigative actions we know the backup server is compromised because we have something suspicious we're not quite sure so we don't really need to investigate this one any further but these two web servers are kind of interesting uh and we have uh some terminal servers that uh where were our terminal servers yeah so uh our terminal servers if you haven't seen these these are basically systems that allow multiple connections uh to come in or sorry multiple users to use the same applications hosted on a server effectively sharing resources they they do have immense value inside of a business but they can also represent a bit of a concern depending on the applications and the level of use

so i think because we have two database servers that are responsible for our backend commerce those would be worth two investigative actions for free if we so chose i'll leave it up to the technical team to decide that or our two web facing servers the external facing ones either or so i guess in this case do we want to do sql boxes uh like database servers or do we want to do web servers

i love it jason uh so technical teams do we want to do we want to look at our web boxes do we want to look at our database servers they are pretty important yeah all of this is important for what it's worth this is all pretty critical infrastructure to the business why don't we look at our web servers let's see what those are doing ah so we have some indicators of compromise which is good a registry query script designed to find indicators of compromise that we know of in the the larger scale was successfully run across the environment and returned assets worth investigating we have four additional ones so we we're down one but you actually get four so you now have a

grand total of five that you can use so now that we now that we know more of what we're looking for now that we've started to peel away the various layers of this onion okay we've got an extra an extra four freebies so now i count nine uh eight because we just used that one so another box that so now we don't have to determine whether or not we want to investigate our sql boxes we go look ooh but the host is a lot of services running we can tell some might be suspicious though investigation is effective but it costs you one turn to run down the list so you actually now lose one of your investigative abilities

because somebody's having to run down this list you still have some more but let's go ahead and take a look at your servers attempts to query the local log data fail expertly when the administrator accidentally deletes the flat file we're not pinning this on either of you two this is an intern recovery efforts take your technical team out of the game until the next round that's a technical roadblock unfortunately for the next round now compliance team has come back they've realized that the files that are encrypted are probably okay but we're still needing some more investigation but we know our web servers have been compromised our sql servers have been compromised those database servers store customer

information and credit card data because we are e-commerce pci shop we haven't integrated any remote payment uh gateways into the mix we're storing everything internally on these servers and now these servers are confirmed to be possibly compromised compliance team we may want to go ahead and inform our compliance people simon i knew you were the right guy for this i knew you were i'm gonna you're gonna you're gonna have a buff for that joke i'm gonna give you a plus one for that that was good i knew i knew it like this guy he's gonna be the ceo this is the good one i like it uh okay so my compliance team i would highly recommend at this point

we may want to go ahead and possibly start involving compliance uh but that may be a little bit premature we're not quite sure uh if we actually run into a situation where we you know create a bit of a false alarm that might cause them to force and look inwards um if you wanted we can simply delegate the remaining investigative actions because you have a technical ability um you can go and conduct some investigative effort on someone's behalf uh if you want to carry on running down the scope of the the system so keeping in mind we're still in the identification phase uh we are at a point where we do have some visibility into what is affected but we

do still have a couple of other workstations that we know that might be a problem so compliance team i'll leave it up to you to think through it uh in this case it's just bruce so bruce whatever you decide yeah let's let's do some more so the reason why the delegation piece is quite interesting is a lot of this work happens in parallel right so the technical teams are constantly running these things down the compliance teams might come in and say hey can you look at this can you look at that these types of things will happen in parallel so i do like to introduce the delegation actions as an opportunity so we know that workstation

one was a problem so let's look at that some suspect files identified and confirmed malicious although attackers only use one method of persistence right is that right do they only use one ah who knows it's hard to say the investigation is effective and the technology teams lose no turn no turns if a team member can detail another method of persistence cool in this case we're going to put a pin in that one because i know everybody on that call would be able to identify at least another we're going to wait for that so we know this workstation is broken uh in this case bruce if you want to because we have a surprising amount of this

affected and we are again hard down and uh you know simon's golfing buddy said hey why is your website down this now has the various uh visibility from the top layers of management and they are insistently impatient so i'm going to grant you the ability to run some more technical investigations if you wish so i know that workstation 2 was a problem so we can certainly take a look at workstation 2 and workstation 5 was also of concern so if it's up to you you're sorry if it is up to you it is up to you do you want to investigate these two as part of the tail end of your turn here or did we want to simply move on to

the next next go i'll leave that up to you to decide [Music] investigate good call all right let's run some suspicious looking files were submitted for analysis and one was confirmed malicious and an additional two were were given so we've lost one but you have another two and let's go look at that other workstation now where is that one here workstation five there we go so some more subspide we have officially figured that out so uh thank you very much bruce for your intrepid efforts we have investigated all of the potentially suspect devices a pretty large scale compromise so we have both web servers are our backup servers both database servers one developer desktop one administrator desktop

one it desktop but our manager managed to make it out unscathed good for them so that's that's a good time all right so where we're at we may have effectively uh scoped out the whole uh incident we still have a lot of assets that are uh remaining seen so i'll give you some other assets right so our main controls here uh are definitely uh of a worthwhile concern we want to consider maybe some investigations in there individually keeping in mind that we do have some free investigative actions that we can still run so we can go ahead and use that but it has been quite a while since we've had any kind of communication and because simon's buddy got back to

him and said hey why is your website down simon wants an update and we haven't sent any for a while so all the teams are now in an all hands meeting all right because we have a headset i pulled up in in my porsche golf clubs are still in the in the front because i think that's where they have the trunk in a porsche although it's what i don't know about cars could fill a warehouse so there's a couple of ways that we could do this our department managers jason can go ahead and say no no everybody get back to work i'm going to brief the ceo once the golf shoes come off and the hat

goes in the back seat if that's what we want to do i'm going to go ahead and leave it up to jason to determine that similarly i have adam as my ciso this is actually a personnel roadblock so adam you can actually go ahead and simply bulldoze this roadblock if you wish uh and say no no i will update the business uh everybody else get back to doing what it is that they're doing uh in in any case uh i'll i'll leave the decision up to the cso and the department manager if you want to go ahead now if you want to pull everybody into the all hands meeting and just let that languish that's totally fine but the incident is

going to take up for another two hours with no resolution if you're a sales guy you come in pointing yeah ken that's yeah fair enough right every time there's a breach all these uh sales teams come out of the woodwork and sell you a bunch of shelfware jason's gonna brief the ceo excellent please the roadblock clearing out-of-band communication strategy so thank you jason for going ahead and briefing the ceo uh now as a result all of the teams get to continue playing so i'm going to go back to my technical analysis and i'm going to allow them either to uh run some kind of a containment effort if you want but we still haven't quite figured

out the scope of the problem we have some other other devices that if we continue to early we might actually introduce a potential back door that we weren't expecting and now we can't really defend well against especially if the attacker is still late in the network so we have our two domain controllers we have our two file servers we have a couple of app servers of generic use our terminal servers actually were reported potentially uh affected so let's let's think about some investigative actions here because you still have uh six freebies that you can use across the board so we can go ahead and use those if you wish if you want to accelerate some of this you can leverage

some of your security controls full disclosure they may have already given you the full picture that you need so in this case it might be good just to simply uh grind out these uh security or sorry these investigations if you wish that having been said the golf clubs the hat and the briefing took another hour so the time is ticked up we're officially at 7 500 lost for the day so far so tj and sarah i'll let you determine if you want to use these investigative actions or maybe go for a coffee break in game obviously please stay at your terminals i i'm relying on everybody here to be awesome because everybody's having a really awesome time

with this at least i am i'm having a lot of fun can we invoke help yes if you really want to you can actually go ahead to your siso and say uh we might go ahead and uh and call in a bird of a feather here or we might talk to the ceo who says okay listen my golfing buddies former nsa um you know he has a lot of this information as well um yeah the gray genius that's actually that's a passive ability uh that you you have so because of this you do have in-house forensics if you want to start to conduct some forensic investigation on any of these devices you have that ability in-house

it'll take you a couple of rounds but determining on the or sorry relative to what the report will find you may actually be able to get an incident power up which will bypass effectively a stage of the incident so if your forensic investigation finds the command and control traffic that it is using to communicate you block that on the firewall and all of the time it doesn't uh nothing can connect outbound then yeah you're in a situation where you've effectively contained the whole incident so if you want to take the time to investigate do the forensic investigation it will take away one turn per round for x amount of rounds depending on uh what i just what i

determine as the incident manager again i i reserve the right to speed up or slow down the pace of the game um so if you wanted to go ahead with a forensic investigation on some of these compromised assets i would suggest some of the ones that would leave good artifacts so uh these web servers would be good they can be pretty heavy in in data space so it may be a lot to kind of comb through and look at but given that they are public that might give you some of the insight into some of the attacks and some of the methods of persistence the sql servers are another good one because your compliance team is sitting

there on pins and needles going did this data actually get compromised they might have a duty to report this pretty soon so it might not be a bad idea to conduct a forensic investigation in one of these directions if you so choose now that having been said if you really wanted i would certainly allow you because again i control the speed of the game as i see fit if you so chose you could leverage networking efficacy to bring in uh simon's very capable uh former nsa friend we can also look to adam's uh see so friend here birds of a feather now may not be former nsa but will still come in with a lot of really valuable

expertise and capability uh in which case those are two options that you can provide so i'll leave it up to you too to decide do you want to conduct a forensic investigation on an asset to try and gain some kind of power up in the game or do you want to bring in some extra bench strength who might be able to expedite this incident in some capacity or do you want to grind out these investigative actions on the remaining assets that are great here so i'll leave it up to you to think yeah shorten the incident time we're burning money right lots of it oh scotch ken yeah i don't know if you caught my scotch collection here just a

few minutes ago before we got started but it's quite a lot here it's good can we do simultaneous actions you know what tj just because you asked i'm going to allow it to happen in this case though you have to do some investigative actions as one of the the pathways and you have to to bring in some bench strength uh you can't do the forensic investigation as part of it so if you want to do the forensics in-house and save that ability for somewhere else down the line you might get two incident power-ups if you choose to wheel and deal there whichever you choose so you can you can run down all of these investigations i'm going to give you

that as one of your actions in parallel as part of it but if you want to run uh an investigation you will have to do it next round investigations in forensics no i'm not going to give you that that's not fair let's put your controllers so the domain controllers for anybody who isn't familiar with infrastructure the domain controllers are effectively the crown jewels in a windows environment you can provision new accounts here you can add computers to the environment whether or not they are supposed to be there it really doesn't do a lot of determination in that space unless you have configured some of the logic so if you are what's called a domain admin that is effectively the whole keys

to the kingdom you possess all the necessary power uh to do horrible things or good things uh for whatever reason so your domain controllers are clean whatever happened didn't make it to your domain controllers excellent we're not gonna we don't really care why right now we're just all breathing a very collective side relief that our domain controllers were not hit as a result okay so that takes away two of those okay let's look at your file servers just here no indicators we're good and let's look at so we're coming pretty close to uh a good idea of what's been impacted now our in uh our compliance team is still very you know they're having concerns about

whether or not this data has been compromised they're kind of like ah you know maybe we should yeah well what can we do to get somebody in here let's go ahead and let's bring in our former nsa friend who can help us with this process there we go so simon's golfing buddy comes in driving an equally nice car although it does look like something you might find hanging around various parts of washington dc with buildings that have no signage and no foliage on the front simon's golfing buddy comes into the environment and says hey i know what it is that you're up against you have an attacker that's targeted your ecommerce databases on purpose they have gone so far as to

take you offline uh whether it's for industrial espionage we can't really confirm or deny but it is primarily looking like organized crime and one of the ways that organized crime likes to target uh financial ins uh financial systems like this is by the theft of the data itself um so we have now confirmed conclusively without needing to conduct an investigation in this network you're welcome tj uh that the the database information itself has been compromised in some capacity we now know that the databases that store the credit card data and by extension some of the customer information have been compromised yeah i think now is a good time so uh my technical team communicates that to the compliance team

and it worked out really well yeah oops hope it was encrypted me too ben because pci says it shall be i think you need an extra shot slots here you guys are welcome to come help yourself uh so the overly technical language is kept to a minimum and your readers aka the compliance team thank you for it very much this is important so when we talk about incident communication you need to assume that most of your audience considers an ipad to be a rare and mystical thing okay and that's not meant to be a dig in any direction that's not how this works this is meant to be an understanding that not everybody's going to know what a domain

controller is not everybody's going to know what sql is or why it's relevant not everyone's going to know uh how the web server plays into the environment and how a website can lead to data theft right not everybody's going to be aware of that so you want to present this in a way that anybody can understand this is an ever evolving challenge i promise you organizations with security budgets that everybody would happily step over anyone else for they still struggle with this step too effective communication is incredibly difficult so congratulations technology team that was an excellent round we now know conclusively that our data itself has been affected uh we have the the nsa friend has since

gone back to the links because uh he was very upfront and saying this is no longer my pig or my farm uh simon whatever favors we have whatever dirt you seem to think you have on me now no longer exists away we go so compliance team i'm coming to you bruce our compliance team now knows conclusively that credit card data has been impacted in this breach i would very very highly recommend informing the card brands at this point which would be our paper dragon's ability but i'll leave it up to you

surfing in d oh okay all right you can have another buff for that one that one's quite funny actually all right this is a good crowd this is a really good crowd uh okay so in light of the fact that bruce is currently uh dealing with this resume generating event um he has gone ahead and stipulated we need to inform legal excellent this is a very important thing that just happened why do we inform legal before we inform compliance here's a good reason legal uh in this sense has the ability to tell us what we can be exposed with uh whether it's uh in regards to compliance whether it's uh in regards to the personal data legal is absolutely an

excellent move here very nice work bruce now how important those money because they charge for every phone call email that you get sent you somewhere around every seven minutes um so in this case the conversation of we've been breached this is what's been impacted this is how we went about looking at it here are the players involved as in all the people involved in this incident all of that matters to legal they have a lot of information they need to collate and then unfortunately saddle with their paralegals to deal with on a weekend but i digress in this case doesn't the cso have to talk to the cfo yeah that's that's a good point um in in

this scenario no the cfo uh really isn't a part of this because we have an official cso um but sarah what you've touched on is something very interesting sometimes security programs roll up to the cfo instead of the cso uh one of the reasons that has happened it's not the only reason but one of the reasons that happens is because computers traditionally were accounting tools first so when computers were starting to roll out in a lot of businesses you started to see the accounting departments getting them first so historically some computer program elements within the business roll up into the cfo office but because we're maintaining compliance we are maintaining a very capable security team

and we have some favors that we may have who knows what we did to get them we have uh a cso who has a seat at the table in the board of directors so they don't actually need to worry too much about that but i really like where your head was out with that thank you for bringing that up that was a good one okay so where are we at we have a great deal of stuff compromised we now need to start containing this right we have some application servers that are looking uh like they may or may not have been affected our terminal servers uh have been possibly i'm just gonna go ahead and grant these

uh for the team right now just so that we can get to the scope of the breach there's too much activity on this we're not sure why all right and but it's worth uh bruce i'm going to take away your little bonus buff there because that was the the resume generating an event has put some grain or temples uh we're sorry about that but it is the way it is so our application servers look good we don't have to worry about those so we can safely ignore them our terminal servers do have some malicious indications on them those are considerably uh you know the the footprint on this is enormous in most cases it's easier just

to pave and replace some of the the terminal servers we don't really have to worry too much about those right now a couple of workstations that need some attention and we have uh definitely our database servers they are compromised they are having some issues and uh of course our web and uh all the rest uh okay so i'm going to turn back to our ceo one of the things that comes up with these public-facing breaches as in your website is down and nobody's able to do the work some of your customers have started to reach in and they're very upset they don't know why they can't get their information that they need they don't know why they can't get the the

transactions going through some of your pretty high value clients are needless to say a little bit miffed that some of your systems have been down for as long as they have because some of them rely on this as part of their business because it isn't necessarily an atomic situation your service might further impact some other services third-party vendor risk management is very important in nearly every security program so it might not be a bad time to start having a conversation with some of your high value clients in which case we can start this customer interaction we haven't lost any data yet have we we don't know we have some data compromise as in somebody has seen

the data which means it's unauthorized and we do have some disclosure now it's entirely up to you you don't necessarily have to uh inform the consumer and the end customers just yet if we don't uh want to you can certainly wait on this this is one of these you can play the ability as often as you want but it is one of these other ones that is based off of pure random chance uh yeah and you're right we don't want the press to find out until we've got our ducks in a row so it might not be a good time to pull that one if we're uh interested in that another thing we can do is we can start

the process of engaging our compliance team we've spoken to legal legal says yeah you can go ahead and have conversations with the with the card brands that's part of our requirements we do need to inform them because this data has been compromised uh in which case you can go ahead and run this interpersonal tactics that does allow you the opportunity to have a conversation with compliance ahead of the compliance team and just kind of prime the card brands if you want to i'll leave it up to you to decide how you want to spend that time or you can simply delegate some containment actions down to the technical team that's also an option

[Music] isn't a delegator when everything's on fire yeah that's a good point uh because if the ceo is having to show up and tell everybody what to do uh you know some layer of management isn't working so in this case did you want to inform compliance uh about the as in inform the card brands to yeah let them know that this is gone okay okay so we have officially engaged the card brands in our regulatory compliance overhead so they are now aware that this is happening okay and they said all right good uh you know keep keep working at it we know that this has been compromised we'll talk to your legal team and get

that information everybody's obviously going to send this business a bill so we're getting close to our gap okay so we've been working this incident for nine hours when we get to hour 12 the whole team quote unquote has to everybody has to go home go to bed okay so uh what we're going to run into here pretty soon is the incident isn't able to work so given that we've investigated the full scope of it we may want to consider some containment effort uh in which case we have some freebie containment actions i can't remember where i put those uh here we go so our compliance team having been former technical they understand things like firewall controls

uh tooling and stuff associated with it so bruce i'm going to come back to you here because we've engaged the card brands and we've told them hey there is a problem we are working on it we do have data compromised now we have to stop the bleeding because we only have three hours before everybody has to go home and go to bed because we can't simply keep everybody sitting in those seats as long as we have been so my recommendation highly highly recommend running some containment actions so that we can delegate those to the technical team and leverage some of the additional abilities that work hand-in-hand i'll leave it up to you to think uh if you want to do that or

update compliance again if you so choose but keeping in mind that this has quite a cool down and it might be good to do this a little later on

unless bruce had to leave in which case i will be the new bruce that's fine too [Music] and it's sort of whatever it takes i'm off to the golf course no problem simon containment sounds good let's do it containment efforts which means uh relative to incident response that could be uh doing things like inserting firewall rules but we have to know where the traffic is going in order to make that a reality so that's one piece that we have to keep in mind um if it's a physical asset we can simply walk up and plug the uh unplug it and move on um any of the virtual assets we can sinkhole them with some virtual

networking and segmentation options so back to my technical team now keeping in mind that you get two turns for the sake of one it might not be a bad idea because you should have some additional containment efforts might not be a bad idea to leverage this incomplete archives because it will give you something uh

fixing so i think at this point if you wanted one of your turns could leverage the incomplete archives to get you some more containment efforts but i'll let you debate that between yourselves

going home after 12 hours people need sleep simon they have to go to bed sometime yeah that's good let's do that we have an extra two additional team in action so i count a total of four you have another turn that you can invoke if you wish uh keeping in mind that it's been a little while since we've had a communication [Music] situation might not be a bad time to send a status update just in case and coffee and pizza and then goes home ken you have the ceo thing wired shut communicate thank you sarah let's have a state the status update contained too much information nobody read it communication is ineffective now there is no penalty for

this but if the status updates are too lengthy or they contain too much information that isn't relevant to the audience on the other side of it uh that can be a problem because then it doesn't get red one of the the consistent kind of human nature gaps that we see in incident response and by extension just operations in general um if the if the manual is too wordy it doesn't get red we run into this situation where the data just doesn't get to its intended target so there's a consistent problem it does happen so given that we have we've communicated where we're at we still have some additional containment actions i highly highly highly recommend uh let's go

ahead and start containing some of this activity and see if we can get it so given that these two are public facing and people are con consistently trying to hit them we may want to go ahead and try to contain these first because if there is any damages to those systems and that damage is allowed to persist into uh the actual uh system the public-facing part of the system itself so if we're thinking something like uh like embedded cross-site scripting somebody comes to the website they get something through a drive-by download we run into a situation where we're infecting our customers might not be a bad idea to go ahead and start running some of these containments here

so tj and sarah i'll leave it up to you to decide if you want to contain your web servers given that our database servers also provide some data and do contain a ton of data it might not be a bad idea to contain these especially if the attacker is late in the network we want to make sure that they can't continue to exfiltrate this if they have chosen to steal it is there any chance that there's a free pen test and we are just overreacting uh you should never overreact to a free pen test cody that's probably not a good idea if it wasn't if it wasn't authorized i would say that's not overreacting um so

we have four containment actions if you want we can go ahead and uh and try one of these two and uh try uh either of these two as well now when i say try it isn't a foregone conclusion that you're going to get it all so it's something to keep in mind so i'll leave it up to you too to decide do we want to start with a web server or a sql server

web server i agree let's look at a web server excellent that one worked out okay do you want to do another web server chop the wires do we want to do a web uh contain another web server do we want to try and contain a different one what did we want to do sorry it's back to tj and sarah

yeah it's a good idea a database server might not be a bad thing to try and contain let's do that there we go so that one worked out too and you know what just because the teams and the jokes are really excellent i'm going to let you try this one oh identified services are targeted for remote stoppage via powershell script some machines fail to execute the script for one reason or another but you get some free contains now on the other side so we burned another one so we have one two three and four so we have four left um but in this case the technology teams wow they're they're pretty beat they've been working this

the hardest so i'm going to take them very close to that situation i'm back after my political fundraiser what's going on need more pizza how about starbucks and fancy food so trotting out the expense accounts for all the really good quality food is amazing uh usually in most incidents you're lucky if you get a uh dish a cold pizza it's been sitting in the break room since you know president taft was around so uh yeah that's that's awesome right on simon um in this case i'm going to go to my operations leads

why did i want to go to my operations leads oh not yet i'm coming to you for eradication that's that's why okay let's let's go back i thought these guys had another one too i built the game you think i'd remember where i put all this no okay that's coming up all right so uh let's let's keep on our technical teams we still have some compliant uh containment effort that we can put in place here so i highly suggest that we uh try our next web server and maybe our backup server uh if we choose to because we've contained both of our sql servers um and that we totally test we've been hogging all the turns for what it's

worth sarah the technical team actually does most of the heavy lifting in incident work but yeah it's all good so i think in this case uh i would like to suggest to the technology teams let's go ahead and see if we can disconnect these two boxes from the environment as well because then we can start to talk about some of the the conversations that come with it later on

so yeah i guess sarah and tj any any disagreements in the containing these two

yeah the backup server was one of the first we found compromise that one's good okay that one works technology is always great when it runs right excellent that one's good too so we've burned two more of those all right it has been 10 hours and our customers have not been able to purchase anything some of them are likely to start showing up in your office spaces whether or not you want them there is completely irrelevant simon i think we need to have a conversation about the outage to some extent now keeping in mind that we control the narrative right now okay but if it's if it's allowed to languish or persist too long that can be a bit of a devastating

situation because we have high value clients because we have a lot of b2b involved with this e-commerce thing we can simply put something on the front page for a lot of our uh custer like our consumer layer customers and just say hey sorry system's down for whatever um you know we can we can simply put something there most of the consumers are just going to carry on yeah we're going to lose that business we may not lose any reputation but the b2b customers that are relying on this service as part of their business they have answers and questions that are coming in their direction and they're of course eating those into your inbox whether or not you want them to is

irrelevant so i would highly suggest uh let's run business charming and see if we can uh maybe placate some of the the concerns that are coming from our high value clients but simon i'll leave it up to you to decide in between political fundraisers and starbucks runs [Music] oh that facebook outage was gold turn on the chart that's it oh this guy your trust is the most important thing and working all hands on deck i'm starting to feel business excellent so our customer has been successfully updated good shape we don't know whether or not it's useful uh because we don't really know at the end uh it's it's it doesn't show up until the end of the session

that's the worst part about an incident is we are blissfully unaware of what happens all right it's now down to the 11th hour literally my department manager jason jason the facebook outage the facebook outage we are really coming up to a point where we need to accelerate some of this incident we have a lot of stuff contained but one of the things that we might want to consider doing is this resource allocation where you simply come to the team and you say yeah listen i just need a little bit more gas in the tank we'll get through to this whatever we decide to do is whatever we decide to do so i will leave it up to you to decide

if we want to push the button and get some more bench strength illicit drugs there is no crystal meth in this it's not how this works we're not encouraging that damn yeah no no amphetamines that's an expansion pack

bringing more people all right let's let's so as a result of this we can now proceed into our 13 uh or sorry hour 14 uh because i give you the your team the break at 12 you get two additional hours that we can delegate down to the the technical analyst team that did turn now technically speaking everybody goes home to go to bed that having been said because we've played this ability we've gone ahead we've shuffled some things around he said hey listen we'll promise you some day off in lou and maybe some gift cards to help you make it up to the people that you're supposed to see today yeah good good question we now reserve the ability to play

another couple of actions so what we can do is we can either play this birds of a feather and bring somebody in who can help us out kind of at the 11th hour or in this case i guess the 12th hour which would be good the other side of it is if we bring them in the next day we might be able to get an incident power up from the other side of it keeping in mind we still have some containment that needs to happen we still have some eradication that needs to happen and we have to bring in uh bring the business back online before we can do that so might not be a bad idea

to bring somebody in if we needed to but it's kind of late we're gonna have a hard time getting anybody on the phone uh no guarantees as to whether or not we're gonna play that one that having been said uh again technical teams your ability stacks so because we oh you could bring in a partnership if you wanted to um keeping in mind that the partnerships don't necessarily bring in results right away so we might be going to a break anyhow uh which we can bring them in uh if we so need it through our incident response assistance process right yeah phone can don't phone ken he's sleeping um but if we want to keep this

going beyond the scope of our our time here uh they charge a fair clip potentially so we're not really sure but we can pick one of these three type shops so a boutique shop is a smaller business but usually very highly skilled people who are around in this case if they're allowed to come in somebody just needs to be available to help coordinate be hands on feet bringing them into the business whatever that looks like for it it's worth you can actually make a pretty handsome living doing this this is kind of a fun part of consulting work if you're incident uh if you're into that so far your incident cost is 18 000 bucks in lost revenue but we're about to

have a pretty prolonged time before we can go ahead and bring the business back up in line so um if we wanted to bring in some additional incident response help they will be allowed to continue working the incident while your team goes and gets some much needed rest so it's i'll leave it up to in this case because the department manager went ahead and pushed the button i'm going to leave it up to jason to decide if we're going to bring in some incident response help or if we're going to simply leverage our existing technical teams for two more turns to try and get something over the line

yeah simon really has this thing nailed we're gonna lose another 18k people i want results i don't want excuses bring in the pros okay all right so we're bringing in the pros um let's uh let's leverage this specialized special deliver some really exceptional results if you're willing to pay costs are higher but results can come faster and may include additional findings that can be used to expedite incident remediation so they are going to show up for the paltry sum of 12 800 which is peanuts relative to what some of them are going to cost otherwise so excellent that's a really good one the other side of it is they are going to deliver results for us in 28 hours so

it might take a while for us to get that to work in this case with our last remaining one hour ceo the buck stops here basically it is hey listen i'm the ceo i'm gonna clean this uh this problem up if i have to but right now i had said our biggest customer is on the phone threatening to leave there is an extra amount of zeros i can put on this check to make it interesting to you what say you insert shop here so i highly recommend with our last turn let's go ahead and leverage this what say you simon [Music] i say yeah perfect yeah ken sorry to answer your question uh the annual

revenue every year

needed rest were able to quickly on board this uh services provider because they know exactly what they're looking for what they need to do no ken please this is this is all good everybody's here to learn and have fun that's what we're here to do please enjoy um we we bring in this team they're here to help us uh they get up to speed very quickly our technical teams because they rock at sending out communication information and the status update that contained too much information was of value to a fresh incident response team they come in they start working it everybody's up to speed all they need is a point person to stick around and be

available by phone should anything arise but other than that they have the information they need they're going to start working the incident so it's been

all right so because ceo managed to talk their ceo and got some very interesting things happening we have ticked up our current incident costs it is a little bit expensive but our intrepid investigators have some information for us they too had an in-house forensics person who came on was able to image some of these affected systems and work very quickly because this is how they spend their time the benefit of bringing in very specialized shops in in and amongst the price is that they deliver really excellent often very effective work so as a result they know how all of the mechanisms of communication happen within the the affected systems they're aware of how the attackers phone outbound and

communicate with their command and control servers they've locked all that to a point but they cannot go ahead and push the the button to contain it all the way because they usually need somebody with the proper amount of access inside the network fortunately your firewall admin uh having lost a fair bit of sleep worried also about this level of resume generating events so as a result we've now successfully fully contained this incident so now our teams have come back everybody's feeling refreshed sort of as much as you can in a late incident kind of situation we've got some time in in the the books here it's going to be kind of expensive we do still have our uh

incident response team remember when i mentioned we have been strength they are here until the end of the game until they also need to go home and go to sleep so we're coming close to the tail end of when they need to go because they've been working all night so we do have another couple of rounds with them and i won't hesitate to introduce that if i need to so now we're at the eradication phase we've gone ahead and identified the scope of the breach we have contained all of the affected assets none of the evil can communicate outbound but now we have to start finding evil so one of the things that our our team did our incident response team

did the the uh sorry the outsourced one uh was they did come in and they said we think we we found some of these mechanisms of persistence we have found uh you know some interesting information so as a result we now get a couple of free eradication actions i'm going to go to the department managers in the process of understanding the eradication effort that needs to take place we need to understand what's supposed to be there and what isn't that's a common problem that we have so in this case uh the uh our our outsourced provider spoke with the department managers and said hey we found some suspect things we're not quite sure uh would you be able to go

ahead and uh and take a look so department managers uh jason again would you like to uh go ahead and just review this list to determine if this is quote not one of ours to possibly get some additional free eradication actions that we can pass down to our technology teams i'll let you decide [Music]

unless jason had to go in which case i i'm the department manager now oh yes sir i wasn't sure if you caught that did you want to uh play this not one of ours ability and potentially get some additional eradication effort uh to help our technical teams do this a little better at scale yes not one of ours let's see all right so two freebies so we have uh four free eradication efforts back to my technology teams i don't think you have anything of value here surprisingly you don't nope maybe i'll build one of those i'm gonna jot that down the best part about just coming out of beta i'm still finding all kinds of cool

things okay so technical team do you want to start the eradication process on our primary facing web servers that might not be a bad idea we can also start the eradication process on our database servers because knowing the amount of data that might need to get replicated and addressed inside of that time frame it can take a while for that to replicate and build back up to to a point of being useful uh especially with these large-scale databases so i'll leave it up to tj and sarah to decide i recommend either running eradication actions across these two devices your prod sql uh and prod web keeping in mind that one action does not necessarily guarantee

success so i'll leave it to you to think through that can we start with the db and move on to the web that is a good idea but i'll wait for consensus on this one and we're shooting from the hip oh sarah music to my ears as long as the last action is communicate wonderful the service was stopped but now it started again someone didn't set it to stop then manual startup eradication is ineffective when we are talking services especially in a windows environment they are most of them are set to start up again by uh automatic right so when the server gets rebooted and comes back that's one of the ways that it maintains

persistence that said attackers leverage the same concern they will go ahead and insert themselves into services or create services that are just set to automatically start up because we didn't set that to uh disabled entirely or because we didn't set it to a manual startup where we have to manually go in and click we now have a situation where we need to try again so i am going to let you try again on this database server just because the as long as we communicate at the end was excellent unfortunately this didn't work out very well helpful help desk hank created a domain admin account for someone on the phone but he can't recall their name

technology team loses two turns looking for suspect accounts department managers lose one turn coaching hank on proper handling of admin accounts damn you hank in this case now we have to communicate that and oof that's a rough go so let's i hate hank everybody hates hank right apologies to anybody on the chat who's officially named hank i promise you the naming convention was purely for the alliteration and not for any uh offense in that direction okay one thing that's come up legal has gotten back to us and said hey compliance is wondering where we're at the business is clearly still down because they can go ahead and determine that hank is the ceo's nephew oh man

simon you're just you're nailing it here okay so you can have another buff for that one that's too funny um yeah absolutely in this case the um the compliance team has gone ahead and said listen we need to update uh our our card brands because legal is pestering us and when legal pesters us it gets very expensive so bruce i'm coming to you here do we maybe want to leverage paper dragons to see if we can get some compliance feathers unruffled a little

this bruce says i damn you hank don't be a hank paper dragons that's right let's do it excellent okay so we've updated compliance they're happy our intrepid incident responders are about to head home so they're calling it a day we uh we're on our own for now uh until we decide to either invoke another favor from another friend we'll go ahead and hang on to those though because i think it's going to be pretty good here pretty soon okay so in this case our department managers we still have this not one of ours and we want to leverage more eradication efforts so let's consider not one of ours again so jason i'm coming back to you do we

want to leverage this because remember our technical teams are still dealing with the account issues so we definitely want to get your buy-in is to figure this out good policy on paper always stops the bad things from happening you guys in these security jokes this is awesome yeah let i'm going to suggest we go with not one of ours because in this case we need more eradication actions we've confirmed that we have malicious accounts possibly uh that an attacker may or may not be using for a mechanism of persistence that's what that did for us we get four oh my goodness okay two three and four excellent all right back to my technical teams let's

continue the eradication process so one of ours both of our database servers still need to be eradicated in this case because of the nature of how this is and how long this incident has progressed i really oh my goodness in this case i'm going to suggest that we burn our eradication actions trying them across the plant to see if we can close that down and get the attacker out of here but again i will leave it up to uh tj and sarah just to confirm data classification project what come on

it's all classified sorry yeah eradicating them all let's let's try and nuke everything from orbit i agree let's let's try and clear this out tj any objections we're gonna we're gonna run your eradication actions we're gonna grind them out and see if we can do that nope let's do it database over there okay well it turns out the coaching did work for hank uh and we managed to catch that in advance i'm not gonna i'm not gonna saddle you there we go some files and likely persistence methods were identified by mixed in with known functionality uh eradication but it costs two turns hunting it down again i'm not gonna saddle you with that um but that's a

real problem in that there's too much on a system we're not quite sure what's good bad or ugly uh there's definitely more than enough things out there that look suspicious without actually being suspicious let's take a look at your other school server here too excellent malicious files and attacker methods are successfully removed so that server is officially clean and good to go given that you still have some eradication actions i would like to heavily suggest your backup server because that will help you in the recovery phases coming up here very shortly so um not to point you in two point a fine of a direction but let's go ahead and maybe consider this let's say

you uh tech team

yeah i forgot about the backup server yeah everybody forgets the back all right i'm going to let you keep shooting on this one until you until you hit it treat it like a game yeah all right that that works ooh but it costs the obsoletes two turns hunting down the business impact well poor ops leads haven't really had much to do uh i'm going to go ahead where's my cso adam let's leverage this battering ram to keep our operations teams working effectively because right now our operations leads are basically spending a bunch of time chasing some of the stuff down the sisa goes nah i can go ahead and hunt this down for you uh you know i can talk to

the business units and get back to work all right so that did cost us leads you have some free eradication actions if you want ramming speed yes daniel simpson day no records of any kind those are my favorite jokes um so my ops team here uh vincent ben do you want to go ahead and just do it and go through the eradication piece just to make sure that we can maybe close off some of this and bring the incident back up to uh bring the business back online what say you

let's do it all right okay so you get some free eradication actions and in this case uh given that this is ticked up to where you want us to get to the end i as the incident manager have uh inserted myself into this process the various favors and buffs that simon had cultivated by being uh the true example of a hysterical uh stereotype ceo have now made it so that the nsa uh the the the favors that were owed from the nsa friend yeah it turns out they weren't quite even and there was some uh mechanisms ahead so they actually managed to pull in a few amount of a fair amount of agency resources and to

the point where we have gone ahead but this makes us even was what was said on the way out the door as in okay we've gone ahead we've cleaned everything up i can no longer abuse government resources for personal gain wink wink um in in this case we now have an environment that's 100 free of the attacker we can conclusively determine that because we knew all the mechanisms of persistence we followed the process of identifying the full scope of the breach and because we made it in such a way that our systems could be brought back online safely so we've successfully contained everything now where are we at so our backup server everything is back to normal

where's our compliance team compliance we told you to put that there this is one of these situations where the compliance team went ahead and validated that the existence for good security controls was needed and by extension uh they have gone ahead and granted some benefits to backup and recovery however i'm going to go ahead and leverage this ability bruce in your stead just for uh just for posterity purposes here and to move the game along a little bit one of the things we fought for was excellent backup resiliency okay data resiliency is really important in organizations regardless of what you're doing knowing that a pile of our data had been compromised we can't confirm or

deny that that's a concern um relative to the threat so we want to make sure that we have some nice clean data because we've put it in two places because our compliance team won a boardroom battle where you know printers were left off the budget cycle in favor of some level of automated patching systems we now actually have the ability all our trust is back online automatically thanks to some of the efforts that went in from our unsung heroes in the compliance team so the business is back up and running we have one more thing left to do simply to communicate that the business is back on and now we're there okay so that's it we're back online but how do

you do right what what fun is a game if there's no uh if we're so our total incident hours were 28 hours across the whole board okay and so at a cost per hour of fifteen hundred dollars uh for revenue the total loss revenue is forty two thousand bucks okay so when we look at the labor hours here that were spent this is across all of the teams right keeping in mind that uh teams had multiple versions multiple people working on this rarely in an incident are you working uh by yourself unless you're in a very very small organization in which case um you know do please take care of yourself our compliance uh sorry our services

providers uh we got a steal of a deal at twelve thousand eight hundred bucks for those really highly specialized people uh again we must have called in some favors or the buck stop here was very very effective from our ceo in any case we have a steal of a deal on really excellent incident processes here so good job our compliance handling effectiveness 77 percent successful which means uh there is some uh fine as a result of it because we had some controls that were out of compliance this data that was compromised was exfiltrated which shouldn't be possible inside of a cardholder data environment we had some controls that were out of compliance despite being in compliance we have a

considerable problem to clean up after as a result of this so our total incident costs 77 800 when we factor in our fines our service provider and lost revenue now because we spent a good bunch of time talking to our high level customers they managed to stick around but 25 percent of them aren't coming back this is a very real problem inside of many uh businesses that experience a breach depending on the stats and where you look it is going to be about a 20 loss that adds up right that 20 of a business uh is quite expensive so when we're talking 13 million uh you know that that really adds up on an annual basis but we

managed to keep 75 percent of our customers we're in good shape yeah no bonuses for anybody except simon this year right um but our total data loss 200 gigs so for what it's worth 200 gigs of uh personal information uh credit card data that's a substantial amount of data that's probably the whole database at that point uh depending on the way it's configured and how many customers we truly have so uh we definitely get some log deletions no do not delete the log data um you guys are killing me this is hilarious um yeah please don't please don't delete log data it's very important especially if it has good telemetry but 200 gigs of credit card data left

the building so chances are what we're going to be faced with on the tail end of this is some really tough conversations with our acquiring banks and the card brands that might be represented as part of this too so it isn't necessarily a foregone conclusion that we're entirely out of the woods in this case we are uh in a position where we're going to have some considerable issues so i have a couple of last minute things here just to present and then i will call it a day for everybody so let me get this back up here

asking to do it he's helpful there's the finger pointing yak yeah poor hank is now sleeping with the fishes uh because you know some of the other favors go both ways uh so let's let's have a debrief one of the things i mentioned uh in this whole process was you need the the lessons learned in the postmortem obviously we aren't a real company obviously we're not going to be making any firm business decisions uh based off of this so i'm just simply going to uh render this here did you all have fun did you enjoy what we were doing even though this might have been slightly panic inducing if for what it's worth if you genuinely

did feel any panic i apologize i promise you no ones and zeros were truly hurt in the process of this but that this is the idea the idea is to practice these in a non-impacting situation so that we can determine response capability within the teams security control gaps potential compliance fines um you know the process in general how often should we be communicating how often when do we get legal engaged do we have a pr firm if we have to go public what does all of this look like this is the whole point of the game is to elicit some of these questions among the various parts of the business when we practice these concepts the

teams get better right that's how it works so while this was a simulated incident the problems were real life i mean companies lose hundreds of gigs of data all the time and it never goes reported because most of the time they don't have a duty to report that we are starting to see that change we are starting to see jurisdictions that have mandatory breach uh reporting laws however that is still emerging as we go through um but in this case the costs in the net impact are considerably higher so when we factor in something like credit restitution right so in this case because 200 gigs of credit card data walked out the door we probably have a

considerable amount of credit card data restitution that has to happen to our customers uh depending on how you're uh or sorry depending on how many users were affected and the kind of plan you can negotiate you're probably looking at about 10 bucks per user per month uh so when you scale that up to you know a few thousand users that is definitely one of those situations where you're looking in the millions of dollars another thing to keep in mind is insurance right if you have insurance the insurance plan might not actually cover this part of the breach that part of the breach or the breach at all one thing we are finding is the the

market is hardening for uh breach response and insurance so we might be in this situation where uh your policy doesn't cover the 10 million bucks your organization has to spend to you know restore all of that and everything else adds up pretty fast all right so this wouldn't be a session if i didn't leverage some kind of shameless self-promotion this is a service that i run this this is actually what you've experienced is a business facing service that i offer the way i built this is inside of your organization i can take in your asset inventories your incident response plans any of the specific elements that go into it and come up with a bespoke

instance of a game just for your business focusing on a relevant threat case that you can expect based on public-facing intelligence if you have pci compliance mandates to attest to on an annual basis uh requirement 12.10.2 stipulates every year that you at least have to do this this will count right so uh in in uh for anybody with pci in scope this 100 applies because everybody i've reached out to on that side went what there's no guidance for this which means this works uh and in the event they try and fight me while they know whatever we'll go from there but at the end of the day this does count as validating requirement 12.10.2 similarly within the pci space and other

security compliance and even just good security practice this does count as security awareness training one of the parts of the service you get the short and condensed version given this the size of the conference but one of the things that this service provides is either security awareness training at the front of it where we talk about some general security concepts that are applicable to the business or if there's already a security awareness service inside the business that's working really well for you and in your uh teams no problem instead of the awareness training we simply do a case study around breaches in your industry potentially even some of your competitors that's not for me to

determine obviously but the breaches that are focused in your industry based on the threat cases you can expect and those outcomes so then it becomes a discussion of oh if we were wrapped up in this we could expect some similar outcomes that helps to put a bit of gravity around the incident as well um but in the event that you're missing your asset inventories ir plans or they don't exist uh that's okay because we have a lot of systems in place currently that allow us to do this for you in a frictionless fashion so our frictionless security foundation service will have this all sorted out for you pretty quickly all you have to do is

provide some upfront information that you would have to provide in order to make the game effective anyway uh so in the pursuit of running that information down we can handle some of the back-end paperwork for you and include some other uh notification services as well to let you know hey your your documentation's a little old it's going to be worthwhile for you to evaluate it here's some things to keep in mind so we can take out a lot of the guesswork so if you're interested in more feel free to reach out info at syntax security.ca our website security dash selfawareness.com it has just gone live so please go take a look it's uh it's it's all right it's not bad i like it i

think it's cool it's gonna be more to come on that right i'm looking to build a blog out for this uh where i can start to share incident response tools security tips uh whether it comes to any of this uh absolutely this is something available and if you want to reach out to me over linkedin you want to stay connected i'm in there under anthony lease but yeah you can you can call me aj we're all friends here it's all good so feel free to do so and with that i'm going to open it up to any q a uh and if you want to go somewhere else and start your friday early please uh don't hesitate to

do so i really want to thank everybody for being on this spectators and participants i had a lot of fun i hope you had a lot of fun i really enjoyed uh playing with everybody today i hope everybody learned something and uh and we're away away to the races