Incidents and Accidents: A Security Incident Response Game

foreign

welcome everyone thanks for coming besides 2022 how many of you in here this is your first conference and yeah in general for security conference yeah cool welcome uh so I Am AJ I am the managing director of syntax security our mission is quite simple we play games and we have fun and in the same vein we'll teach you something about security along the way I have my uh developer here Nathan who has done an amazing job taking my initial Prototype game posing all of the AJ alphabet and then making it actually a whole lot better than it originally was uh so today's session we're a little bit condensed for time so for what it's worth the true gain really plays out

over a couple of hours it gives us time to sit talk and explore the scenario really don't have that kind of luxury today because there's just so much amazing content out there for all of you to get to I don't want to hold you up any longer than I really have to so I appreciate your attention and your effort we're going to get through everything just nice and tickety-boo we actually have a whole new build of the game to show you today so you're getting the first exclusive look of the latest version of incidents and accidents I call it version four because the other versions were just so far behind will be built that this is just really something

incredible but before we get to that we have some Death by PowerPoint because what's a presentation without a little PowerPoint so go ahead so today's agenda I'm going to talk a little bit about incident response because I want you all playing from the same sheet of music so we're going to cover the six main outcomes from incident response really how the business really needs to interact with incident responders what they need to consider as they're building this uh plan and they go towards so we're going to be talking a little bit about that then we're going to get into the gameplay so like I mentioned before we're a little condensed on time so we can't get a full

proper game because there's just too much cool stuff in here to show you so we're going to run it a little more like a game show and less like a Hands-On so traditionally in a smaller group we'd have a bit more of a Hands-On thing this game does scale well to that size but again just for the the shortage of time I want to make sure we have a chance to explore a little everything so we're going to do a little bit more game show uh and a little bit less Dungeons and Dragons it's still going to be good I promise if not well um you know don't tell anyone so a little about me AJ I got some

letters inside my name some of them are probably set to expire soon enough uh Nathan here is actually a former student of mine uh when I was teaching back in the day and is now a world-class developer who's helping me put this all together overall I'm an information security geek there really isn't a lot about information security that I don't find interesting to some extent yeah except maybe risk and compliance but I did my time in those in those those areas in the paperwork is vital to Security Programs uh the compliance folks they play a really really excellent role in the business so in the game sessions the the full game sessions there is a role for compliance to play

we do bring them to the incident response ideas because they have such a vital place to be within other than that super into other things within information security and super Avid Gamers so if any of you like to play if you'd like to have a good time uh you know hit me up later we'll uh find something to do and we'll enjoy it so why are we here simple we're just here to have fun right like there's so many good presentations out there so much good content and we're just here to have a good time we're here to have fun learn explore Network and socialize all these people that you're sitting here today we're all attached to the industry

in some way whether it's your first security conference maybe you're here for your multitudes of security conferences there's a lot of people in this room all of whom have very interesting opportunities this is an opportunity for you to kind of sit and chat and network with them as well so uh we're we're going to be having a good time so today we're going to have an incident and in most cases in a business you're never going to get a heads up that you're about to have your Friday ruined and your Saturday ruined and maybe the next month or two ruined with postmortem meetings uh so today huzzah because we're having fun at a conference yeah

something bad is going to happen to a company fortunately none of us have a real strong stake in it uh nobody has any shares of it or anything like that so the impact is completely fictional uh but the scenario is real everything that is conceived here that we're about to talk about today was born out of this is how and could happen and the logic all goes up so again we're don't worry about it if you're feeling a little bit like oh the impact don't worry about it there is no real business no actual ones and zeros were harmed in the making of this but our goal is simple we want to resolve the incident and get the

business back online so there are some session considerations again just because we have a shortage of time we've stripped out a whole ton of cool stuff it's really cool I can't wait to tell you about it at the end um as with new technology though it is this is beta this is a beta test you're all beta testing this today so welcome thank you very much for the effort uh but welcome to the bleeding edge so sometimes things go horribly wrong uh we're not anticipating any of those months today we went ahead and swatched most of them when we were working through this so I didn't have a music that would have been a good idea

um as with all of my sessions questions are absolutely encouraged so you're under no obligation to censor yourself when it comes to questions about this the only bad question is the one that you didn't actually ask right so no matter what somebody somewhere is going to find some use of it please go ahead ask any questions that you have we can pause we can move on if we have to come back to them later on no worries I'm around we'll make it work so let's talk about incident response it is a never-ending story most of the time you are putting out fires uh it is usually a bit harrowing and a little bit stressful so here's some Basics that you

can keep in the back of your head no matter what role you play in an incident response team if you're even a part of it there is something here that you'll find useful I promise so first let's let's get some common definition going so we have events and we'll get to the next definition that matters so an event is basically just information that's relevant to something within the business something within information security so events happen all the time believe it or not some of you have probably looked at a bunch of these and maybe have forgotten uh if you're brand new to this information security space turn on logging sometime and just see how noisy it gets and just

watch that all flood and take down your hard drives you see these so do so carefully um but with events we lock a lot of the context around what happened so the event itself is just simply an indicator of something happen we don't really know why or how or if it's good bad or even so for that we have to look towards some other definition so when we talk about incidents incidents really dovetail into this intent to do harm malicious activity an event that correlates to some kind of malice or some kind of uh bad actor doing something malicious within our midst uh in short if you click it the secret ingredient is fine that's really

what it is it's it's basically somebody is looking to do something malicious on our Network and the events tell the story of how that happens so for the audience a failed logging attempt is that an incident or an event an event yeah absolutely next what about a malware outbreak yeah you bet what about an antivirus service that's been stopped on a host oh I'll take a poll right because we lack the context around it administrators will stop AV services to install software all the time monitoring tools and malware actually share a lot of the similar taxonomy if a monitoring tool is grabbing your CPUs Services OS information logged on users all that really cool stuff and sending

it to a command and control server or rather their Central server replace command and control with their Central server but you know when you think about it malware acts differently right it would have similarly so it really doesn't what about a ddus yeah that's it right that's there's no no legitimate use for that although funny story I remember uh when Michael Jackson passed apparently a lot of the news websites were experiencing a massive flood of traffic that started to look like DDOS attacks just purely based off of how breaking that news was so I guess it's important to really analyze the traffic the headers the information that's coming into your network so that you can define

whether or not you're truly under attack or just getting some kind of hug to death what about a large upload to an unknown location okay who says incident one two three therefore okay and event yeah it's an event because we lack the context around why that upload happened large uploads to Dropbox happen all the time attackers will also leverage those same services for what that's worth they're free they're available and by and large they're approved within the business so a large upload to an unloaded unknown location should likely be investigated as a potential problem because you want to understand the data that went out combined with where it went to who put it up there and in what

capacity okay everyone come on in all right so when we're talking about incident response we have a few desired outcomes that we want to achieve anywhere within the business wherever we can so preparation is obviously the very first step preparation is mandatory when it comes to incident response if you're an IR team or you're a sock manager or you're some kind of security professional one of the things you kind of always want to do is just sort of be prepared for the nest incident we'll cover how so make sure that you are ready prepared and able to action any of this as it comes out because again nobody's going to warn you that an incident is inbound

unless maybe you're getting a red team exercise so make sure that you're adequately prepared from there nope go back from there uh you want to identify all the compromised assets so this is called getting the scope of your compromise you want to understand all of the potentially affected assets within your environment because you have to move to contain them away from the rest of the network how you contain them is entirely up to you we'll cover that in just a minute but realistically what you do is you find all of the problem areas of the incident you go ahead and isolate them you then eradicate the threats that you found in the process of doing your

investigation and you recover the assets to bring the business back online and then you spend a whole pile of time in meetings with c-level Executives trying to discuss how to prevent this further hopefully some of that advice gets taken into consideration but at the very least you learned something along the way hopefully to avoid this in the future so we talk preparation you'll never rise to the occasion anybody who says they will they certainly won't reason being is you'll only fall to your lowest level of preparation so if all of your tools are missing if you haven't worked on some of this stuff in a while uh if you are new to it and you haven't practiced

whatever that preparation looks like it's important to keep that going as best you can because when an incident hits time is really critical and the more that you can do to move this along in a way that doesn't really impact the business too much the better it is for you so all of that preparation shows up so make sure that your tools are sanitized that goes right hand in hand with your forensics media if you're fortunate enough to have in-house forensics they should already be handling this but it's not a guarantee so it is something to just kind of follow up with whoever's in charge of that if it happens to be you if you're

doing some of that because that becomes really important as part of your evidence gallery uh and then of course make sure you have a clean shirt and maybe you've gotten a little bit of sleep uh incidents take a while incidents I remember locking myself in an office once upon a time ago for 12 14 hours working an incident at a client with ransomware and uh it started to smell like a petting zoo pretty quick so definitely going with it was really good um this is actually a really important one I want to highlight this one specifically documentation available in multiple places how many of you have a documentation repository inside your business that's actually a bigger number than I

thought yeah cool so how many of you have that documentation replicated in different spaces in the event of that Central repository somehow down yeah that's that's more like it so having that that resilient documentation because your standard operating procedures in an incident those are what people are going to be following and those are what everyone's going to defer back to when it comes to your training and your development so make sure that your documentation is available in different places wherever you can make sure it's up to date in both directions whatever that looks like within the business make sure that you're following them so with regards to Identity notification it's quite simple you're going to get it

from many sources so the initial Genesis and event can come from a phone call from somebody through help desk hey I went to this web page and I'm starting to get all these weird redirects and now the request won't complete uh hey I went to go install this tool and the antivirus software turned itself off and now I have this problem so you're going to get all these notifications from all these different directions it's important to be aware of where those can come from and keep an eye on them for any of the sock analysts in the room identification sources are primarily alerts and they're very noisy and they're tedious to follow up so I do

appreciate the work that we do but when you're looking through any of those alerts try and correlate some of them to some of the others if you're seeing some activity on one host and some other activity on another host but it kind of looks like it might be connected don't be afraid to pull on that thread because that really goes a long way but the long and short of it is you're trying to catalog the entire scope of the problem and the reason for that is that you can document where the entire scope is because the one thing with incidents is you're going to run into a lot of panic you're going to run into a lot of

everybody wants to set their hair on fire and try and resolve the problem as quickly as possible finding all of the mess and documenting it in a way that's clear and ambiguous flexible within your incident response plan and available to your incident responders is going to make a big difference whenever you're trying to get this work done especially so once you've found everything we now move to containment so this is where we isolate the assets away from the network so there's a few ways to do it each business is different some people straight up can plug unplug the network cable and Life's good others have to create some kind of special Network routing in order to make it so these

can't phone out to the internet but anything that you can do to isolate these devices from your network is vital but you still need to be able to get in and manage them because some part of your investigation process is going to require you to probably put some hands on a machine in some way shape or form so whether that's through forensics whether that's through some kind of console where you're going through and Mining some of this information depending on the asset we might just move right to eradication recovery where you just pave it for more of it and move on it's all dependent on the business but whatever you do to contain your assets is really really important one

thing though you want to avoid powering down any assets if you possibly can because a lot of the attacker surface lives in memory so you can lose that rent if you power it down and don't you know it doesn't come back on its own so if you can avoid avoid powering it off that's great sometimes it's easier said than done in a lot of cases you might get an incident right I panicked and shut down the computer okay well so in that case you can either move to forensics if you have skills in-house or you can start to work on it through some other way but just isolate it from the network so eradication this is where you're

getting rid of the threat to to the organization so because you contain the threat you you now have isolated all of the potential backlash that can come from it so if your attacker is still latent in the network they might go hey I'm still here and they'll thrash your backups or they'll make the damage worse maybe they'll leak the data whatever you can do to to minimize that is always useful so with regards to eradication you're closing any network back doors you have one that gets missed quite frequently are automated tasks and scheduled jobs cron jobs uh schedule tasks in Windows any automated scripts that might be there so you want to look for some of those just because it looks

like it might be legitimate doesn't mean it actually is so you think about something like a scheduled task that's set to upload a copy of a database to Dropbox I don't know why you'd ever configure that from a business standpoint but let's say that exists you know don't ignore those kinds of things because that might be somebody put that there as a temporary solution and now the attacker is abusing it or the attacker put that there for you and they're continuing to leverage that so always look for the automated jobs very very easy um any firewall rules it's it's not common that firewalls are going to get popped I mean they're they're usually pretty robust

um but if somebody put some custom firewall rules in there that you can't track down make sure that you are what are these for let's power them down and see what it does to the business that's a tough conversation to have in a boardroom by the way so you kind of have to come ready to go with we don't know which rules are ours and which ones somebody else might have put there unauthorized so you might have a bit of a boardroom battle in your midst on that so remove any unauthorized firewall rules if they exist especially if they're too permissive that's going to go a long way and then recovery this is the best part

you get to bring everything back online um apparently that didn't show up but this is just make sure everything looks good sorry um but basically when it comes to the recovery phase this is where you're bringing everything back online one important note that data custodians have to or sorry the data owners have to sign off on the data and the service being properly configured and ready to go before you bring it back online now in most organizations they might defer to the IT team who are usually the data custodians but let's say you have a credit card data breach and the finance department is the one who has to signs off on that make sure that they get a

chance to look at the everything that's in there verify that all the apps and everything attached are working as expected the data has the Integrity it needs before you bring it back online and roll it back into the business so that that step can take some time be ready for that but what it's worth the recovery tends to take the longest in a lot of cases but your backups your resource everything like that um you know that's that's the next part that gets along here and then the Lessons Learned again hopefully everybody learns something in in the midst of the incident so what did we learn you know the post-mortem meetings they're long they're lengthy uh

they take forever and a lot of times you have to repeat yourself just be ready right so if you are the incident responder and you're going into these meetings be ready to answer a lot of the same questions multiple times depending on who your audience is you might have to keep those tidbits very very short uh so just be aware of who you're talking to at the time one of the things I like to talk about is never advocating for dismissal you know full stop happens right so unless somebody is acting truly maliciously like an Insider attack let this be an HR decision this isn't your decision to make unless for some reason your incident response in HR

which I mean maybe see if you can find somewhere that'll let you work with your house so in that case I I always just say these are the facts and I present those in the incident report of this happened how much of it to what assets here was the impact and that's it and I let the decisions happen to other people where that's their entire job so very very important foreign

this is the point in the time where I normally have a break but obviously we're a little short on it so hopefully I'll brought me up uh and away we go any questions about incident response before we carry out yes you are normally after it containment like you're not assess it so you can't have evidence yeah absolutely so that really depends on the severity of the impact and if you can determine who or what kind of got to that if you want to pursue any kind of law enforcement action or if your lawyer or your insurance company say hey keep these isolated until we can send our investigators then then that's it so if you have an insurance provider defer to

your uh coach your you'll you'll receive like a breach coach uh who will help you through that and if they say yeah leave those contained until we can get mandiant or something to come to the forensics then that's the way it goes in which case then your incident is in a bit of a holding pattern so it's important to be able to bring the business back online without those assets in some way shape or form if they've been compromised because it might be a week two weeks three weeks who knows before they can get to you some places will have a really quick turnaround where they have a forensic responder ready with suitcase all the

time but that's big money so be ready yes

who runs the IR none that I've had they all have internal Incident Management but if they are if the breach is severe enough that they're going to be making a claim they refer back to what the insurance provider says and typically the way that they brought their that they got their policy was here's how we handle our incident response and they went yeah okay that's fine with us if it gets to this point we'll assign you somebody in that scenario importance they they act like the incident kind of like a guest incident manager if you will so they'll come in they usually have a lot of expertise a lot of visibility into what's going on

they might have alternative connections to like if it's a ransomware situation they might know ransomware negotiators if you can't get the data back whatever that looks like let them handle some of those you know more nuanced approaches to it and they will advise you as to yeah go ahead and isolate these and leave them that way they'll um but that would have been figured out between likely your Senior Management and the insurance company long before you get to this point [Music] uh as far as the number goes I I actually I haven't seen it at all um I just know that most of my clients they have their own incident handling that the insurance company was satisfied

with again up to a certain point so if the impact is really severe then it's like no no hey we're we're gonna take over in that case because now it's their money on the long term any other questions yeah I'd add um that's a good point yeah actually that's a really good point so business email compromise for anybody who hasn't seen it is basically somebody pops a set of credentials they go into an email inbox they create a forwarding rule usually like Finance or money or wire transfer usually something very specific to you know dollars and cents uh and then any email that goes out regarding that that also gets copied into the forwarding rules and sometimes into the Inbox and

then from there the attackers they just sit and wait until an opportune time and then they get in the way steal the money and then you know way to go so yeah when you're dealing with a lot of your eradication piece look for email rules that either your client didn't create or yourself or whoever you're working with um but just be aware that list can be enormous so when in doubt you can probably just turn them all down and see what happens somebody won't be pleased but when you're working in an incident scenario depending on the severity of it and the impact that happened you might get all kinds of blessings for management to do nearly whatever you

need in order to get that going yeah it's really just a comment

for inspiring um so you make sure to challenge these accounts or phone numbers yeah yeah one of the things you can do when you're dealing with an incident if you see a number you don't recognize hey whatever number this is identify yourself and then make sure the room stays quiet with anything incriminating until you figure out if that person is supposed to be there yeah so out of band Communications that's basically what if your email's down what if your teams are down right what if the conventional methods you use or what if your attackers latent in those and they've compromised that email and they can see you chasing them yeah you want to assume that those might be compromised in some

way so having an ability to get on a phone Bridge that's separate from the day-to-day business that is the the piece that you really want to get to so yeah that's good thank you you got a question yeah I'm sorry oh good question when do you inform your stakeholders uh truthfully it's depending on the impact what assets have been hit so uh there's some organizations where as soon as ransomware hits and it compromises this data they have the clock is ticking they have to get in there and let these people know XYZ has happened early as early as they can in other cases uh they might have a regulatory requirement within like a federal government

situation where they have to or the provincial government or state government they might have to notify people within 24 hours 48 hours of detecting a breach right so the the short answer is it really depends defer to your incident response plan if you have one make it data Centric so this type of data if it's compromised we let these people know right away so sometimes you have to might you might have to inform like a privacy commissioner right away if personally identifiable information is believed to be affected it's okay we've got to start priming that pump but they might ask you to come back and be sure so uh the if you have anybody internal whose

responsibility is that stakeholder management that external one uh hopefully they've thought of that to some extent within the business that makes sense my personal belief is when you have enough information that you can stand confidently by the story that is told in the evidence that's when you can start to inform the impacted stakeholders so if I see a credit card data breach on my e-commerce platform and I or I think it is the e-commerce platform may have had a problem if I think that's an issue I'm not going to run out and issue a press release right away because now I have to fight two stories in addition to an attacker in there so you have to be pretty careful

about how you inform them uh internal stakeholders if there's a part of the business that's been turned down that's going to impact and where it's going to get out so managing that is is quite difficult at that point your your response then becomes more along the lines of hey uh something is happening within the business we're currently working on it yeah we're aware that insert whatever tooling here is unavailable please try to do your work with this alternate or as long as it isn't client impacting you know call it a day kind of thing right that's that's a common way to go by too a little bit more of a comment I find with the stakeholder notification that

is a good idea to have something dedicated to you yeah yeah it sure is and some organizations of considerable size will have a full-time person uh even like medium-sized organizations they'll have a full-time person whose stakeholder Outreach public Outreach PR comms they've gone through some media training like you want to have all of that because brief stories can get out of control instantly I mean we all saw what happened with very public companies very quickly before they were you know when they were hanging out with it too fast so yeah be careful with stakeholder management but it really is very important it's a good question any others it's a good topics it's a sharp group

I'm worried I didn't bring him down [Laughter] cool let's do it uh we'll go back we have one more to go back yeah there we go so we are the stakeholders the illustrious folks of lemon jello Industries uh we are an amusement park food and beverage company with a whole lot of infrastructure nobody really understands or manages well as we are familiar with uh basically the this morning lemon jello Industries experienced something malicious on their external perimeter they're not really sure what but some Services were offline for a while uh and you know we think we got a handle on it everything seems to be back up and running no more resource exhaustion we're still not convinced that we've

kind of you know sealed that shelf so uh we're going to be digging in a little bit more on that it's a very elegant mix of assets we have firewalls web servers databases and various configurations authentication servers and assorted workstations all right I have the biggest IR team I've ever seen let's get to it some ground rules don't block the simulation oh good lord this didn't really show up very well okay well we're playing the shipping formatting um so don't block the simulation this is real right these things have happened uh it's it is all something that can happen to an organization so this is one of those like oh no trust me this is all

rooted in reality but have fun have a good time we're here to have fun right uh as we go along feel free to ask any questions feel free to work together uh I do just ask if you're having little side conversations about stuff if you could just keep it to us a doll Roar so we can kind of hear everybody that's going on um but please Network have fun ask questions dig in enjoy if you have fun tell everybody if you had no fun tell nobody uh yeah now it's game time cool all right so we tried to get the lights turned off correctly but we didn't want everybody to feel like they were kind of

remote person uh so you know I'll read this out to you so today's scenario is rooted in reality but we keep it fun for everyone no prior incident response experience necessary please feel free to ask any questions explore the incident Network and most importantly learn something everything you find interesting today we're living yellow Industries they're already covered from there so the company website experience the distributed denial of service attack so website was down they phone help desk and everything went along uh we got it all up together everything was good it exhausted a bunch of resources though so there was some downtime from it but nothing that a business couldn't recover from they just sort of rolled into the

cost of doing everything but the company's founder Leonard lemon jello the ever paranoid former Tech CEO who has now started his own food and beverage company for Carlos uh isn't convinced that the attack is fully mitigated or that it wasn't part of some other nefarious actions going on so our mission statement investigate this information structure infrastructure find any malfeasance get it off the network and we go so we have asset statuses everything that you're going to see here you're about to see a network diagram of this environment the asset statuses have different colors so whatever is rated purple you don't know whether it's in trouble good bad ugly it's unknown anything in the white has a vulnerable status meaning there's

some kind of Gap some kind of thing in there that might be worth considering anything in red compromised popped we know it to be sure you know anything that we have in yellow recovered blue is our is secured everything's back to Blue uh anything with the black coverage has been scorched meaning we've cleared out the mess everything has been eradicated but it hasn't yet been recovered so if we went ahead and paved over it we did some kind of Corporal bombardment whatever that looks like we've gone ahead and gotten rid of everything but we now have to restore the system support so originally we were going to split up into different groups but that's going

to take some time this is a small room and we're all sort of occupying the space anyway so let's just have one big group of incident responses right let's just let's lean into it so everybody's responsible for different parts of an incident traditionally um but again just for the the interest of time I want to keep an eye on that yeah okay so we're doing good okay um so just so that we have everything uh you know well in hand we're going to be working through this in a little bit more of a measured fashion although we're still going to keep in mind uh as uh again as we go through any questions feel free to ask

no and it's not displaying somewhere any bugs or features I promise there we go just took a little coaxy there we go all right cool uh maybe zoom in a little more yeah I can't really tell from here yeah right so we have a lot of unknowns right

all right so from here we have a series of assorted assets of mixed understanding uh basically everything is in a bit of an unknown State we're not quite sure because remember all we know is that we have the distributed denial of service attack this after this morning so that's all we know about resources back online things are good we're trying to look for stuff so one of the things we can do in identifying actually I'll posit this to the group where's the place that we can look if we want to find a lot of information about the traffic inside of our Network right away yeah firewall logs show me firewall logs pay no attention to the man behind the

curse it's part of that part of the appeal we're working on an avatar that's right that's it yes which is bizarre because we have blue team tool right looked into firewall logs so let's dig in a little further what are we what are we likely to be looking for so remember our management team has come to us and they said I'm not convinced the attack isn't still ongoing what are some indicators we might need to look for inside of the firewall logs that can give us some indication of if we have a problem sorry vegans beacons beacons can you talk a little bit more about those yeah you said that there you are looking all over the place

yeah so a beacon is basically just a little it's a ping that creates the evidence that we have that there is some uh some problems within it so uh what might we be looking for with regards to beating traffic DNS what about DNS yeah absolutely so because DNS traffic tells us where all the where everything is headed we want to combine some of the the beacon activities and we want to see any frequent requests to DNS traffic that really stands up so in this case how far back do we go because DNS traffic if anyone's looked at DNS logs you could be looking at it for an hour and get a billion records without breaking a sweat so how far back should

we go if we experience the attack this morning how far back should we be looking do you think yeah 48 Hours yeah that's fair be there a while but we can do that yeah okay so let's do a 48 hour query of the DNS logs for any Beacon traffic because I bet you we'll find some in there well you would hope and so there are various tools and everything that you can have inside of your organization that really helps do this at scale um bro Zeke and Rita are a winning combination that are open source and widely available so those will work really really well if you're trying to find some of this Beacon activity the

other thing that you're really looking for is weird DNS uh like domains so domain generated algorithms are just examples of domains that have really weird kind of naming conventions to them none of them make sense because a lot of attackers might be a little bit obvious right they use those domain generated algorithms they're hard to stamp out but they can be easy to spot if you know what to look for so when you're doing any kind of mining for Beach and traffic you want to start to look for any kind of the weird and wonderful you can probably ignore the Googles and the dropboxes and everything like that for now yeah if you have a threat intelligence

fee inside your tooling you can absolutely turn that on and away again oh couldn't we oh uh yeah we can do that afterwards what did we how did we do 48 Hours of Beacon strap analyzed one of the web servers was found to be sending out some weird traffic so we still can't confirm or deny whether or not we have anything malicious here but we have a web server that's sending out something that we just can't understand so I'll positive to the group do we want to keep mining some Fireball alarms or do they want to look somewhere else scan the server yeah all right let's go mine our web server so can you zoom in on the web

service yes [Music] oh good question uh so in this case with the multiple IPS yeah you're right like that could be multiple command and control servers for the sake of the scenario and for everybody here yep I'm sorry responsible no DNS flux attacks in this scenario nope but good question did you click okay still up here we go scan the web server asset is found to be compromised so we ran a vulnerability scan against this and we found some open balls and then we went ahead and looked at some of the traffic and some of the connections that went through so we have a busted server yes so if that was required you could wire structure network but be

warned it's going to be a lot of ppap data so in an incident responsibility that's valuable that is valuable work but you have to be okay with looking at a lot of data and you have to know a little bit about what's going on so we have a web server and we have single IPS running through our Network traffic so now we could go ahead and put a peep out on this or that and on this specific post instead of the whole network because if you're doing the whole network like you'll take no time for that to fill up right but now that we have a single host yeah we can absolutely start to do pcaps on that so

is that we want to do you want to put you want to start capturing traffic so what we're trying to do we're trying to capture the packets that are going across the wire from start to finish so that we can reconstruct some of the sessions of whatever might be living on post so if there's any kind of malware or any sort of malicious activity happening that traffic is going to get spotted or you get to see it sometimes in the clear depending on how that might be delivered to us so let's run a pcap for 10 minutes on this thing and see what we can get from that I used to be out here and see yeah

yeah exactly so um here we go way back like car seats so we uh when we talk about IPS and this information looking at what you have available is also really valuable so one thing you can do is you can go to any threat intelligence feeds on the Open Source One virus total abuse abuse IP uh Alien Vault any of those similarly you can also go to Showdown and see what that host might be bringing up then that might give you some hints about the protocols that this could be potentially using so it is a thought process in any direction so we've gone ahead we've grabbed the IP address we went to our various open source intelligence places

and we found that it it doesn't really stand out as being malicious but it doesn't serve a legitimate business purpose either so we know that it kind of stinks out loud but we are seeing some connections between this IP address and a popular browser extension some developers might be using inside the environment but that there's all right let's have a look at our keycaps and see what happened with this here web server how do we do

Dr couch was started on the web server for 10 minutes that's a good long time the other web server webs server was found to be emitting similar Network traffic so we have a couple of other web servers that are showing up similarly yes can we contain them now yes if we find something and we move to contain you have to be careful though so in this scenario yes we can but if we're worried that the attacker is still latent in the network or for a very large organization with a lot of resiliency containing this might be a little bit more difficult so you might want to come at some of that containment in a bit of a measured

fashion so you don't accidentally tip off your attacker that you're on to them and then they ratchet up the attack in this scenario yes we can go ahead and contain so do you want to contain those two web servers yeah let's go ahead we'll kick them off the network so uh when we go through our firewall Network configurations instead of allowing these to Transit back out into the internet and back into court we just simply say hey firewall anytime these two servers are talking you go ahead and sinkhole that traffic let's see how we did web servers are isolated from the rest of the network what's up okay so these two have been contained which means now the spread of whatever

was in here has been Limited but attackers only ever hit one or two spots right yes so knowing that we saw some malicious traffic on our web server think about e-commerce systems and how some of them might be configured e-commerce systems typically have a web-facing internet presence that is shot at all day long and they might have specific Pathways into the environment in very very specific connections think about maybe a database of some sort what do we want to look for there I was going to say should we should we be scanning the backup and then trying to isolate it in case it's still spreading you want to look at the backup service well in case I mean if it's clean and

this is still uh incident and it's still spreading yeah keeping your backups isolated would allow them to not be compromise that's a good thought yep what's up you want to check the database server I like where you're headed we're going to check the DB server and then I figure it out yeah okay or if there was some other indicator yeah that's a good idea how about we start with the logs on the web server because that will give us because these are public facing they do get all kinds of people messing with any SQL injection everything like that so in this case yeah we definitely want to consult the logs we also have to be aware that

there's probably a lot of noise in that so we got to know a little bit about what we're looking for so let's consult the logs for the last 48 hours and let's see if there's been any kind of malicious attempts at injections or anything with that survey says

logs from the web servers from the past 48 hours are analyzed communication was found between the backup server so we have can we have comms that would help with that that also uh lines credibility to our database problem because if the databases are backed up from the same servers that might be worth considering so let's go ahead and investigate the database servers while we're out true so if we can zoom in on those where did we go there we go so we have a front sql1 it's a mirrored database with uh with SQL one and SQL two so mirror database is basically everything that holds true on one it's going to hold true on the other so prod SQL one is our

primary database for e-commerce information e-commerce being things like credit card data personally identifiable information anything else useful in that regard so because these systems have communications with our web servers it's absolutely worthwhile to go ahead and consult we'll service equal one because you can't look at all the servers all at once uh unless you have Tooling in which to do so and lemon jello decided to buy printer ink instead of bonafide sequel so we're stuck dealing with doing this piecemeal um because you know that's just the way we go so let's take a look at product SQL one and see if there's anything suspicious or weird or showing up that might be addictive of potential compounds

we found a bug we might have this they're with us we killed sequel one

scorched down yeah oh boy looks like two has been busted too yeah one one in theory and it looks like it disappeared cool what we're finding all kinds of good stuff there thank you everybody for that I really appreciate it yeah it's it's vital yeah watching secret one we're analyzed and the login was found from a dead account oh so we have an account that should have been disabled but wasn't it wasn't accountability where we checked out yeah we don't have a sense yeah we do have authentication servers so we should have two domain controllers they are paired together down here there we go so to the main controllers they are paired so again what happens to one

will happen to the other so in this case because we're using uh an account that shouldn't be active we now have this situation where yeah we could probably reason that or domain control so we can go ahead and take a look so I find them takes moments to scrub through these logs that's it take some time to start through the logs um uh we have a turn counter we have a time that keeps sticking out right just to keep everybody kind of on their toes so as that time really ticks up um that's where the expense starts to come in for the business overall so yeah we do keep that in the back of our heads

because this investigation effort does genuinely take time it might take you a few hours to mine through all of the log data Sim tool and do it correctly in a way where you can point to here are the problems let's go ahead and isolate these so uh that's that is an important consideration so how do we do on our SQL mining on the domain controllers or on the domain controllers we know SQL is bad so many controllers both contain the data comp and access the accounts granted by the controllers okay so in this case we have some accounts that were managed within the domain controller environment um but in this case the the DC didn't

actually issue them where might these be a problem where might these accounts be a problem with regards to the web servers local accounts yeah like local Avenue right so that is something to keep in mind with incident response just because your attackers didn't actually get into your domain controller environment and create new accounts doesn't mean that they didn't necessarily create any problems down the line so local admin accounts are all over the place inside of an environment they're everywhere and most of the time the passwords haven't been changed since the environment was stood up and for what it's worth yes you can't turn those down in most Windows environments so a local admin accounts were the main

culprit of this in this scenario okay uh okay so let's take a look at uh okay so in order to make controllers are good where so we have some web servers that are compromised but again we're still looking for malicious activity so one of the things we have to keep in mind is we have a whole pile of workstations under here and some of them do different things some of them belong to developers who have the ability to communicate with nearly everything some of them belong to system administrators who have the ability to really communicate everything and do something malicious and others uh well they're just simply workstations for people who maybe just need to put

some ones and zeros into a computer uh so what about our system administrator workstations would it be worth investigating any of those why would we go looking for some of that privileged accounts yeah that's a very good point yeah because we saw local admin account was used on the controller controllers or another so let's take a look at our uh at our um system in workstation I can't remember which one it was but then there's workstation four was Finance okay yeah then yeah whatever the admin the sysadmin one was manager nope system yeah uh prod workstation too always consult your network map always consult the network Maps so we're going to take a look at our

system administrator workstation and the reason why those are important is in a lot of organizations especially uh like small media business the system admin is responsible for everything they have the the real keys to the organization uh in some cases if those ever get compromised or if some of the activity uh that an attacker can snip across the wire you know it's in the Insight they need into your business that definitely creates a problem so you want to make sure that your system administrator workstations are investigated at some point knowing full well that if they are compromised you're going to have a serious problem because anything the admin can do potentially your attacker could do

a video survey says [Music]

vulnerable yeah so there's some vulnerable so I mean we carry the community in some capacity yeah really uh in this case

what's that we don't nope there's no EDR Tools in this scenario right now again because you know it's a startup and they didn't really have a lot of budget to spend on it and the printer ink is kind of what consumers oh so we have a local admin account from this workstation uh-oh so for all intents and purposes our system workstation has been compromised so I'm going to shed a bit of light on how this might have happened there are no shortage of applications that we leverage in our day-to-day within a business and no uh no exception are the browser-based uh tools like extensions and everything else so uh one of the things that you want to keep in mind

when you're dealing with any kind of software inside your organization is this understanding of these are the tools that we have we're aware of who owns them and who manages to you know keep them up to date what they do what assets they touch what data they have in and everything else associated with it so this system administrator here their workstation has been compromised so what what should we do next now that we have a very very high value Target that's been hit what do we do access device

yeah so on assist me computers we can go ahead and do a local log query in this case because that's what we have access to so we run some local logs and uh with the specific account information within the time frame that we have and that gave us some information uh that was pretty useful which was essentially this account this workstation has gone ahead and started doing a lot of reconnaissance activity uh kind of all of a sudden so we now have a situation where this one is doing a lot of research into the organization uh without kind of the permission that it needs to or in a way that's really outside of exactly so we gone ahead ran some logs

what's that we know it's not an internal correct nope this is not an internal threat so when we talk about Insider threat say you do have to be careful especially if it is an administrator who's gone rope I promise you I wouldn't subject you to that uh so no there is no Rogue administrators in this case so how did our log career do few minutes

well you have to sort of look at patterns of behavior along those lines so that's sort of where you have to get HR involved earlier and say okay have there been reports of this person being disgruntled um if you're doing any of this kind of Investigation uh has the have the managers reported like a drop in performance um is there troubles that maybe aren't public to the rest of the business but somebody in HR should know so at that point you start to look around the patterns of the person itself um If you experience more incidents than you really think you should is another way to look at it are you finding a lot of data leakage on a very regular basis

same idea so if you're starting to see a lot of really anomalous patterns in people's behavior and by and large the infrastructure that's a good way to determine that developed yeah fair enough right Sigma back in uh and where you go uh so the identified access logs have established that an attack timeline other workstations were found to be communicating within this time frame with the administrators and shootings ah so the plot thickens so we have an admin workstation and we have an IT workstation and we have a developer workstation so to shed a little bit more light on this we use utilities again all the time in browser that do something in some capacity or another but assume that

everything the browser can touch has the ability to impact your infrastructure that really is a big part of it so um we have some workstations that need addressing which should we do next and we have a we have a compromised backup server our database server is still around our web servers however have been contained yes work students we would want to isolate those but now what happens if these people are also working on the incident yeah that's it yeah that's sort of where you bend over to yourself in the foot in this case we can give them fresh workstations so our administrators do get to continue working through this so let's go ahead and isolate these three

workstations we'll kick them off the network in some capacity or another so one of the things you can do with workstation isolation is make amendments to access control lists and vlans not every Network should be flat hopefully so basically when you have these segmentation options in your environment you should be able to say hey anything from these workstation areas instead of talking to the rest of the network you now talk to nothing so that's called Reliance segmentation as an option or Access Control list if you have them if you have that handled by your firewall or basically hey you can't Traverse out into these various parts of the network so how do we do on the isolation

workstations have been removed from the network but left powered on why is that important forensics yeah absolutely right so in this case we want to go ahead and image these down the line to figure out what's really happened for the sake of the scenario I'll go ahead and you know fill in some of these blanks here to go along so uh the developers were using uh browser-based extension tool that was just fine it worked out just fine until one day somebody malicious made a better offer to the person who kept it in control and now it was being used in essence as a botnap so a lot of the distributed traffic that we were starting to see came from affected hosts

within this browser-based environment so because this went rogue because this was evil a lot of developers of these tools will intentionally Harvest a lot of extra data without you knowing in some cases so that they can run Diagnostics and everything else on their various tooling so this is why it becomes really important to kind of manage the assets and the underlying tooling that you have in the environment because it's in that capacity in that blind spot that you can actually really get a lot of various activities so these were actually the source of the of the bridge was these three that were the problem so we have three workstations they're successfully isolated but we still have a database

server and a backup server and we still have uh potentially some other workstations that we need to investigate so in in our game we will fire the fire the ability scroll up not selling this oh it's on that side so we have ability the abilities do something so in this case our uh Mr lemoncello's Chief security officer knows a person who owes him a favor uh and one of the things that I saw happen in a live incident a CTO made a phone call and then all of a sudden somebody who didn't work there simply descended into the channel and started coaching us through this and then vanished so basically uh the way that we we built

these up these are non-player characters that have some purpose to an incident again because we couldn't put all of these in just for the sake of everything um we went ahead and just put one in for now so birds of a feather is an ability that typically our chief security officer can play so remember when I mentioned the networking piece where everybody can hang out and know people and gather some favors this is one of the reasons why so if the business is comfortable bringing in somebody who is external to the business and willing to kind of keep a good secret we can go through that so we'll go ahead we'll see if there's anything that we might have

left behind in our investigation process

yep that's it and it's as quickly as he arrived has vanished Into The Ether and uh and away we go so remember we still have some we've contained these we've continued our workstations uh we have a database server that needs some attention and we have a backup server so remember our backups are not compromised take me through what we might want to do with that I have some thoughts but I'm willing to hear from you they are compromised yeah the backup server has been compromised and as a result of backups yes

no because those are expense those are expensive we have a consultant who said we should have those questions so so the report says ah yes so that's really important to consider if we can roll this to a Noah the last known good backup that's really really valuable and really helpful one problem that I've seen with this though some organizations might not have been aware that they've been compromised so because this was a browser extension that suddenly went back we've been using it for quite a while so we don't really know the last known to get back up uh and in most cases I've seen uh ransomware accidentally get backed up too yeah and that's always a really one

conversation to say sorry uh hopefully all that data is still there somewhere um so the the backup server we'll go ahead and we'll contain that from the rest of the network because still it's still there anything malicious is still happening and remember the attackers were messing with us with a DDOS attack earlier this morning so they're not necessarily as interested in our business as other businesses but we do represent some kind of collateral damage so let's see how our backup isolation goes

there we go I don't either one moment a little momentum yeah that's how bad it went yeah that's right yeah it became self-aware no it's our timeout oh sorry timeout issues

what oh birds of a feather trigger wow [Music]

okay so our our workstations are isolated we have uh our web servers successfully isolated we have some database servers that still needs some attention uh and as we took a second look at it we actually have a couple of file servers that are showing some problems too so this really got out there this impact really got out there because everything the administrator was able to access and do work with genuinely was a potential Target and because the attackers were harvesting this data in a very surreptitious fashion they now managed to you know go ahead and do whatever they had to do in order to prolong the attack so we weren't necessarily the sole Target of

this but we were simply collateral damage uh which is a problem right just because you've been hit doesn't necessarily mean you were the main target you might actually just be used in this convention there so uh let's go ahead and we'll isolate our backup and let's see how we do here I believe that consumer was isolated was it is

[Music] bad right there so the distraction scenario in this case is the DDOS uh was there to keep our security teams busy fill up our logs and hide some of these tracks while they harvested data exfiltrated it out into the the nefarious Nether um and again just you know overall uh malicious activities so our backup has been isolated our server is now contained we still have a SQL box that needs containment as well so let's go ahead and roll that one good morning

so we took SQL 2 off the network uh and I think was that all of them I think that was all of them oh no we have two file servers too yeah let's so right oh so sorry two thermal services so terminal servers for anybody who hasn't seen it they can host a whole series of applications in there uh those applications are largely available to everybody depending on how the server's configured so again anywhere the administrator could go and make configurations there are some possibilities and because multiple people had access to the progress pension stands to reason that those could be there too so let's contain those those servers

got internal servers off the network so we're good and I think that's everyone yeah okay so when we uh when we talk about eradication remember we have to close all of the back doors so we've gone ahead we've contained everything we've identified the whole scope of the breach we have to contain all of the assets we've gone ahead and done but now we have to go ahead and eradicate everything so in a traditional sense uh that would mean finding any malicious backdoors scripts anything that have been deposited by the attackers um in our case uh just for the sake of the scenario basically because it was just a browser extension we now realize that these three were sort of the main

culprits so with regards to workstations most organizations will take on a bit of a pave and replace uh idea because it's just simply easier it's been re-imaged we know these images are secure everything is good we can't really do that with servers all the time some certainly you sure can others not necessarily as much because that data might not be as resilient in some cases the backups are certainly corrupted no Goods you can't always restore the notes um so when you when you go to do server rebuilds and in the eradication phase or sorry recovery phase you want to make sure that that you know goes ahead uh as swimmingly as it can so any configs that

you can save anything that you can do that's gonna be good so let's go ahead and pave and replace our our uh administrator workstations there or all of our workstations let's nuke them for more of it

foreign

cool how do we do on the on the pavement of the place well they've been paid our data was not saved and personal preferences will need to be rebuilt so uh there was a bit of a collective screen that came from uh the various parts of the administrator

uh now we have our servers we still need to eradicate some of these but remember we can't sync the paver replaced Liberties especially not our backup server so what are some options that I have available to me to potentially get a backup server known clean known good keeping in mind on my backups aren't as good not on the servers but at this point maybe I missed it but should we not have been locked out that yet account and check that no new accounts are created yes that's a good point so one of the in our hurry to get rid of these things yeah we forgot the actual Genesis of the account and to remove and erratically

that to begin with so that really uh when you when you have those indicators yeah you absolutely want to go for that but that account was actually part of the local event configuration of that administrator workstation so it kind of went but yeah so not but absolutely so if you're using local admin accounts using any of those uh you want to make sure that those are changed if you can lock them out entirely or remove them that's great but yeah disable make sure that they can't do any further damage absolutely uh okay so with our backup servers what are some options that we have available to us that you think we could do to rebuild some of those

so we don't really have an uh it's not a resilient cloud backup necessarily but some of the backup can fix were stored in the cloud so we have to rebuild the server the operating systems and everything else but some of the configurations that went into it we managed to keep those pretty resilient uh so let's see how that went how did that how did that go

so now we now our backup server is in place so remember we have these other application servers uh and our terminal servers here so we did manage to find some known good backups but to presumably hidden under somebody's desk and a hard drive that nobody really talked about that happens all the time so it's never a bad idea to start mining the rest of your organization to see if somebody accidentally took back up in some capacity or another uh or if you had a well-intentioned citizen you might have put you know a demo system in place and managed to keep some of that going so because we had some evaluation of a cloud service and we had some of that in

the capacity our backup servers now up and running so now that we have a backup server going what would we do with our remaining affected assets here we still have to eradicate them so what would be what would we start to look for knowing that we have these problems that are compromised

protect the vulnerable ones for sure what would we start to look for for some of that yeah we have two terminal servers uh two web servers and a SQL box that are still needing to be repaired and brought eradicated and brought back on them so we have a working docket server and for the sake of the scenario some of these will have actually been kept through that process as well so we could leverage the existing known good backups again provided that we can confirm ah did we patch the vulnerabilities right so in the process of investing in your terminal servers we actually found a couple of vulnerabilities that could lead to it the attacker didn't exploit

this but do we want to bring a vulnerable system back online after a fresh incident no probably not so why don't we go ahead and Patch those two servers before we bring them online now when we talk about patching in an incident you know one of the things you have to be careful of is any dependencies or any critical vulnerability or critical libraries that might be used in the business so it isn't necessarily the it is advised to patch for a weekend that certainly is but just be aware of how that might impact a business if a patch undoes something or reduces some kind of compatibility issue just make the business aware of that so that when they

come in Monday morning why doesn't this work anymore and now you already focuses are having a bad day so how did our how did our terminal server patching go Secret attached okay so uh after this we have our web servers and a database server that needs some attention as well so database servers because we have a lot of data in here one of the conversations we need to have is stakeholder management right so the data that was affected is potentially credit card data from customers potentially personally identifiable information from you know the various uh customer transactions and everything else associated with it so one of the conversations that we need to have with the the various stakeholders

are you have people like our compliance folks for any card compliance people like our PR who might actually have to do some Outreach and say hey we experience the breach it impacted some of this data we're still investigating figuring out the root cause but you know just so you're aware keep an eye on all of that so because we have that credit card data breach we do have some concerns we have to notify performance yeah absolutely so let's go ahead and and eradicate SQL 2. to get that one back online SQL one disappeared yeah it's a little bit uh hidden right now yeah okay oh it's there okay cool well JavaScript tools

yeah take a look at SQL two let's see if we can get it eradicated and brought back online

a couple of days of advanced a lot okay well so keeping in mind a couple of days of transactions that could be for Millions right that could be that could be Millions many millions so this is a conversation you want to have with the business as soon as you can okay we brought it back online we lost four days of data well that just happened to be our big sale well all right invest in better resilient backups and whatever we can do so this is where that conversation of uh security spend and security control costs go in nicely to this impact of the business so let's let's bring our web servers back online too sure because every

minute that these are deaf nobody's buying stuff

we want that to work again disconnected

yeah so you know we've gone ahead we have a single box we have to bring back yeah that's authentication please oh it's beyond vacation yeah that's fine all right so we had a timeout issue yeah so in the pursuit of Grant we our our web servers fortunately we backed those up regularly uh which means we did have some lots of good ones hanging around the database itself yeah that's gonna be a conversation with the business that we have to have it's going to be really uncomfortable um but the web servers yeah we back those up every every few hours so we have livestocks web servers and Dropbox cool okay so officially the environment is back up and running we're all good to go

we're good for business life is good so let's let's uh let's see how we did let's let's end that let's have a look there we go so these graphs apparently are a little hard to see one of the things that we do keep track of is asset costs cost so as we build out a scenario we then take into account what assets uh are likely to be and how much money this costs to cost you to keep it down so as we see this uh taped up over time really it's not explaining about it it will add dominance yeah um some of the other events that we bring into this uh communication so there are status updates and

Communications that have to happen uh we purposely excluded those just for the sake the size of the room and everything else that went with it but in a traditional incident scenario constant communication with the business the stakeholders that are affected and involved in your incident response that's really crucial so we keep tabs on those as well as any events and everything that happened in game uh and because this was a compliance impacting event where we aren't attesting the PCI yet we actually don't have to technically notify them however our acquiring bank is going to come back and say uh Hey so we heard you had a breach and um here's a new compliance standard you have to assess too and by the way

the insurance for the service that you have might now start to go off as well so it is important to keep in mind that some of the compliance affected stuff is going to turn into uh any questions uh at this point a little presentation

uh if we had to do this kind of like uh kind of the point defense the initial uh on the initial assets yeah that would probably take you the better part of a couple of days to get that done and especially because the the business is down and it's affected by this breach um and then uh understandably you have some senior leaders tearing their hair out you have to spend a lot of time kind of running the the human element of incident too so a scenario like that again depending on the tooling that's available and the skills of the incident responders probably could clean that up in about a day if you had everything lined up nicely if you didn't it might

be a couple months in some cases before that that all comes back online in the way that it matters um so the short answer is it depends but that's where your skills your tooling your training and your processes really come in so let's debrief this is the Lessons Learned part so it's uh I appreciate the patience on the the buggy nature of the new game it was we did some play testing but obviously there were a few things we still got work done so I appreciate your patience on that thank you so much um but one of the things that we'd like to Advocate is practicing these Concepts however you do a tabletop exercise between you and your business that's

really the core uh but just practice because as you can see these things take time they take effort they take energy and they take those skills to be refreshed so make sure that you're practicing them regularly given that this was a simulated incident no real business impact happened but this is rooted in reality and incidents can come from any part of your business and you know you just have to be ready for it so keeping your teams kind of you know well hydrated and capacitated a good way to do this but the cost and the impact of the incident that in and of itself that can really depend because if you lose your entire credit card database and

that data has to be rebuilt that's expensive if I lose a week's worth of transactions that can be very expensive if the penalties from that come down the line that really Stacks up too so in some cases a breach can actually shut your business down and in other situations uh you might lose up to 20 of your customer base maybe even more than that so it is important to keep those things in the back your head as you're working through this because your stakeholder management Communications efforts ongoing you want to make sure that those all align with what's best for the business while also trying to get everything back online and protecting the data as best

yes all right so everything we couldn't get to uh so we have a very full featured game certainly didn't look like that right now but I promise you it's there uh some really cool stuff is in uh is in game and has been built so we have an asynchronous game uh event instance so everybody could log in and play with their phones or with their their laptops or whatever else so if you have more time that we would have been able to troubleshoot a lot of that connectivity and everything else that went with it so uh in uh private sessions you're able to actually dial into a shared instance events pipe to you and you have things

that need to happen that you get to move along uh in the game as well we did put in some in-game messaging so yeah we will keep lid on the trash talking uh between the very scenes No this is an opportunity for marketing to sit there and talk about whatever it is you got going on um but we also introduced an office worker NPC and this is designed basically as a way to keep firing tickets into the incident um because that will happen one of the things you have to keep in mind is your business still needs to run the lights thing you need to stay on the water needs to be running so making sure that

you have a process in place for you four work on this incident you make sure this stuff gets dealt with anything of this capacity address it immediately anything else tell them to wait till Monday right so having that ironed out is important so our office worker NPC is designed to Showcase that those conversations do need to be happening we do have automated attackers inside the network that is available with the blue team mode just because we didn't want to leave that late into the network that's a evicting uh an attacker inside your network he's quite difficult so if you wanted to make sure that that wasn't anything to worry about and of course robust defense and threat mechanisms so

basically what we do is we take in security information including the controls that are associated come up with a number for how defensible that that asset is and the attacker now has to rise above that in order to be successful in compromising so a lot of really cool mechanics in there and because it's a very small footprint for infrastructure we can shove this into any environment basically we can run it on a potato chip so some cool stuff on the way uh we have uh the the mini game the meeting process so we're gonna be putting together uh anytime something happens in Game like a roadblock as an helpful help desk paint hands out domain and credentials to

whoever calls that's going to be a mini game for your department managers your compliance managers your itos leads and your I.T folks to start working through in a very specific measured fashion before the game can progressed so we've got mini games on the way uh we're actually looking to do some live log adjusting too so now we can actually take some live log data and replay a scenario through the game so if you had a red team test and it didn't go well we can have a tabletop associated with it with all of the logs and see where the process might have fallen down along the way and of course more styling artwork and Bug fixes along the way

but so in the interest of some Shameless self-projan uh this is what we do this we we play games we have fun there's a lot of cool stuff coming along the way uh so I started this besides 2020 was uh this was the first Workshop uh that I'd ever done and I was furiously coding this prototype app to try and make this game sync uh because I had this idea when I was teaching in gaming card compliance uh at a local uh College here so I got really good at teaching people how incident response is supposed to work especially Fridays at four to six pm um so yeah I put together this app ran it through besides 2020 had a wicked

good time doing it did again in 2021 here we are in 2022 it's now a full-fledged business uh so this is a service that we do offer uh we also do custom game development for your business so if you have a training solution that you need to have gamified we can bring that up and it works really well uh incidentally to show off all of the the cool features and everything that's been built we'll be doing a tabletop Tuesday where we're going to do it we're gonna have a tabletop exercise virtually we'll have it reported and if some of you would like to join please click here and reach out to me there uh if you're in an organization that might

be light on information security Basics as in we have no information security artifacts to speak of uh no problem the act of onboarding and the the work that we put into building your game instance also coincidentally can be dumped into artifacts for your security program so not only is your training immediately useful but you actually get really tangible outcomes for your security program down the other end so we're more than happy to do that so feel free to reach out contact us you can reach us directly info at syntaxsecurity.ca uh we're on Facebook for professionals here uh and we also have our own website security-selfawareness.com so feel free to reach up have a conversation love to

hear from you got more stuff coming lots of cool stuff so stay tuned all right any questions excuse we've got you've got cues I got A's

that makes more sense I should have done that I should have put that last Ed

awesome yeah yeah that's a future feature where bring your Red teamer and your red teamer can control and stay late in the network of somebody that's yep yep yeah so if you have somebody even it doesn't even have to be a red team or specifically if you have somebody in your sock who's starting to do a little bit of hacking on their own have them do it yeah see how see how you stand up on that yep absolutely that's a feature that's coming it's on the way yeah but that's uh that's very much a plan so uh that would be available to organizations who choose to to play the more difficult version an extra little bit of yeah

yeah absolutely and we would make it so that the messaging can go back and forth so if the attacker was really having a good time they could trash talk in there as well yeah absolutely that that the the Plumbing's all in we're just we're connecting with it

and generate an alert accordingly yeah the post-mortem activities actually they do happen after the game so everything wraps up hey you know and then that's where we start to have the conversation of yeah if you had these in place because it takes in your security program and it's in its current state uh if you don't have those tools engaged but you want to evaluate them we can add those in so you can actually see you can play two rounds one with and one without and we can see how well equipped that is and what that would have done to the business absolutely and then those conversations become very material because it cost us 100 Grand but it

would save us 10 million that's that's a no-brain and if anyone can find logging for 100 grand let me know

you know I actually I have no steam releases planned um but I'm going on an arcade that's going to have a bunch of free and browser games as well uh that teach information security in a tongue-in-cheek kind of way um as far as this being released on Steam and I will think about it we'll do some kind of consumer version of what that looks like I think actually just leaving it in browser is probably going to be working because not everybody has a steam client not everybody's allowed to play games at work but hey if it's a browser and we're learning something while that's a whole different that's not we're not playing that's we're learning that's a good

thing so yeah I think what we're going to do is we're going to put it in a way that people can consume it in a way that's that's useful without kind of giving away all the seafood sauce any other questions

yes

did you learn something about incident response yeah good oh cool yeah us ux is important really important can you correct them yeah yep so I take the position if they have an incident manager I become the omnipotent incident person um but if they don't have an incident manager then I take on the role of incident manager and I help guide everybody through all of the steps so nothing does get lost yes so if you uh try to recover the asset before you get rid of the threat then yes the threat is still in the environment and then you have the situation where they're just going to re-attack you um so we don't we do go

through the steps in a very measured fashion but every business does handle it very differently based off of the types of assets that have been uh compromised so small handful of workstations yeah just move on with our lives get rid of it and then away we go but some of the more critical servers or some serious information or pretty damaging breaches they're going to go through those steps in some in that similar fashion but again each business has unique incident response plans um some businesses have no incident response plans that's a Common Thread um so really it just boils down to it's kind of an ever-evolving problem the business wants to get back online right now because they're losing money

it's a tough conversation to have as like hey the attacker's still here we have to do something about it um so yeah everybody does it a little bit differently but for example I couldn't get them to shut down a web server at three o'clock in the afternoon that we had confirmed was compromised because while it isn't compromise so badly that we're losing money so let's wait until after we're done making it and then we can shut that was a real conversation I had to have and sadly a battle that I had lost so it's it's just the way it goes sometimes but those are the that's that is an optimal way to do it it's a it takes a while it's slow but

it works

no there's no bad questions today they're all good ones um

the individual wandered how they do their investigation about like what first or yeah yeah so the short answer is companies will have uh it depends they will have a mix of both I was once very firmly named like name and phone number in the incident response plan and it was basically if evil call AJ that was just sort of how it went and I didn't have a weekend for about two years so uh it is that is how some of those are built that's not a really effective way to do it a lot of incident response plans should include who to contact into what capacities how to contact them in a lot of situations um and what types of you know how the

the incident is supposed to be managed so should be very unambiguous who's in charge who can tell people what to do where is all the information where is all of that and then evaluated correctly we actually we can code those into the game as well so if somebody is looking to test their IR plans that that is the design so we can take all that information uh security a security person we'll look at it first and see if there's any holes because we wrap the scenario around the gaps in the IR plan as well as the threat case that be facing the industry so we really take into account your information and your your security program in its current

state build you an instance of it and then the attacker has found their way through it and we can see where the resiliency is and how well your team performs any other questions yeah you guys are working with organizations do you find there's also outcomes and maybe helping the business plan or Budget on things or is it more just an education both the short answer to that is both so one of my clients their marketing team actually beat the threat intelligence feed so they found something before the threat Intel feed after coming to the training which like I'll take it that's really cool um but other clients that I've had we we found some pretty severe gaps and as a

result I actually said I'm going to hold back this report until you have a plan because I don't want to fail you on this I want you to know that you did well you just have this problem and then so they went they put them in the risk register they had plans ownership and some of that underway then they got the report that said here's all the problems and then an asterisk that said as of the time of this report this is now being actively addressed so a lot of organizations will lean on the outcomes from this game to figure out where the clear gaps are are we communicating with our partners well how do we even reach

out to them if we're stuck what does that all look like so that is yeah the outcomes are really quite real and they do elicit real change in the business plus marketing is into it so I'm good that's fine any other questions yeah have you guys looked at any kind of silver and mssp a blended environment where say you have a client and uh mssb yeah you have it built in so you guys can have both both entities existing and handling their yep so the NPC section that's where we put your vendors so if you have vendors that you call under certain pretenses they go in there and if you're not calling them when you get

to that certain impact that becomes a gap in your response plan because not everybody knows if evil is this bad call your MSP and the idea being if the MSP wants to come to the session we're more than happy to have them but most msps are a little bit hesitant to do that because for insert variety of reasons here so it is one of those it is an option nobody's taking me up on again

anyone else cool well I'm still here I'm sticking around so if you have any others that you didn't want to ask publicly no worries I'm going to cut you all loose uh a little early just to go let you enjoy the con thank you so much for your time your attention and your willingness to be paid the testers very very much appreciated we have a lot of stuff in our tracking log that we're going to go through so wonderful to have you thank you so much