10 Things I Wish Every CISO Knew Before an Incident: A View from the IR Trenches

all right we're ready to go this is our first talk of the day and we're running a little behind so we're going to get going but please wrap up your conversations and give it up for Patterson cake presenting 10 Things I wish every CSO knew before a cyber event cyber intimate have you ever considered the Peril of choosing a walkout song if you choose a really Kick-Ass walkout song by the time you get to the stage the audience is thinking I wish he would shut up and sit down so I can listen to the rest of the song for those of you who are unfamiliar that was uh that was Mick Gordon mcgordon fans anybody okay we got a couple rip

and tear Doom in any case again thanks for your patience this morning as we get started flexibility one of the things I love about b-sides is how casual it is to be really honest besides is among my favorite security conferences to attend low-key fun low-cost high quality all the really cool things about the cyber security Community you look around the room and these are people who live in your neck of the woods uh very cool stuff touched on the sponsors at the outset I would also encourage you a ton of time energy and effort goes into making this happen you see somebody with a purple badge shake their hand and express your appreciation for all the time energy and

effort that goes into making this happen it's huge huge and I really personally appreciate it you with me okay sweet excuse me so again uh this was my my concept slide once I chose my walkout music this was not part of my cfp so we'll go back into the more boring actual presentation material today I want to talk to you about 10 things 10 things math Wizards Among Us the 10 things that I wish every CSO knew before a security incident occurred you don't have to be a CSO it just made for a catchy title don't you think I'll talk to I'm going to ask you to use your imagination just a little bit a couple

times throughout this conversation I know it's early yet but work with me the first thing I would like you to do for me is Imagine that you know that you have two weeks to 30 days and you're going to experience a major security incident your organization is going to have a breach sometime in the next two to four weeks what are you gonna do what actions can you take are you going to rapidly roll out micro segmentation in two to four weeks Implement zero trust intuitive actually it'll take me two to four weeks to figure out what that means but anyway what are you gonna do you might think I'll go to chat GPT chat GPT put

yourself in the role of a beleaguered security professional understaffed and underpaid I know I'm going to have on outbreak major incident in 30 days what do I do update your resume yeah what I want to talk and talk to you about today is some practical things that I think you can legitimately do in the next days weeks most of which is free or almost free and fairly low effort tactical practical things that you can accomplish in a short period of time that can help you to be prepared if and when that day comes and you have a significant security incident just a tiny bit about me I I've spent the last just about two years in an active incident response

digital forensic and incident response for an mssp called a verdium I have spent most of my days nights weekends responding to active security breaches very very very challenging I often wonder why I do it to be totally Frank I have two points on this slide one of them is that I am by Nature a tactician when you call me or my team out to help you respond to an active breach an active threat actor engagement you do not want me to wax philosophical let's talk about your Five-Year Plan that's not my bint and so that's going to color the entire remainder of this conversation I am absolutely intent on doing something right now to improve your security posture

especially in the face of an active threat number two IR is a weird thing one of my favorite things about incident response is the clarity that comes from a breach we talk regularly about risk risk his likelihood times impact as a general rule How likely is something to occur and if it does occur how bad will it be how do you calculate likelihood a little bit of magic a little bit of guessing a little bit imagination when I come on scene that question is answered How likely is it that something really bad is going to happen 100 it just did and with that comes this Clarity this intense Clarity that I thrive on and that is the leadership in the

organization is like what do we need to do to make sure this never happens again how many of you in your job role would love to be asked that by senior manager that is one of my favorite things about the job and my hope today in a brief period of time is to channel that and to help you see through that lens so that you can take action a few things that you can do practically to be prepared to set the stage I want to talk just a little bit about the current threat landscape now I have a narrow view of the current threat landscape s majority of the cases that I have worked in the last couple years have

been fairly local so this is the East Tennessee threat report unofficially curated by me just a couple of quick tidbits again to set the stage for what we are up against and again some actions that we can take ransomware who's heard ransomware yeah whatever this is boring right everybody talks about it all the time there's really nothing new and exciting on this front except for that it gets uglier and harder all the time we do see kind of a dip in ransomware attacks in recent memory but it's I think it's just a lull starting to pick back up again significant decline in payments for lots of different reasons I think we're getting a little better at backups we're

getting a little more skeptical of making payments there are of course Federal restrictions and regulations which just means that the threat actors are playing dirtier so double extortion of course is just the word of the day I'm going to encrypt your stuff and I'm also going to steal it and I'm going to publish it and I'm going to embarrass you and an effort to get some payments cloud cloud is a new thing in many ways relatively speaking anybody ever used a floppy disk five and a quarter there's some old people Among Us in any case cloud is new cloud is new and there is and we'll talk about this in just a minute there is just zero margin for

error so Cloud misconfiguration means really bad day for you we see this a lot we like to throw around the trite phraseology that identity is the new perimeter and truthfully I think that it that it is so we'll talk a little bit about that moving forward last and uh certainly not least business email compromise this is huge rampant ongoing growing extremely frustrating extremely frustrating for the businesses and the security people in the room if you're not if you don't have MFA on all of your externally facing authenticated portals shame on you I'm sorry it's your due diligence it is not a silver bullet I'm afraid but this is just such a common and rampant inroad in

today's wonderful modern world and this is just a quick snippet of the current Trend which we are seeing and that is I popped a mailbox in a company I hang out for a couple weeks I see who that company does business with then I take advantage of a trusted relationship with that external entity I send them an email message and because I'm extremely security conscious I do it in an encrypted mail message the user receives that from a trusted external entity they click the link it's a valid link to Microsoft they authenticate to Microsoft totally official totally normal and then there's an additional embedded link which escaped your email inbound filtering click the link prompted Delong it one more time not

legitimate in this example through evil Jinx capture capture MFA session cookie I have a primary refresh token now and I can stay logged into your mailbox for oh ever bummer the next step in that process again and I'm speaking mostly in our neck of the woods just from my personal experience actions on objective for that business email compromise are obviously often Financial so I always throw this slide in here just to say multi-factor all the things please MFA as a technical control is a must it is not a silver bullet step two have multi-factor for other process in your organization please we work with an entity recently who does significant transactions all over the world

financial services company through one email thread three four messages from a popped external mailbox transferred 1.6 million dollars to the wrong person there should be a process right I mean I'll go that's crazy it happens all the time so again multi-factor all the things if you're going to do a transaction over X number of dollars based on your risk tolerance involve two people involve three people 1.6 million dollars involve everybody all right that's what we're seeing and that's what we're encounting and having to roll back from on a continuum before we jump into the top ten I have a presupposition slide we all come to this conversation with presuppositions with preconceived notions based on our

experience right and these are some of mine one is that we as an entity as cyber Security Professionals we often focus on the wrong things I'm sorry if that weren't the case I would have far less work second we have a tendency to over complicate everything I won't ask you to raise your hands because I know you I know me part of why we're good at what we do so we're going to work through the rest of this conversation to do a couple things one I'm going to use a little hyperbole everything I'm going to talk about today is first-hand relevant experience in the last 18 months all absolutely true I'm going to pull from

extreme examples because I want to Rattle your cage just a little bit and get you to think slightly differently and if I can do that even just a tiny then it's a victory second I'm going to work to oversimplify again I'm a tactician it needs to work so I'm going to work again throughout this conversation to make things super stupid simple because that works for me I think it works for you complexity is the enemy of security thank you last but not least every business is a bit of a snowflake it just is you have unique requirements unique priorities unique resources at the same time I've noticed its snowflakes have a lot in common with one

another so I'm going to give you some generalities today and I'm going to ask that you take those generalities and then apply them to your particular snowflake you know you far better than I know you

the 10 things I'm going to walk through are not necessarily in prioritized list and of course I would love nothing more than at the end of this conversation for you to say I've already done all of that high five happy day major victory imagination again if you will imagine that you just started a new job in a large organization and I don't know do we have Cube Farms anymore it doesn't seem that long ago that we waited through Cube farms and major Enterprises I've been there I've done that so imagine if you will you're in a large unfamiliar new office environment you're the new person so you're working late naturally everyone else has gone home

you're a little confused and disoriented about which way is up left right down but you're brand new so no big deal in the middle of that the fire alarm starts to go off and naturally you begin to panic a little bit I'm not exactly sure how I got into this building how am I going to get out I go to the nearest major door and to my great joy and amazement there's a little sign there's an emergency evacuation site sweet I'm saved and when you get there this is what you find [Music] why we need a fire escape plan let's talk about actually the definition of a fire let's talk about some common ways that a

fire started let's do some run books let's do a lot of run books maybe 30 40 or 50 of them one of two things that's going to happen at this point you're going to die from smoke inhalation or you're going to be a sane human being and just run for a door this is what Enterprises do for their csirp and I find them to be largely useless out of all the incidents that I've worked in the last two years I can count nope nobody's ever ever leveraged they are our plan in the middle of active incident response I'm sorry it's just the reality of it that frustrates me it probably frustrates you in many instances the IRB

is 70 pages long so many people it's an emergency guideline bad stuff just happened stand by stop it there may be some value in that plan but what I would like to suggest is that you have a useful tactical abbreviated plan to use actually in the event of an emergency keep it simple keep it 10 pages or less if you ask me 10 pages is still pretty long frankly if you need to have all these other additive components make it modular indexes it's not that complicated just a handful of things that I think belong in that plan things that you're actually going to want to know in the event of an emergency if your IR plan literally contains a

section on what ransomware is you need to review that plan I work for an mssp part of the parallel is you pay us good money to develop a plan and it feels like if I give you a six page plan you're gonna be like what find that right balance find the right tension for your organization you must meet your awful quiet no nobody's with me sweet this slide hurts people's feelings sometimes the next thing when I roll into an engagement with a customer bad things just happen really bad things possibly the worst day of their professional lives one of the first questions I ask is do you have cyber security insurance and you want to know what the by Far and

Away most common answer to that question is actually it's I I think so you think so might be a good thing to know in advance and then the next question is if you have it do you think maybe we should call them no no it's just a little dent plunger and some Vaseline I can rub it out do not call the insurance company you paid them a half a gazillion dollars a quarter I would like to suggest that your cyber security insurance provider should be your business partner and if you do not relate to them like that you either need to enhance that relationship or find a new provider critical component of your IR plan

critical something you can do tomorrow okay not tomorrow Monday do you have cyber Insurance nail that one right out of the gate who do you contact when do you contact them develop a relationship with that person call them on Tuesday just for the heck of it develop that Rapport develop that understanding delineate these things this is one component of your IRP even if it's standalone third-party resources have these numbers handy have the names handy know when to call them these this is not sexy right you're super excited about this one I know you are I'm sorry have these things lined out how long does it take your organization to engage with a third-party contractor if you

need me tomorrow how long is that going to take I have literally seen it take three four weeks for large Enterprises incident response is not a game of seconds and minutes I'm sorry but it absolutely is a game of hours and days get this stuff lined out before an emergency super simple not a big deal there are lots of zero dollar IR retainers now sometimes you get what you pay for but that is better than nothing to have a pre-arranged engagement with somebody to help you if you need help make sure that they are approved by your cyber security insurance company in advance legal again large Enterprises often have in-house legal and they are reticent to

call on external resources they're probably really really good at their job this is not that sorry the legal requirements and involvement for a breach are very different I would strongly consider having those folks arranged ahead of time right all right I'm not going to say anything else until somebody else responds to me logging and auditing again not super exciting huge huge issue for me as a digital forensics and infant response professional so it worked a case very recently again a financial services company you got a mailbox popped threat actors hung out in that mailbox for a couple three weeks we get called in to ascertain how it happened and what went wrong and of course the business entity their credit

card information their vendor information there's HR information in this mailbox and what's the burning question on the Enterprise's mind can you tell me exactly what the threat actor accessed to which I responded do you have is your mail in M365 and everybody goes yes almost everybody sorry it's just a fact the next question is do you have E3 or E5 licensing to which almost everybody responds E3 E5 is expensive it is indeed it also lacks mail item access to auditing in that particular scenario 125 000 mail messages in that mailbox and we have to assume the threat actor access all of them because we don't have the audit data to prove otherwise that's just one tiny example and this is

the snowflake conundrum I need you to take this and I need to think about your critical data infrastructure things that are important to you whether it's email SAS infrastructure Erp CRM and make sure that the things that you need to be audited are audited please Microsoft really frustrating if you're not familiar if you're using M365 don't raise your hand but you are thank you and you're not familiar with the differences between E3 and E5 licensing you need to know and you need to make sure your management knows and the business leadership knows the things that are unavailable to you based on auditing and that applies to all of the above pay attention to that one and maybe you

can't afford E5 licensing most of us candidates are legitimately significantly more expensive than E3 so buy a few licenses and assign them to your accounts payable your accounts receivable your HR people your admin assistants and then maybe your sea level exists that's fairly manageable usually I see people buy them and then apply them only to sea level and or I.T people for some odd reason and that's usually not the attack Vector to be honest again looking for the money so one more time make sure logging and auto auditing are adequate before you call me and I say I can't help you I literally have customers say I'll buy E5 right now will it be retroactive

don't think so we touched on this at the very beginning and I keep saying zero margin for error you mess up on a local server configuration internal to your Enterprise it's not good might even have a bad day or two you mess up on cloud infrastructure and you expose S3 buckets or unauthenticated apis or or or or really really bad deal working with a healthcare Enterprise recently just making a move into Azure It's A Brave New World for healthcare pushing on-prem infrastructure to Azure they're smart enough to realize they don't have the internal expertise so they engage with Microsoft directly they send their engineers and their security Engineers to do this training Bravo that's know thyself and know your

resources so they go to this training and a very well-meaning security engineer is sitting in that training learning all about nsg's network security groups effectively your firewall for Azure infrastructure playing around doing some testing thinks he's in the test tenant for their environment opens the firewall completely leaves it open to the entire universe for fortunately not an extended period of time for all their Healthcare infrastructure zero margin for error if you don't have internal expertise get help please on this one this is a big deal and most of these are pretty simple in a lot of ways do not expose S3 buckets publicly to not store your keys in GitHub etc etc finally on this side do some active

monitoring because of that zero margin for error you need to be paying attention to this stuff on a Continuum you can't check it twice a year right it's Brave New World

the uh the next topic of discussion is of course the oh so exciting asset inventory say CIS top 18 right number one and number two are what asset inventory Hardware software why do we hate that so much and why do we stink at it I don't know uh it's not exciting it's Dynamic I think it requires a dynamic solution and it's just not our favorite thing to do especially in a distributed Enterprise becomes pretty hard and I'm not asking you to do all of it right now today again we're talking about the two week to 30 day interval what I am asking you to do is Define at least tier zero and tier one immediately

you need to have 27 tiers in your environment fine tier zero critical infrastructure let me tell you what it is you ready active directory and payroll maybe not for your environment but for most and you would be maybe you wouldn't you might be shocked to learn how long it takes an Enterprise to make these decisions we roll in we do detection analysis containment we stop the bleeding we stop the threat actor we then say to the Enterprise what do you want us to do now where should we prioritize our efforts what do we do next and they go uh honestly they never think of payroll until payday is looming basic tiers of criticality the most

important things to your environment don't go crazy don't go Hog Wild again in the interim do something practical and tactical and help me to help you at least by defining tier zero and tier one and then maybe we can argue about tier two three four later

if you don't have an active inventorying solution that should be a priority to you I'm sorry again it's boring it's lame but if I don't know what you have and you don't know what you have how are you going to protect it I know that's right you've heard it before but it is important you're doing awesome by the way you're staying mostly awake and alert which I find to be encouraging a few of you are actually laughing now and again I think maybe at my jokes but I'm not really sure uh that's okay I'll take it internal columns how important are internal Communications to your business on a regular basis outside of crisis exactly I would like to suggest

that you identify and provision a role a liaison role in the midst of active incident response what I see all the time is that bad things happen boots on the ground Hands-On keyboard identify bad things they get to work on rectifying remediating bad things and the CEO wants to jump in the middle of the War Room and then nobody gets anything done and this happens on a Continuum so I would like to strongly suggest super silly simple but it helps me I see this Excel me more than anything else in this conversation from a business process perspective Define a liaison role and at least two people who are capable of filling that role who can communicate

between the folks that are actually doing the work and then the executives The Business Leaders the decision makers go back and forth and go between hugely beneficial and again just super simple and then I would also suggest that you get agreement beforehand at the nature of those calls and interactions specifically between the technical tax Tactical Team and the business decision makers

additionally I just mentioned business email compromise is really near the top of the list of my concerns from a security perspective I think it should be near the top of the list for security current concerns for you so when you experience a business email compromise 30 days from now how are you going to communicate in and around the Enterprise when email has been compromised why does nobody think of that email's down how are we going to let everybody know I know we'll send them an email I mean seriously you've seen that right I mean I've literally seen that happen you got to go ah I need to question your logic have you heard of Gmail

it's this cool free email service before bad things happen Define out of band comps it could be free obviously you may have compliance concerns you may have business requirements so you may have to spend a little money you could spend up an alternate M365 tenant lots of email Solutions do it in advance have an outer band email solution an out-of-band chat solution and last and definitely not least once you implement it tell everyone seriously we do tabletops all the time and we'll get to this question do you have out of band they'll be like yeah the I.T person that a security person will say yeah and all of the executives go we do every time

so step them through it test it in advance simple cheap extremely meaningful Communications will help dramatically to improve our Effectiveness and response

a couple more and we're almost done I'm trying to go fast to keep to catch up just a little bit of time incidentally and if we have time for Q a at the end awesome if we don't I would love to chat with you sidebar as as you are willing and able

do we even need to talk about this one you have backups right of course you do you test your backups some of us do are you confident that they are offline and resilient to the latest transomware attacks again I'm sorry to say the gloves of course are coming off more and more with these threat actor groups and the first thing that they're going to Target once they achieve initial actions on objective are to Nuke your backups and if they're all online direct attached in your Nas 80 authenticated don't call me we worked with a large Publishing Company recently that had a tremendous backup solution lengthy retention ridiculous storage frequent Cycles all on network attached storage all 80

authenticated they've been in business for 20 25 years third actors came in nuked it all they had nothing had to read just a publishing company I think they had some assets nothing it's horrifying and I have nothing to offer them I just want to pay the ransom no uh no we don't rebuilt from scratch don't don't do that again this is just your due diligence you you have to have backups you have to work towards immutable backups that's a fun word that's right up there with zero trust just means backups that can't otherwise be altered and if that means old school do it on a USB drive unplug USB drive put it in safe not ideal

but a heck of a lot better than using 25 years of corporate data have your backups this oh heresy almost came out of my mouth this is more important than cyber security insurance right I mean this is your ultimate insurance policy insurance will just pay for me to help you Maybe they're not going to pay for you to rebuild your business I'm going to talk about Priority One and priority two in just a second when it terms of active incident response but this is I think honestly Priority One in terms of your I.T infrastructure you know it make it make it a priority and then of course this is also I don't understand this but the the mean time to recovery

and the retention periods are things like that people often don't think about we do backups every night how long do you keep them three days what seven days that's not enough that's not enough I rarely rarely engage in a situation where there's an active incident that the threat threat actor hasn't been there for at least three weeks that's pretty normal sometimes faster than that but almost always two to three weeks that's a minimum again talk about a hard conversation well-meaning I.T folks that have implemented this robust solution of immutable backups that only extends for seven days don't do that and then last but not least how long does it take you to recover we stored them all in super

cheap Glacier deep archives we can restore each of those servers in approximately four days how many servers do you have do the math not good think this stuff through this is not difficult to calculate in advance make it make it a priority oh

I know you again I know me we roll out day zero day one on active incident response usually is at 18 20 22 hour day especially if you have a small it team a small security team that is brutal and it is not sustainable and it is not good for you your business or your employees so one more time in advance of an active incident be aware of health and safety the health and safety of your staff not to mention the productivity I'm a little older than some of you when I hit about 22 hours my work product not so great not so great I was working an incident recently a small dedicated I.T team and they were just not going to

go home until things were rectified and Priority One was restoration of active directory and late in the night 20 again 22 23 hours into day one and we accidentally restored all of the wrong ad snapshots guess what we didn't get to go home or rotate staff at that point think this through in that particular instance we learned immediately and we appointed someone as a health and safety monitor who would walk around and go do you need something to eat have you been drinking water let me get you a hamburger and I kid you not that that transformed that department and our response we're like yeah I do need something to eat I haven't eaten it

can't remember when simple cheap extremely meaningful if you have a small staff again you need to come up with a staff rotation process you can't have them work three days straight you just can't so keep that in mind and pay attention some people have a tolerance some people can work 24 hours straight and still be productive Red Bull to the rescue I can't do that I just explode so know your staff know their capabilities keep an eye on it make it an active part of your incident response again almost done incident response team I love this quote no man appears to do that which he knows he does well I would strongly encourage you to

empower and Define your internal response team get them some training do not expect them to be incident responders on a whim they're system Engineers they're Network Engineers they may even be security analysts this is not that it's not and I walk in and they're petrified in many instances they're petrified of two things they don't know what they're doing and they're afraid that someone's going to make them look bad because they screwed up so cultivate a culture where you get some training for these folk that you set them up for Success they don't have to know how to do everything but they do need to know how to do a few things and I'm going to talk

about Priority One and priority two on the next slide but give this in preparation set them up for Success not again a huge deal have a tactical plan last but not least consider at least consider evidence preservation for me will you please and that just means you want me to help you to tell the story of how this happened and why it happened mostly because you don't want it to happen again right if you shut down nukes removed deleted don't have Vlogs in the first place makes my life really hard sometimes impossible and I have to say I don't know can't tell you so give some thought it's not that complicated to think about do you retain firewall logs

who retains firewall logs for six months yeah not good good man not that many people firewall logs are they're noisy right who wants to retain a list of 352 000 denies for every two weeks I mean nobody right that's okay truthfully if you keep seven days then when bad stuff happens have a part of your plan to Archive that seven days for me you don't have to keep a year forever just think about it think about it when something like that occurs you've already talked about audit logs and retention save them save them for me all right last and definitely not least when I roll in again question number two or three often is have you unplugged the internet

my greatest weapons for containment against an active threat actor are egress and Authentication if I can stop Communications egress meaning outbound internet we can breathe the second thing what can I do in your environment without authentication hopefully not much I need a credential to access sensitive critical things make these Priority One important two of your incident response tactical program have a granular plan for disconnecting the internet and if it means unplugging the internet for the entire Enterprise do it sorry do it anyway if you can manage granular site-based or functional or whatever more power to you but have a plan in advance and authentication be prepared to rotate all ad creds on a whim and then secondarily recognize there's a

whole lot of stuff that's probably important in your environments it's not ad integrated right Priority One and Two and I'm done you're going to make a practical tactical IR plan it's going to be short and sweet less than 10 pages if you can swing it you're going to pre-negotiate and arrange third-party access to Legal cyber security incident response help you're going to review your logging and auditing to make sure you have the data that you need the auditable fields that you need the retention that you need to meet your compliance requirements you're going to pay close attention to your Cloud resources get a little external help at least a little external training along with active monitoring you're

going to do asset inventory you're going to inventory all the things so that you know what you need to protect you're going to talk about internal Communications liaison relationship between the technical management leadership teams you're going to develop how to ban comms for little to no money and then socialize that those exist you're going to have backups you're going to test your backups you're going to work towards offline and or immutable backups you're going to pay attention to your incident response team because you care about them and their physical and mental emotional health and well-being you're going to get them just a little bit of training and set them up for success and then you're going to have a tactical

plan for disconnecting or disrupting threat Communications via egress management and a plan for rotating creds thank you [Applause]