Metrics Mess: Why the Lack of Clear and Common KPIs Undermines SecOps

thank you for having me um I'm going to tell you three quick things about me I don't like to do a whole like here's my professional bio there are only three things you need to know about me one I am actually from New York so I don't live here anymore and the reason I tell you that is that if I ex occasionally slip into fits of profanity which I will try not to do now that I know we're recording you'll have to forgive me the second thing you need to know about me is I am not actually a security person although after 24 and a half years of Faking it kinda sorta I am actually an

NBA bean counter weenie who stepped into cyber by accident in 1999 and haven't scraped it off my shoe yet and the third thing you need to know about me is yes I do in fact work at JetBlue Airways where my remit is currently uh threat Intel threat hunting sock incident response attack simulation and detection engineering basically everything that has to do with threat or detective controls rather than preventative controls and I only tell you that part for two reasons one it's Jermaine in my talk as you will see and to New York Homes New York's Hometown Airlines so let's hear it for JetBlue all right see I got to stay close to the mic all

right um first thing you need the next thing you need to know I always like to start with thank you all right we are going to do a little Icebreaker and I am going to encourage you all to participate because you will win valuable prizes and by valuable prizes I mean cold Hard Cash fold and pocket money all right show of hands who knows what that is awesome okay keep your hands up now in the context those of you know what this is in the context of that picture for one Greenback can you guess what just shout out this is initial access and he's sitting closer than the other guy would you pass this down to the gentleman in the green shirt

please all right cold hard folding money so see now you crack the code what's this one

you know what I'll give you credit for that because I go to the gym about once a year so I'll give you credit for that it wasn't the one I had in mind though elevation or escalation all right you know what I just got to start handing out money all right all right you said persistence you said escalation which is also true and who said lateral movement I mean yeah lateral movement all right by the way I had to rob the 7-Eleven to get this many one dollar bills would you kindly help me out with that thank you all right um click click what's my point any Douglas Adams fans okay I love miter attack and it's not just

because I'm a Katie Nichols Fanboy although I am also a Katie Nichols Fanboy I love miter attack because it gave us a Common Language I'm also a former linguist I speak a couple of languages I'm also annoyingly anal retentive about definitions I believe the old adage that a problem will Define is half solved and miter attack gave red teamers blue teamers purple teamers vendors government officials a lingua Franca where when they say T 1504-b sub c number two we all can look it up and agree that means persistence on a Windows box established by modifying a registry key Hallelujah 20 years as a vendor and the last three defending a Fortune 500 Enterprise let me tell you just being able to talk to

each other is a huge win so I am a big fan of miter attack it's a rosette sorry was there a question no okay you can like raise your hand or Shadow scream at me I got a thick skin I I've done 700 of these so just talk over me if you need to or tell me I'm wrong by the way um I call this talk how metrics mess and blah blah blah and how we can fix it we are gonna fix it not me we all of us the community practitioners students academics attackers Defenders because nobody else is going to okay I just cheated one of my later slides I love miter attack because it is

that Rosetta Stone it allows uh Ian I don't know if you're here but the guys from title cyber to say here are the most common ttps for initial access this month and a guy who's writing Sim rules can be like oh cool and a guy who's working at the company that makes your mail security Gateway all go ah yes I see poisoned email attachment with Excel with a VBS script that downloads a bet uh batch bash script that downloads batloader that ultimately brands in Royal ransomware got it and everybody can look at that attack chain with its little tags on it we all know what we're talking about now I mentioned that I also run among

other things thread Intel and hunting and detections I also run or oversee sock and cert and that kind of common language and mutual understanding of what everything means when we talk about the job that we do is also anybody here ever uh work in stock okay so the same is true of metrics when you work in a sock right no no okay um by the way I totally blew my first joke the Icebreaker is Russian I was gonna do a whole route anybody here no so um I was going to do a whole thing with an accent I would keep doing callbacks through the whole talk anyway no more funding now it's serious part yeah okay show a hands

um if you have worked in a sock Cold Hard Cash can you revenge or yeah raise your hand because otherwise I'm going to run out of dollar bills too fast raise your handle calling you ha if you have worked on a sock what is one of the things you should measure or track in a socks okay what does that mean and you said you just you two guys just made the whole point of my talk and you don't even know it dollar for you dollar for you all right sorry if I you know shorted somebody a buck come see me after Okay um I wasn't kidding you just made the point in my whole talk in the first five

seconds two people who have experience working in a sock said mttr and he said meantime to uh respond and he said meantime to resolution or vice versa anyway thank you for making my point it's a I'm from New York ciao right it's a mess it's a disaster so I decided to being an old NBA bean counter weenie spreadsheet guy I decided what else to make a spreadsheet so I reviewed 30 sources from credible people I would do air quotes with both hands but I gotta hold that anyway you get the point it was so bad the data was so bad that I had to reduce it to 12 more formal Publications like white papers or

blog posts from the chief something or other from 12 security vendors who have collectively raised 2.5 billion dollars from people who are supposed to know what they're doing okay so let's just leave aside the fact that I had to drop 18 out of the 30 sources because they were so bad they were unusable here's the top of the pile ready it contained then by the way my Google searches were things like sock kpis and key secops metrics okay it contained Pearls of Wisdom like this one hang on who said mean time to respond Define it for me I'll give you another dollar double your money uh the time it takes for you to go thank you

would you hand that down to I need my tip all right I know this is not fair to you but I know seeing you get a delivery trash all right so It's a Small industry I'm old I know a lot of people all right so here's my point your definition was actually clear and it included an end point congratulations you are now better than 11 of the 12 vendors who collectively raised 2.5 billion dollars in funding here's my first pearl of wisdom before I had the Good Fortune to meet you sir mean time to respond is the average time it takes to respond

I believe being an old spreadsheet Guy this is what Excel calls a circular reference problem okay more Pearls of Wisdom so I'm going to go in ascending order of people's money right because NBA bean counter guy is time to detect the time it takes to become aware of an incident Maybe fetish for definitions how about first we agree what excuse me both become aware means and what an incident is sorry I I'm gonna let these guys off the hook because they only raised eight hundred thousand dollars of other people's money come on Clicker is it I didn't make these up the time to become aware of iocs and other threats so if I become aware of

jetblue.com being registered or going live is that when I start the clock or when I stop the clock if it's when I stop the clock when do I start the clock when I start the clock when do I stop these guys raised like 30 million dollars in funny but let's go with somebody we all know if you've been around this industry at all you know mindcast theirs was time to detect is the time to identify an attack so is that when the alert goes off or when the person reads the converter when the person decides the alert is not false positive and it actually isn't attacked or I don't know do you it would be helpful

if they'd actually explain that in their white paper about sock metrics but they did not um that is better than the one from Cipher oh by the way mime cast 90 million dollars in other people's money Cipher just listed time to time to end time to and included no explanation at all awesome okay my personal favorite and you'll see why and obviously I'm not afraid to name and shame my personal favorite is the time it takes your security team and technology is to notice abnormal behavior first of all I think we all know especially you students on the college bar scene that noticing abnormal behavior is rather subjective and second of all you cannot have one metric for both when

the alert or when the control notices and when the human notices that the control noticed my favorite part of this story is the company is called security scorecard okay now mean time to detect I mean a lot of things there's ambiguity okay but mean time to respond or Resolute we would hope mttr is better and by the way those are the only two that people consistently quote or think of mttd and mttr and I think we've shown that you can't really get consensus on that so the average time to respond to a cyber threat cyber threat is the mean time to resolve okay for anybody who's been an incident responder my whole life just changed

responding is resolving I don't have to do any work this is awesome right I think you're getting the point I haven't actually seen which Marvel movie this is from but I'm told it's a really famous quote so I kept it all right by the way forgive me for putting my phone over here but I have no idea how fast I'm going so I'm going to keep an eye on the clock there's one on the screen okay I'm doing fine on time in fact I'm probably going a little fast so let's talk about the actual data not from the 30 sources but just from the 12 that were credible enough to have raised billions of dollars in capital

spreadsheet guy there's my fancy spreadsheet in case you want me to do the math for you that is 46 instances of 13 different metrics my favorite part of this one is that if there's nothing else we could agree on a sock who's no disrespect again I oversee one their job is to sit there stare at things and when they go ding notice them you would think that we could at least get agreement on the fact that mean time to detect whatever detect means should be on the list nine counts for time to detect 12 sources writing white papers and thought pieces on psot metrics in other words 25 percent of the people in this business do not think that time to

detect is an important metric for asoc I'd slam my head on the desk but it'll mess up the recording okay of the Publications that I included in the data set this is the number other than the fellow in the audience that actually provided definitions that were both explicit meaning it was clear what they meant and that that definition was then functionally usable as a guy who actually has to report metrics and run a sock what hat tip two logarithm I should note that he took their CTO authoring the piece to get it vaguely right and by the way they were not great they were only the best of a bad bunch okay now um I assume that you all have your

opinions on these next questions or you wouldn't be sitting here or the red team side was just too incomprehensible to go to so you're like well what am I going to do until the five o'clock either way here's the part that the NBA weenie B encounter cares about so what why do any of us care about this why are we talking about why am I talking about why are you listening to it okay let me break that down a little further this lack of common metrics this lack of a Common Language what Miter did for ttps and doesn't clearly exist for sock and incident response as business processes why is that a problem if we fix it what do we get for all the

trouble and if we agree that it's worth doing which I will try to convince you of how do we do it okay let's take a step back I'm going to ask someone anyone to go out on a limb this is not a right wrong answer there are no right answers this is an opinion question Cold Hard Cash if you are a practitioner if you work in a sock or assert or you fiddle a Sim or an EDR or you basically do Intel or threat hunting or engineering or anything that has to do with securing an Enterprise or an organization here's my question why do we have a job any of us any of us in

this field or students who want to be in this field why do we have a job trying to double his money to keep the business thank you very much double dream run anybody else I got a couple of bucks left to get paid no no that's why we do it why does the job they pay us to do exist because people are dumb and click on things yeah I mean I can't disagree with that statement clearly I think I got a dollar left keep the lights on no wrong answers he should get two dollars but I don't have that many all right um like I said I have been doing this for 25 years uh there's no right answer

there's no wrong answer so I'm going to give you my answer and I know this seems like a tangent but it isn't it's at the core of my talk today what I'm talking about and why I'm talking about it we may take the job to earn a living but the job exists I would argue for only one of four reasons and you are welcome to raise your hand and tell me something we do in as cyber professionals or I.T Security Professionals that doesn't fit in one of these buckets I haven't found one yet which is why I held at least one dollar in reserve in case you can prove me wrong all right here we go I propose

that everything we do in cyber boils down to one of these things ready reduce the likelihood of a loss event loss can be money loss can be data but basically this is your preventative controls your ssdlc your network security architecture doing all the stuff right so you don't have to deal with the problem later keep the bad guys out which is as I said exactly the half of cyber means that I do not do I work in domain number two which is assume that that failed and the boogeyman is in the house right Intel hunting draft detections institutionalism sock Playbook escalation assert Circ Playbook and then when you think your controls are in order you test it because an

untested control an unvalidated control is not a control it's a hope right if you haven't tested it assume it doesn't work detect faster to reduce the likelihood of damage from that event ensure compliance I know this is the one that most people in cyber roll their eyes the nist 871 I promise you there are fines for regulatory violations that will dwarf any ransomware paint okay The Government Can impose bigger fines than Russian criminal gangs gang I guarantee it don't believe me ask Equifax okay the obvious joke um finally defend the brand and for those of you who hardcore security Geeks Who want to open a c prompt and think that defending the brand doesn't matter

defending the brand I used to work with a very large investment bank and they said when we started doing thread Intel this was third of our three priorities right keep operations going minimize Financial losses oh yeah and protect our brain over the 15 years of evolution of both their thinking and actor sophistication that literally reversed they now consider defending their good name more valuable than the loss for many particular event okay so today's talk is about this one okay defend faster detect faster riddle me that's Batman why does that matter by the way does anybody have something in cyber that does not meet one of those four domains I challenge you pen test vulnerability management socks or seem take a break

okay the one we're talking about today is detect faster why because dwell time equals damage period right the dwell times on some of these well-documented cases are unbelievable guy gets in on a Tuesday and it's not like they run through the network and crypto Locker everything on Wednesday the average dwell time before the really adverse event happens is like weeks or months that's a lot of time to read your Intel draft your detection run a threat hunt put things in with your sock there's time there shorter time less damage full stop I just clicked it okay so what do we get if we fix it less damage pretty simple right we do this better we do it faster we have less

damage Okay so lots of pictures lots of stories lots of hand waving all of a sudden I'm going to get into the uh like where's the guy with the stopwatch hard part right because up here right on right now I'm just philosophizing wig my hands telling stories how do we fix okay um it is we it's all of us uh I mean literally I have a colleague not where I work who is struggling with the following they're relatively new to their job they have been asked to report metrics for the performance of their sock and incident response functions and they cannot make the numbers line up except by doing monthly gymnastics with a spreadsheet because they have a seam

a sore and a ticketing system that do not agree on the definitions of time to detect time to respond time to contain time to mitigate time to remediate time to close right if you're using Sentinel one Chronicle cortexor and God help me serve us now and they don't line up you're literal and by the way of course half of them are local and half of them are GMT and by the time you put out your monthly metrics report it's like painting the Golden Gate Bridge you get to the end you got to go back and start because the other end's rusted again that's all this person does is put out these stupid reports because there is no common language between the

systems that are producing so how do we fix I have a proposal I call it the B6 timeline model bonus question oh by the way I should say I promised him I would do this this is largely the brainchild of me and a brilliant young engineer named Andrew Malone if you like it I get all the credit if you don't andrew.malone jetblue.com feel free to send your critiques to him all right bonus prize I'm actually gonna go big here five dollars this is a trivia question outside of security anybody waiting to have any idea why I call it the B6 mode it is all right out of code hand that man of Finster thank you very much

so if you ever get a ticket on JetBlue flight 1545 your ticket actually says b61545 thank you I don't want to put it out I'm not looking for credit for me or JetBlue I had to call it something here we go can we agree that and I know there are outliers and we could argue about DDOS which I would argue is actually not a security issue it is a resource exhaustion issue which is not actually a threat in the same sense but that's a get me over a beer can we agree that there is an initial generally speaking if we're looking at a miter type event there is an initial compromising act okay somebody gets in initial access

right then click there is some amount of stuff that happens and then God willing unless your time to detect is infinite a control goes ding now the reason I put brackets around letter B is because depending on whether what they're doing is something very well known and your EDR recognizes it and goes ding or it's not where your EDR sucks which by the way all of them do I believe traditional EDR is dead and that's another talk at another conference but that could be measured in milliseconds or months right so for all practical purposes initial compromise and detection can be one millisecond apart or an unknown amount of time okay but then there's an event that's detected

the human picks up the alert from a queue or a control or a thing or a pager or whatever yes I'm showing my age and then the analyst goes thing not a thing I don't know if it's a thing if like generally speaking of a tier one sock cannot dismiss it as false and there's false positive and false alarm I got a whole thing on sock terminology too but let's just say if they can't dismiss it as not a thing to worry about then they have to investigate it if they are two three fours or they escalate it to a cert team that can't stop me if I'm like missing anything or make a generic timeline right

um and then they take some mitigation actions to stop the bleeding or whatever uh you know will at least contain the damage and then there's an ultimately some kind of return to actual normal operations now notice I do not say pre-event because there is often not a restoration to pre-event circumstances conditions there's only we're back to the new normal based on what happened to us and then finally when all that is done the business is operating a month later mandiant drops the thud Factor 50-page report on the desk and goes here's the story of how you got buggered great we got on the timeline as a generic series of events kind of Circa miter attack okay so here's my question

based on my research of all these thought leading companies is this time to detect alert goes ding to the time the analyst looks at it okay I could ask you what is but just stick with me you say no anybody say yes okay is it the time the alert goes ding to the time the analyst decides if it is or is not a thing I agree but I have seen Publications that argue that is time to detect how about from the actual Act of compromise to when the alert goes down is that time to detect alert's gone off no one's picked it up out of the queue and they don't for another 17 hours is

that time to dissect or is it when they pick it up out of the queue or is it when from the initial compromise to the time they decide it's a thing I think you get my point so which one's time to respond this one or this one depends where you set the end of the slider on time to detect no doesn't it maybe this is time to triage so if that's time to respond maybe that's time to triage depending on whether you start with the alert or the detection uh the human peeling off the alert or the Judgment of the alert or the initial you get my point right when does the watch start and stop I don't

know and neither does the industry and that's my point okay so here's our proposal just as a conversation starter which is how I marketed this talk we propose that the event starts at the initial Act of compromise okay agree disagree you can beat me up at the happy hour later at least I'm putting a stake in the ground and the detection ends not when the control goes ding because if the control goes Ding and the guy misses that alert your time to detect is infinite and trust me that happens in the real world so I'm going to argue that time for the sake of argument the B6 proposal of the day and we iterate on this thing all the time is

the time to detect is from the compromising event to the time the human looks at the alert if that's true then we would argue that time to respond is the window between when the alert goes ding and the time the analyst looks at the alert not judges the alert not determines whether it's a thing just ding to peeling the sticky off the top of the queue and going I wonder if this is a thing now this is really important and I by the way I have five more but this is really important because I want you to get the idea that the first one is actually the aggregate of two things the time from the compromise to the ding

and the one in purple the ding to the human in other words we are not judging just the human or just the control in the red one it's an amalgam okay I would argue that time to mitigate is from when the alert goes ding to when you have stopped the bleeding that's not return to normal operations you've just you've isolated the machine you've chopped the thing with the fiber with an ax whatever you need to do to at least stop things from getting worse okay time to triage is the time from ding to I'm sorry time to mitigate we just did that one there it is time to can't see a damn thing anymore time to

triage is review of the alert to deciding whether it's not a thing or they need to escalate it to the cert team time to resolve is from peeling the sticky to return to normal operations got two left again we can argue all of these I just want you to get the idea I am trying to get the rock rolling down the hill the way that we needed miter we need to talk about this until we get to a consensus here's the last two you probably hadn't thought of because they're not tying to anything and they don't appear on most people's sock or cert dashboards one is the exposure window that is the time from the initial compromising act to the time

you kicked a bad guy's butt to the curb not return to normal that's total incident duration so that's a proposal it's like this equation is it right hell if I know I'm just telling you what I think there is no right there is no consensus and there is no body there is no standards body there is no ISAC there is no organization of erudite professionals or robed Elders on a Mountaintop that is making these decisions so I am proposing that together we do that so is our version the right version hell if I know I'll probably change it by the time I get back to the office but here's the thing I can tell you that while I

don't know whether it's the right answer we better figure it out and I'll tell you why and that's why I said there are three things you need to know about me one is I'm profane one is I a security professional at an airline and one is that by background I'm a bean camper Okay so question for the group I got one more dollar which of these does management care about most incident duration of exposure window who said exposure wind your second he's first would you pass that along for me thank you

sorry pass that down to the man in green I had one more doubled his money okay in that order that's what management cares about when all is said and done okay put it in my terms if we ever had a really bad day which thank God in my two and a half years or so with the airline we have not when all the mess has been cleaned up in aisle seven right Robin Hayes our CEO is going to have two questions right he doesn't care how long between when the ticket went Ding and when the guy looked at the ticket he's gonna go how long ago they get in how long was my ass hanging out in the

Wind when do we get back to normal because I've got to go tell the board and the street the answers to those questions and you better have him Olson or you will find yourself on the street right all the stuff between a and F I guarantee them to you that a publicly traded CEO doesn't know doesn't care never heard those words all he wants to know is why did it take you so long and how bad was it and how fast did you fix it okay so as a guy who is not a CEO of a Fortune 500 company but as a guy who runs in Talent hunting and sock and certain all that crap excuse me sorry oh it's so

close behaving myself um all that stuff I told you about why do I care why do I care not why robin cares why do I care I am a disciple of the Church of Drucker like most NBA weenies and people often including me misquote it because it's a great phrase you can't manage what you can't measure the actual quote is if you can't measure it you can't improve it so what does that have to do with anything let me back this up one more time if we agree axiomatically definitionally that green is always too long let's say you have 10 incidents in a year unless I'm just making up to make the math easy and the average time is

100 hours and everybody goes oh that's bad 100 hours that seems like a lot and you need to make it better maybe you have mbos to make it better maybe your job depends on making it better if you don't know which component of the green line was 98 of the hundred hours you can't fix it if the guy picked up the ticket and one minute later determined it was a thing three minutes later isolated the machine and in five minutes it was all cleaned up but the total time was a hundred hours why the hell it takes 99 hours and 45 minutes to find the problem or vice versa if the control went down in one millisecond right

and somebody saw the alert and then they went I think this is really bad and they escalated to the cert team and turns out the cert team is in the Philippines and they're on holiday that day because you didn't know it was National go to the beach day Ian you lost 36 hours and you don't know which bracket your time stinks in then you can't fix it you're not going to fix we need to fix total incident duration of 100 hours we need to get that down to 80 hours Bob total incident duration is a summation what you need to know with granular visibility on agreed definitions that require people and tools to have a

lingua Franca where they're all speaking the same language is which part of the process is broken so right now to use a Biblical analogy we are all babbling at each other in the historical sense because time wait what was it time to detect is the time it takes to detect thank you very expensive vendor I appreciate your thought piece um I'm glad that was included in my three hundred thousand dollar annual fee right it's absurd it's laughable except that it's not because this is my this is my job like I get up every day and I have to deal with this and so um is this proposal as I said the right proposal well I know

but for those of you who very kindly contributed I know it was purely out of pecuniary interest but for those of you who spoke up and contributed said yes I've worked in assat yes it's meantime it's mttr which is resolve or respond you can fight it out in the bar later you're making my point for me we need a common set of words so that we can measure so that we can get better because well time equals damage and of the four things that a cyber professional does the people who work in sock and cert are job primarily can be boiled down to the second of my four quadrants which is reduce reduce time to detect the adverse

event that you could not prevent because time equals damage that's it so I invite you all to join me blog tweet scream at each other and for those of you who are fortunate enough in some ways to be on the Enterprise side start slapping your vendors around I was one for 20 years I've been in plenty of those meetings start demanding some clarity because when a company raised a hundred million dollars says time to respond is the time to resolve an incident and I'm going so all I have to do is pick up the ticket and it's fixed that's awesome except it's wrong and that was a thought piece by a company that people go oh look at that

big Booth over at RSA next week you got to be kidding me if I can find the guy who wrote that blog trust me I'm going to tackle him to the ground and slap him because this is my reality this is the job I get paid to do and it's ridiculous that I am getting paid what I cost to deal with this nonsense so please help that's it q a

search

sorry can you say it again

you said like five people

take place the upside talking about reality where we get the damage thank you let me restate it and you tell me if I if I'm paraphrasing your question right for people to hear is time actually the right metric is the relationship between dwell time and damage linear certainly not is it case specific absolutely are there cases where dwell time is not really correlated to damage at all you are absolutely right and I'm guessing some background in mathematics um as a general statement for measuring performance of a sock and an incident response team I would argue that and remember we are not talking about preventative controls right this is detection and response um I invite you I am not disagreeing with

you I invite you please if there are kpis for a uh SEC Ops or incident response team that can be consistently measured across teams across Enterprises and across Industries over time that are not time-based please add them to the model and show me where I'm wrong in certain cases yes but I I haven't made the leap you just made because I'm just not that smart so just to follow up with it like my point is the way you define these periods are based on what you actually want to get but it's easy the drunk looking for his keys under the Street Lamp as you said right so that's why I say I like my trip because because

we started talking about like what are the the DVDs me too

thank you I don't disagree I challenge you to answer the following question to me or to yourself or over a beer at that thing tonight which is how does miter help me measure the performance and look the average Enterprise um today unless they're really large or have specialty needs like the defense industry more and more folks find that they they Outsource their tier one song right so I'm making up the numbers to make the math easy I spend a million bucks a year or a hundred thousand dollars a month on a sock do I keep them are they doing any good are they bad if relia Quest comes in and says we're better than arctic wolf

switch to us and I go oh better better sounds good same dollars better better for money good what's better I don't know I would argue that time certainly has a place in the evaluation and please add non-time metrics to the thought I do agree but if we create the best server everybody studies to the test right

words not as much time to respond or customer support control advocacy because I've had several members say and I'm tired of every vendor is saying through tool stops

why wasn't control affected or how do you plan a use cases thought of any kinds of metrics that's engagement absolutely the question was um interest at the board level seems to be shifting from Time Performance to control efficacy um there's a marker but you probably can't see it if I draw it on the board let me just say this very quickly one I one million percent agree with you I was speaking parochially about detective controls stockings or performance is that actually the set of metrics that anyone at the Mount Olympus level cares about absolutely not control but you heard me say at the beginning right an unvalidated control is not a controllable right so super quick let me

tell you how we run things on a daily optimbo every morning my Intel team meets we have what we call the morning Intel stand up everybody brings their coffee we go through the overnights everybody's got their own their feeds and their vendors we go around the room boom threat ttps urgent uh exploits you know whatever Geno political we go around the room we go is there anything that need to drop everything worry about right flipping that if not and we've got a backlog with interesting hypotheses we want to write the text board we want to hunt you draft something we uh do a friend hunt on it or a retro query or whatever you call it on your

your tools when it goes ding we've got some sample data we send it over to our sock to give us feedback noise level tune it this way right they figure out a Playbook we institutionalize the strength of the Holy waterfall production they escalate an example to the 13 the 13 goes what do we do when we catch this thing they're pitching over the wall they write their procedures in their playbooks and they're xor run books and all that stuff and they go great for that kill chain campaign whatever we're good whether the control is detective or preventative and then I go great and then I go hey Andrew the one who created the model if you don't like it Andrew grab our

attack simulation or a python script you wrote or whatever we use um attack IQ but Red Canary or whatever there's open source I'm like great now go attack us with that because then what you get is you did the control actually go in how fast did it go dang how soon did the analysts at our team are sent home one or about strike notice the thing they go dang how long before they decided it was a thing it wasn't a thing they asked all the stuff on that timeline helps the staff manager manage the stock and the cert manager manages her team what the board wants to know is did the attack succeed in the first

place and the answer should be no see none I think you've hit on a gorgeous point if I may I thank you in a perfect world none of the metrics I spent all this hour talking about would not you know why because all of the attempts would fail and so and by the way on another day another talk or grab me in the bar there are ways to actually get very close to achievable I will give you one example that is incredibly hard to do at a large Enterprise but is in fact possible which is application safe listening give everyone an iPad and if it's not an app that's authorized that's the end of the discussion if it's not required it's

not allowed it can't run period we have organizations in a commercial environment in a threat landscape where we hand people a general purpose computer connected to the entire internet and say how about it that is insane but that's another talk so to your point yes control Effectiveness and the action the effectiveness of preventative controls that make everything I just talked about irrelevant is actually he says undermining his entire 24-year career what I do shouldn't actually be the important part and probably isn't except that the guy to my left who runs all the preventative stuff I'm speaking generically as an industry not my friend who runs all the preventative stuff have failed respect spectacularly that's why I'm here that's why I exist

that's why the 16s I over C have a job so yes at the board level none of this matters I actually had a vendor CTO asked me the other day so uh thanks for being our advisor uh tell me here how many times did the board found out what do we do three people come talk to the board about what you did I don't talk the board about what I do my boss doesn't talk to the board about what I did you know what the board wants to hear from the security guy were there any major incidents do I have to sign off personally on any of the compliance ranks that we had something terrible

happen no get out I'll see you in a year that's what the board wants to know cyber security okay A1 dial tone reliability and go away that's what the board was sorry long answer to a short question anybody else I think we have like one minute sir how does your B6 model play out with the last bass dribbled outs oh my God it just got works oh my God it just got worse because it feels like there's like several attacks they're all mums so and this gets into another confident another talk on third party risk um so when something goes horribly wrong at a provider where I don't control the data I don't control the network I don't

control the systems I'm not aware of the incident and I can't respond to the incident I guess the answer is this model doesn't apply at all no I'm saying you were at last match oh the breach of LastPass in the actual tradition of 20 30 million volts through a BYOD Flex server on a personal device at home for the SRE you know how does that fit more than one event that went on they exhibited they turned the intentionality of the attacker isn't always focused is one event one breach one fish got it so long series of events slow and slow all kinds of steps in the kill chain etc etc et cetera I would argue simplistically

and I haven't actually mapped the LastPass case to this in particular it still started with an initial compromising event now if it's an Insider that means abusive privileges or access or an account means unauthorized activity by an authorized user in the case of an outsider it's unauthorized access and use um here's another thought to the whole like sorry quick aside to the whole solving this systemically here's another thing almost nothing can be done maliciously on on a network without an account so we need to pay real close attention to how accounts are behaving but I would argue that my clock for almost everything still starts when The Insider misbehaved or The Outsider gained initial access to an account and

began to as long as they behave exactly normally there's no euba that's going to detect it and no compromise has happened but the minute they start to behave strangely with those privileges permissions or account that is when the clock starts and then it's a question of if each event over a series of events has its own there was exfiltration there was lateral movement there was upload there was low prevalence destination IP or domain or drop site or whatever every one of those should have a chain on the model thank you for taking my money foreign [Applause]