It's M̶a̶p̶s̶ Gaps All the Way Down

wants to drop with this talk so maybe needs no introduction round of applause for dave all right thanks everybody uh more importantly round of applause for our sponsors who made all this possible all right and with that as this tradition i've got way too many slides so uh we're going to move pretty quick here i think so the title of the talk it's maps i mean gaps all the way down it's going to be about why detection and incident response are hard and maybe a look at some things we can do to make them easier and maybe we can get some ideas from the audience during this as well can you guys see this opaque so what you're looking at is a

process timeline this is part of a detection uh that where i work at red canary we would ship this off to a customer this is the start of it anyway before we kind of go in and make it make sense for people the top line there for folks in the back that may not be able to see it it's a windows explorer process spawned followed by a con host child process of windows explorer con host if you don't know is the windows console host process and it typically fires when you run a command shell application or powershell application some anything that runs at the windows console or the command prompt that kind of thing uh you'll see

conhost spin up as a child process in this case we've got conhost spinning up this child process of explorer and then conhost is running command shell if you were somebody that was getting data like this or looking over the timeline is there anything here that you would find concerning

anyone anyone multiple exes all right well i'll break it down a little bit and so again the console host application runs when you run a command shell so these are in chronological order from top to bottom here so we've got explorer is explorer a console based application or is it a gui based application it's a gui based application so to see explorer spawning the windows console hosts really weird the windows console host almost always has command line arguments and in this case there are no command line arguments in this screen and conhost never has child processes and here it's spawning command shell so this is seems a little suspicious if you were the detection engineer or

the security instant response person you were looking at this what additional data would you want to try and what we call adjudicate uh whether or not this was malicious or just weird level access what level of access the user has what else what other data you want if you're investigating this what else do you want to see what kind of commands they try to run from what commands will run in the command prompt i can tell you right now there are no child processes of this what was that startup like asaps that kind of stuff the startup locations what runs at startup on this machine anything else [Music] uploads and downloads what about security event log

anybody full disk image no memory capture i see one head shaking over here yes for memory capture so think about when you're doing uh security instant response work and you get a set of data like this generally people want additional data points to try and adjudicate whether or not something like this is malicious so i just wanted to first off kind of set the stage here and get you thinking about detection engineering and the kinds of problems that you're going to be faced with as a de and then we'll jump into this this is kind of the overview of what we're going to be talking about so we're going to do introductions going to find out a little

bit about who you all are tell you who i am very briefly and then we'll go into why uh detection and security response are hard problems we've already started with one example and there will be plenty more so first off let's do some introductions raise your hand if you do detection engineering work security incident response work digital forensics blue team in general all right so there's a number of you out there uh what about pentest red team any of those folks in the room so for those of you that are doing security and cert response work digital forensics anything like that do you find your your job to be difficult it's a lively audience yeah so

if you think your job is hard you're not alone so this is dan gear he's an industry luminary he's got a long back story keynoted at black hat a few years back and i've included this slide in other presentations before because it makes me feel good about thinking my job is hard so dan gear said in this black hat talk security is quite possibly the most intellectually challenging profession on the planet for two reasons complexity and rate of change are your enemy he doesn't even mention the fact that we're fighting against adversaries that are trying to make our jobs difficult and i think that's a big part of it so if you think your job is hard it's not

just you a little bit about me so i do detection engineering at red canary try and make it easier for us to catch bad things more efficiently and that's really enough we've got plenty of stuff to cover here so let's dive into uh the meat of it so there's going to be some things that are out of scope for this talk right if you've got 50 000 endpoints in your environment or 10 000 endpoints in your environment you have a hard problem of just centralizing all of the interesting security telemetry and doing analysis on it i'm not going to cover that in this talk it's beyond the scope of this it's largely a solved problem in lots of

environments so that's out of scope we're not going to talk about easy detections we're going to talk about hard detections uh and so let's get right into it with a quick tour of kind of the problem space and we'll start out with uh false positives and false negatives or why am i so negative about false positives and i'm not trying to pick on any edr platforms here specifically so just keep that in mind they're going to be several different platforms shown here and i had to rip out a bunch of slides for other platforms because they just don't have time what you're looking at here is kind of a chopped up screenshot from one of the

edr platforms that's out there in the world and the top line if you can't see it there we've got a command shell running a batch file and it has a child process of run dll32 there's a redacted lsas pid on the command line here that i've blanked out just to protect the nsn and if you can't see all the details here that's okay we're going to go through it so lsas for people that don't know it's the local security authority subsystem service and it handles uh enforcing the local security policy and it handles authentication and because of that lsat's process memory really contains the keys to the kingdom if you've ever run mimikatz it opens a handle to lsas and scrapes

memory and finds all the credentials and uh so elsa's is just a juicy target on the endpoint but you have to have the right rights you have to have the right permission level to be able to read lsas process memory no problem we've got an account here nt authority system it has the necessary rights and so what we have is run dll32 calling this function that's the mini dump function in this dll com services dll that's a dll that comes with windows it ships it's on the box it's signed and it's saying dump the memory from this pit which is redacted and write the data out to this file the edr platform detected this wonderful

and it's a high detection it says down there at the bottom high confidence high criticality i don't know but this process actually failed this didn't actually this won't work i don't recommend that you go try running this command in your environment because you'll probably trigger your edr products um but as an analyst if you got this alert someone on my team got this alert and they spent a bunch of time trying to figure out where is this dump file uh and it wasn't on disk because this command as it's written on screen won't work even if you have the pin in there for lsats there's something missing about that command so you've got a false positive

there which is going to cause an analyst to spend a bunch of time looking at it when in fact it's a waste of time this process however create dump from snapshot.exe which is a child of command shell wrote this file lsat.dump in the temp directory zero alerts from the edr platform in this case so you've got a case of a false positive that's going to be a time suck for the analyst and a false negative that the analyst isn't even going to know about so that's one set of problems false positives false negatives you've all seen them another platform out there and there are a couple of different edr platforms that work this way instead of centralizing all of the

security telemetry that they're collecting into some giant data store they keep it all in the endpoints and you have to query the endpoints for that data so this one this little section i'm going to call catch me online if you can so this screenshot is from uh the red canary portal so when our analysts are looking at alerts they see a screen like this and in this particular case we've got svc host which is a managed or sorry unmanaged code so this is old legacy windows application written in c c plus plus it does all its own hardware integration it's a memory management and all that so it's what typically is referred to as unmanaged code

this thing is uh note the command line arguments here not super important but just to keep it straight that we're looking at the same svc host process a couple of screens this thing svc host is loading system management automation which is powershell's dll which is uh managed code you know written in.net there's a it runs in a virtual machine that handles all the hardware interaction so it's really weird to see unmanaged process loading managed code like this and typically when you see something like this it's because you've got an attacker who has used metasploit or powershell empire one of the other tools out there and they've injected into some process like svc host and when they do that

injection typically the client is going to bring along the powershell dll usually if you see something like this you want to go investigate it so in this case we jumped over to the edr platform and we started running some queries and if you can't quite make that out in the back it says uh search files and processes for file svchost.exe on hostname redacted it's able to catch this endpoint while it's online but it comes back and says i got more than 500 results if you want to grab more than that go use the api i'm sure this edr platform has a wonderful api i don't have time to master the api for it unfortunately so i keep using this natural language

parser that it has to try and query for this data in other ways figure well if there's uh 500 hits percent svc host i'll just grab the specific process id that i'm looking for and pull the data that way well i try that and it comes back and says oh endpoint's offline or no sensors match your query which is a little ambiguous like uh does that mean i'm not writing my query correctly uh it i think means the endpoint's offline so i try again undaunted a couple of different searches just because you know it's somebody's laptop maybe they close a laptop they walk to a meeting and now it's back online and they're in the meeting they open thing back up so i

do a search here for show me file data for system management automation ni.dll that's the dll that we saw was imported into svc host and in this case the machines online have results i can go view go view the investigation no hits well the platform just told me this dll was loaded by some process i know it's on disk on at least this workstation why is it telling me there are no hits so undaunted i go back to my pid search because i know the machine's online do a search for the process id uh limit it to the scope for the date and time that i'm interested in and i know some of this is probably a

little difficult to read in the back but this is the screen that you get back in the web ui for this particular platform uh so it's got a time stamp up here in the upper left that i've redacted note it says pmutc which a quick side gripe about that in a second uh but it's telling me svc host this thing's running his system it's got the command line arguments and those are the command line arguments i told you to note a minute ago and then it tells me you know the process terminated this thing ran for a few seconds it doesn't give me any other there's no other data here like what did this do

with the file system what dlls did it load did it make any network connections i have nothing to go on here it's probably in the api uh if i could query it that way and i get out lots of juicy information in fact i know that's the case but just trying to use this as a sort of novice user without going into the api not a whole lot of useful data here so quick side grip about this pm utc business and this seems like i'm complaining about nothing and it's just it's a trivial complaint i'll i'll give it i'll give you that uh but if you know anything about utc utc is 24 hour time 24 hour time doesn't have the

concept of am pm why am i complaining about this when you're doing security insert response work it's a mental sprint and you're trying to run as fast as you can if you have to convert time stamps for one system that's doing 24-hour time in utc to this bizarre form of utc that requires am and pm you're just wasting cycles so quick side right there but this screen as i said it provides little to no new useful information nothing about network connections nothing about file system interaction nothing about registry changes so there's really not a whole lot to go on here as i kept scrolling through the results though the very next process after this spc host process exited

is a powershell process remember the search was by pid so this is the same process id as the svc host process that exited so we've got process id reuse here so note the the md5 hash and the domain that it's running under this is different than svc host and what it was running under so yeah take note of the md5 hash there and then this is the process termination screen so this powershell process ran for a few seconds and exited and the process termination screen that the platform is giving me uh this is the md5 hash for spc host this is svc hosts command line arguments for the previous process so it seems like the platform is

confused because the pi because the pid got reused within two seconds of the svc host process exiting so these are the kinds of things your analysts are contending with when you're trying to do security answer response it's not making the job any easier uh another platform here this is the last one we'll look at and i'm calling this you know there's there's plenty of data uh and just not enough information so that we're going to go back to the the thing we started with we've got windows explorer gui application parenting a console host application that's normally parented by powershell or command shell exclusively and normally i mean you'll never see this thing have child processes i don't think i've ever seen

it legitimately but in this case it's spinning up a command shell so we want to dive into this and figure out you know what's going on here and so we don't we jump into the edr platform it gives us a nice tree view of what's uh happening here there's my command line at the top of the screen windows con host no command line arguments which is highly unusual for conhost the platform helpfully tells me it's signed it also gives me some information about the number of mod loads child processes and cross cross-process events that's kind of down here in the bottom left corner and apologies if it's a little difficult to read it says there are 33 module loads for

this con host process two child processes and one cross proc it doesn't give me any indication of what's normal here which is uh something i wish it would do so i don't know if 33 mod loads is a lot or 33 mod loads is too few or if any of those mod loads are unusual it doesn't tell me whether or not conhost normally has child process or cross-processing events this is all kind of left up to the analyst to figure out and so as the analyst we can dive into this information further we can go look at all of the loaded modules all the loaded modules here show that they're signed and this is not all 33 of

them but this is all the ones that we fit on a screen so they're all signed so everything's good nothing to worry about possibly uh what about low bin attacks so we already talked about you know spc host importing the powershell dll that could be evidence of a low min attack uh so you know we just don't know there's just not enough data here what about stolen code signing certificates so that's happened a few times where companies have been careless with code signing certificates or they've gotten hacked by three letter government agencies and have their code signing certificates stolen uh so that we can sign malware with it so you know uh as an analyst you might be tempted to

say well everything's signed so we're golden and uh fortunately we've got padme there saying you double check the certificate revocation list and looked for a little bits right and you've got to go pretty far down the rabbit hole on this kind of investigation if you want to be thorough but remember we're under time pressure this is a mental sprint so as analysts you don't have time to go down every single rabbit hole here uh unfortunately because time is of the essence so how could you do this in a timely manner well one way would be to get base rates you know this this platform and there are there are platforms that do a form of this they'll

give you base rate information they'll tell you this executable is very common in this environment or it's uncommon in this environment and even in this platform there are queries you can run and you can say well how often is this executable scene or how many machines is it on they'll give you that kind of information but they don't give it to you at this level to say you know it's highly unusual to see this given dll loaded in this process would be nice if they would do that they have the data i think they could do it for now they don't so let's just make an uncomfortable assumption here say everything in this list is fine

and we'll move along let's go look at child process and cross-process events for this con host process so there are three child host cross process events uh two child proc events and one cross process event uh for this for this condos process so 14 milliseconds after conhos started command shell starts there are no arguments to command shell what does that tell you if there are no command line arguments to command shell does that tell you anything [Music] it's an interactive shell exactly uh i have i have prizes up here for people that are participating so if you're interested come up and see me after and i'll give you one of those yeah so if you see command shell run

there are no command line arguments odds are it's an interactive shell same for powershell so 14 milliseconds after conhos starts command shell starts it terminates after three seconds it has no command line arguments as we mentioned it has no child processes it has no file or registry rights no network connections it has one cross process to the client server runtime subsystem uh which if you know what that is makes sense uh the client server runtime subsystem is basically like a separation of concerns you've got a command shell that might want to be able to reach out to low-level windows apis and all of that is done through this intermediary of the csrss process so you're typically

going to see this particular cross-process event but it tells you in the edr platform that this handle was opened from command shell into csrss with change access rights i'm like well what do change access rights mean exactly if you dig down a little further in the platform they don't make it immediately obvious but they'll tell you the specific api calls that were done here so process virtual memory operation and process virtual memory write you go consult with windows documentation figure out what those apis can do dig deeper in the platform and the edr platform is going to say these access rights allow this process to change the behavior of the target process all right well we should probably go

look at that uh this thing could be doing anything so we jump over and we look at csrss and this is what it looks like in the tree view uh not a whole lot going on in the screen here there are no child process events for csrss it has 33 registering modifications i looked at all of them none of them were interesting none of them relate to persistence mechanisms it's not reading any weird data out of the registry it has 20 mod load events they're all signed so we're good right uh two cross process events one from our command shell and one from its parent process which hopefully is unknown but there's really nothing if you look

at this from the 10 000 foot view it's not interacting with the file system it's not you know reading or writing weird data it has no network connections there's nothing obviously malicious here so we're going to move along let's go back and look at this other cross-process event so 17 milliseconds after that convo started there's a cross-process event from conhost into svc host again change access rights same api calls and this could alter the behavior of svc host so we'd better go look at svc host so we jump over to svc host in this particular platform again no child processes 478 cross brock events is that a lot for svc host i don't know is it

too few i have no idea it has 21 mod loads they're all signed so we're good uh one reg modification no network connections so we want to figure out what's going on with all the cross processing events

on this thing so we can load up the cross proc events it looks very similar to that dll screen i've found our con host process how unusual is it for con hosts to have a cross proc into svc host well it's this is a multi-page web ui so you get pages and pages worth of data and you can page through them as you go this is two of seven on this page and it's one out of 35 overall for this spc host process so it seems fairly common it's like seven and a half percent of that uh 478 cross process events are from conhost processes i mean is that is that bad good it's relatively uncommon but there's

enough of it that as an analyst i'm not sure it's something i need to worry about a bit of trivia here and this is where you're getting plenty of data and not enough information uh there's conhos parent process that explorer process the only reason i know that is because off to the side of the screen there's a bunch of time stamps here that have been redacted and i've verified that that explorer process started at the same time as the one that we were looking at earlier uh interestingly right above this conhos process there's a smart screen process who knows what smart screen is uav uac it's in the ballpark doesn't it isn't it used with a partner

with remote desktop basically transfers the train from one like one client to another i mean that's a good guess but no

yeah so the answer there was uh and i think you're pretty close uh basically when a user does something that could be potentially dangerous according to microsoft uh if you visit a webpage that has a questionable reputation uh if you download any active content vbscript powershell batch file those kinds of things uh and you're about to run something on your machine microsoft puts up this handy pop-up and says hey you're about to do something dangerous that pop-up is smart screen so this is really the first thing i've seen as an analyst looking through this that's giving me pause and going okay maybe something's fishy on this box and if anybody else you know sees anything here by all means

uh let me know i can't claim to be the expert but smart screen gives me a little pause here and the fact that happened right before this con host process that we're seeing weird behavior from it's really the first uh well it's not the first it's kind of the next link in the chain here it gives me some indication that we could be up to something no good uh so we've already kind of covered this is smart screen commonly opening handles in svc host out of the 478 cross process events this is the only smart screen process opening a handle into svc host so i'm going to say it's uncommon uh what is it we've already answered

that um interestingly though i mean i've looked at investigations before where i've seen smart screen and that's always a clue oh like go check the file system and go check the network connections from this machine a few you know right before the smart screen process started there's probably going to be something there that's interesting and in this case there were no file mods and no network connections in the few seconds prior to that smart screen process running so something is definitely weird on this machine but i don't know that it's malicious we've also got in this particular timeline their ctf line [Music] and again you know same questions is it common for ctf mod to have a handle into

svc host the tool isn't really giving me any indication except that i can see on the screen uh with the benefit of the time stamps that are redacted about every five seconds ctf mon is opening a handle to this scc host process so what is what is ctf bond ctf mon is the microsoft process that controls alternative user input so if you bring up the on-screen keyboard and weirdly uh it also has something to do with the office language bar so the fact that i've got ctf mod that has some interaction with microsoft office and then i've got smart screen running you can kind of jump to some conclusions here as an analyst uh with a little bit of

knowledge and maybe not enough knowledge there's also where fault who knows what where fault is yeah it's the windows error recording so windows error reporting service and this runs anytime something crashes in windows where fault's going to run uh and it collects data and will happily send it off to the microsoft mothership if you have it configured to do that in your environment and i think it's something you have to turn off uh so it comes that way out of the box is it common well just like ctf mon it's about every five seconds opening a handle into svc host on this machine something's weird on this machine uh we've already covered what it is and

then there's also run dll32 in the timeline here and we know from previous things that we've looked at run dll32 can be used for all kinds of interesting things like a memory dump of lsats well we could go down the rabbit hole here and go look at every one of these 478 cross process events and we'd be here all day and and we need to get detections out to people so that they can react to them and go and actually do something about it if something bad is happening so again as an analyst i might spend a few minutes looking at all this stuff but then i have to go back and look at this sort of 10 000

foot view and say what is sbc host doing there are no child processes there are a bunch of cross-process events there's 21 mod loads there's one registry modification no network connections and no file mods it's not obviously malicious but smart screen and this leaves the analyst in this state of uncertainty uh whoever made this meme as a genius uh you get the stormtrooper sitting in the hotel room at night wondering were those the droids we were looking for that's enough uh about the edr platforms and i don't i'm not trying to pick on any edr platform they have made life as a security responder so much easier than it used to be uh they're not perfect you know they all

have warts and they all have beauty marks but have having been in the security instant response space now for like 15 years i wouldn't want to be doing investigations without these tools they're super useful but they're not perfect yet just like the rest of us so that's enough of the tour uh you know stop the ride here i want to get off are you being primed probably so we've looked at examples right so the examples just to recap real quick you can miss the forest for the trees you've got so much data and so little information as we've just seen these tools for the most part they don't give us the base rates they don't tell

me yeah this thing always loads 33 dlls it's always these same 33 dlls i want that information as an analyst they're prone to false positives they're prone to false negatives they're utc time wasters they get confused when process id reuse happens which on busy servers happens all the time uh so they're not without their problems and their shortcomings but again i love the world with edr platforms better than the world without them so enough of kind of the problem space let's talk about the trouble with maps and you're like maps what does that have to do with anything well maps are hard as you can see here the experts at cnn misplacing hong kong into south america

and i'm going to tell you like in my mind edr platforms are maps so if you think about um maps this guy alfred kuzmi is the father of general semantics which is way beyond the scope of this talk uh but he recognized that [Music] you know humans perceive the world a certain way and our our perception is basically a map and we're pretty bad at perception there are lots of animals and bugs that can see things that we can't see and my dog can oh my dog's death now uh but for a while she could hear things at different frequencies that i couldn't hear and her nose is a thousand times stronger than mine so um

he's kogypsi is definitely right that you know we don't perceive the world uh as it really is and he basically had this quote where he said you know the map is not the territory and it's kind of a duh uh type of quote but he went on to enumerate all the problems with maps maps can be incorrect as we may have seen did conhost really spawned command shell did the svc host really load system management automation and maps are lossy they have to be they can't tell us everything so when i was goading you all earlier prodding you and saying what additional information do you want as an investigator we want the territory like i want a

memory done i want full packet capture i want all the logs i want all of the data from this environment if i could rewind the environment to a specific time and play it forward and backward that would be amazing we want the territory and instead we get maps right we get a lot less information than what's really out there like all those cross-process events you can change you know these could change the behavior of the target process they don't actually tell us what behavior changed if any so they're lossy they're not giving us all the information and they also require interpretation so you looked at a map before there's almost always a legend down in the corner of

the map the legend is a map of the map uh so map steve maps need maps need maps and these are all things that could alfred koscievsky enumerated about problems with maps so enough about the trouble with maps let's talk about systems of cognition so this book changed my life and i've got five copies of it up here for people that are interested uh including the person in the back who answered a question earlier uh this is probably the most highlighted book i've ever read and i've read it twice now in the last couple of years it's just full of all kinds of great information daniel kahneman if you don't know he was a nobel prize winning

economist and like kind of crossed over between economics and psychology it's a really fascinating book and he says people when engaged in a mental sprint become effectively blind and if doing security instant response work is not a mental sprint then i don't really know what is so as an example of uh just kind of setting the stage for daniel kahneman real quick i'm just going to go through these slides and you just shout out your first reaction you know what is this person feeling i don't think he knows he looks scared

[Music]

shock surprise so two people here guys guy in the white shirt [Music] he's frustrated she's trying to keep her cool one of these people is not a person he appears to be laughing and the other person looks to be grumpy and i think we'd all agree that this person is really surprised to be meeting swift on security [Music] what's white's next best move [Music] nobody so what you just experienced is what danny kahneman referred to as two systems of thinking so there's system one according to kahneman and system two uh and this is not a common original idea but he's done more to sort of proselytize the idea than anyone else so system one ninety-five percent of our decisions

that we make on a daily basis are system one uh system one is automatic it's effortless it's emotional it's intuitive it's how you're able to dodge things when somebody throw things throws something at you system two is expensive uh both in terms of the the calories it takes to use system two and from a risk perspective imagine you know over uh millennia being on the the uh savannah in africa uh our predecessors thousands of years ago having to concentrate on some heart problem and completely missing the fact that there's a herd of lions approaching through the grass because they're too busy concentrating on how much rice they should be exchanging or whatever the problem might be

uh so an example of a system two problem you know 347 times 67.4 most of us are gonna have to pull out paper or really concentrate on that chess puzzles if you're not a great chess player our system two optimistically uh down here in the bottom left system one sort of the implications of this i love this system two tasks can become system one with enough training and practice so like this uh chess puzzle if you showed this to a grand master i'd give you the answer really quickly because they recognize the pattern so that's one thing that is really encouraging we should take away from this we can all get better and we can

convert system two tasks into system one tasks uh other implications system two generally because it's lazy it accepts whatever system one says and this leads to all kinds of interesting biases uh which kahneman goes into a great deal of system one is what tracks familiarity with a concept or with a topic and this is where you know familiarity like if you repeat a lie often enough becomes the truth this is where this kind of thing comes from because system two says oh yeah system one knows all about this it must be true uh so we can kind of combine these ideas about maps and the problems with maps and system one and system two at least

this is kind of where my head is at in terms of detection space and how we can make improvements uh and kind of tie this all back into how we improve what we do uh some other implications of danny kahneman he had this idea about activated ideas uh which is you know the you remember the classic donald rumsfeld you know there are known knowns and known unknowns and unknown unknowns the concept of unknown unknowns like if you you have no concept that an attacker can inject into explorer and make explorer spawn conhost and inject into conhost and make conhost spawn command shell if you don't even know that that's possible you're going to immediately assume that your platform is

wrong there's a gap in your knowledge there you don't have this activated idea of what's actually possible so that's the activated ideas concept base rates uh and regression to mean this kind of tied together and we've already talked about those a little bit uh we'll show some examples of how we can take all these sort of ideas and use them to improve what we have to do as detection engineers so what do we do with this knowledge how do we make recognizing that this is good or bad as easy as recognizing that this kid is terrified or that this person is angry well here are some things that this isn't unique to red canary there

are other places that are doing things like this and you might be doing things like this in your own organization with detections that you're providing to other teams [Music] you can do simple things like color code things to indicate severity so we do this today if you get an alert from us and it's got a great banner on it it's probably not that critical you probably got some adware running that you may not want running in your environment but on the list of the thousand problems you need to address that one's probably pretty low so color coding is one thing you can do this goes back to the explorer spawning conhost and conhost spawning command shell

there's a lot of gaps of information there if you don't know that the windows console host always has command line arguments and never has child processes that's an activated idea that you're missing right all you see is all there is you see this list on the screen you're like i don't know what this means so this is an area where as sock analysts we can fill in those gaps and we can give more context and we can let people know hey conhost never runs without command on arguments and it never has child processes so the fact that you're seeing this is weird and we can explain why things matter so run dll doing the the lsas dump and for

those of you that are paying attention the reason that command failed earlier is because the word full was missing from the end of the command line so that full makes a difference uh we can give people context and say you know what's going on here as somebody that's not familiar with this data it might be a little difficult so fill in the gaps and get rid of those or get rid of that what you see is all there is by uh supplying additional information and here we can tell you you know the results of the action so there's a file that was written uh to disk called lsas.dump and we can also tell you when something didn't happen so

you've got a case where an office document was downloaded to a machine that we know is malicious it was opened on that endpoint but we never see the vbscript run so that's something else you can provide you don't just have to leave the analysts out there in the wind trying to figure out did ls really get dumped did it get copied off to a network drive or a usb stick has it been deleted do i need to go pull full disk image so i can find out you know what it was there and what credentials were in it we can give you additional context and these are all things that we should be doing as detection engineers

this is something i would love to see in the platform that we looked at earlier these are module loads show me that it's highly uncommon for this given dll to be loaded by this process that's not something the platforms do today and i wish that they would uh here's an example of a screen that a red canary engineer sees when we're looking at a process event so we've got several thousand detection analytics we call them they're basically you know signatures for bad behavior you can think of them that way we get customers security telemetry goes into a big pipeline we run several thousand detection analytics against them if they trigger against those detection analytics in this case there's 11 of

them listed on the screen here an analyst on our team the ones that aren't handled by automation an analyst is going to get it if you get something like this as an analyst you've got 11 different detections that triggered on a single process odds are pretty good something bad has happened we rate our detection analytics so we track them over time we know how often they convert to real detections a score of 15 is a perfect score every time this thing has fired it's been something bad happening so as an analyst i can look at this and go well my goodness there's 11 detection analytics that fired on this one process and most of them have pretty high scores

uh and that just reframes how you're thinking about the investigation you go from thinking well this could be a false positive to you know this is uh definitely something bad happened uh and you'll just kind of analyze it differently so uh i think we're just barely scratching the surface on things that we can do to make this job easier for people so quickly let's just recap and if anybody has ideas on other things that can be done i would love to hear them um so i've argued that edr platforms are maps right and defenders we want the territory we want all the data we can get instead we get maps and maps are lossy which leads to this problem of what you

see is all there is maps can be incorrect they require interpretation you've got to have experts who understand them system one and system two uh remember system one is automatic it jumps to conclusions it doesn't even know it's jumping to conclusions it doesn't know how big those jumps are system two is going to believe whatever system one says and this makes it really difficult to overcome biases because we don't even recognize that we have them um we also have this issue of what we see is all there is when you're dealing with lossy maps that's problematic on the optimistic side keep in mind that your system two tasks with experience and repeated exposure can become system one tasks if you've

ever learned to drive a stick shift you've converted a system to task into a system one task probably don't even think about it uh we need to minimize how much we rely on system two and this is really my focus area and something i'm trying to tackle uh and simple ways we can do this you can color code we can annotate we can measure our detection analytics and show those scores to the analysts you can convert system two tasks to system one task by studying and practicing so those are things you should be doing understand base rates and regression to the mean and actually show people that data you know whether or not it's common for a given dll to be

loaded by a process or not and again i think we're just scratching the surface if anybody has other ideas won't raise them during the q a i'd love to hear it and with that i will open it up to q a slides are available scan the qr code or if you don't trust qr codes there's a nasty url here that you can punch in so any questions

yeah what was the right move what was that what was the next booth what was the next move for white you know the answer uh move the knight into the position well that's not what the chess puzzle website says but

it's probably a good move i'm not a chess expert anybody else yeah so i really appreciate your color code concept but i'm currently dealing with like um not specifically an adr tool but like a step back of how do you convince people to even use the tool in the first place yeah how do you convince people to use the tool in the first place change is scary and i don't want to change how i do my normal day-to-day job and it's another thing yep yeah you know that's probably a whole nother topic i can tell you from my own experience that you know i have exposure to a lot of edr platforms there's a few more that i have slides

for they're in the deck but didn't have time to get into um i wouldn't want to do this job without those platforms even as bad as some of them are but they don't they don't recognize that they need that though yeah yeah people if people don't even recognize that they need it then yeah it's a hard problem yeah so show your presentation appear to focus on edr which correct me vermont is a lot about the host visibility yep what are your thoughts on this ability at the perimeter and the network uh that needs to kind of follow some of the directions that you're suggesting yeah yeah um i mean where i work currently uh we're largely

focused on edr although we're looking at branching out into color covering things beyond the the endpoint uh yeah it's a valid question about uh network level visibility and all that stuff and i've worked in a shop for years where we had network visibility and we ran a bunch of smart sensors back in the day and all that stuff is invaluable but usually it all comes i mean there are probably exceptions to this uh but usually when the bad guys want to go steal secrets they go to the end point eventually i mean there's stuff you can you can scrape off the wire but uh i think in terms of priority i would prioritize endpoint first

and network second but that's that's just me i don't think there's a right or wrong answer there yeah did you have a question are you good

uh you had asked um from a user interface perspective like what would make that particular product easier um it seemed like there was a lot of bouncing around between data clients like okay i found this thing but then i have to go search like three levels deeper to go find the next clue um i would suggest maybe a tree deal we can just have it expand yeah yeah there's uh i think there's some billion dollar ideas and uh for people that are good with web ui and adding nice features that would make all that easier and again for anybody that's interested just on a whim i bought five copies of this daniel kahneman book i owe one to a

gentleman back here if anybody wants one don't make me take them home i don't want to obligate you to read something you know it's a big time commitment but this is probably the most impactful book i've read in the last 10 years i highly recommend it if you want a copy come up and see me and happy to give you one take him up on that round of applause everyone's phenomenal