Passive detection doesn't work: lessons from a hunter of elusive

all right folks it's uh it's four o'clock thanks for stopping by my talk here um the title is a little bit uh scandalous it's meant to be controversial a lot of folks are investing a lot of money in security technologies and my job today is actually to eviscerate the entire security community which i kind of live in and influence and i'm going to do it for a really important reason so hopefully you can get past the the snarkiness uh my general social approach is to be really flippant so i'm going to do that um hopefully that makes this entertaining and and not a little uncomfortable but um it's meant to generate discussion so that's really where i hope this goes um

we're gonna talk about who i am um i'm gonna give you my manifesto which i posted on twitter just the other day uh i'm gonna take some super cheap shots at some very common security technologies that you should all be familiar with i'm then going to walk through the attack life cycle for targeted intrusions and we're going to point out some ways that we can be improving security through review of that that threat model which is only one of many models we're also going to look at some case studies that illustrate my central premise which is that these passive security devices by themselves are not sufficient to protect our environment though i will point out as a as a kind

of a pacifying control that they're all really necessary anyway and i'm going to explain why and then finally i'm going to walk through some recommendations some things that organizations can do to prepare themselves for the inevitability of one of these types of compromises and hopefully leave enough time for you guys all to interrogate me but please be gentle because i'm soft so myself i'm a 315er please no one boo i grew up in ontario county and i was a student here for a number of years my background is informed by systems administration as a teenager i inherited a number of linux systems that i maintained painfully through the 90s so this before modern package management

this was before um you know you could really easily compile kernels this was all very painful and as a result i found myself with a career but i really know that no one really told me in the late 90s that being a sysadmin was a thing i grew up in a rural community so there wasn't anybody to give me that that perspective i was also a network engineer for a really long time i spent about 15 years in rochester as a sysadmin and network engineer working for data centers working for telecommunications companies working for isps working for academic institutions and i feel like that was a really formative part of my career in fact when i talk about sysadmins

later in this talk i'm going to explain why they're really great incident responders and why they should be tapped for those roles now since 2011 i've been with a security company called mandiant uh recently acquired by fireeye a few years ago uh where i've been leading investigations if a major intrusion investigation uh associated with a nation-state or financially motivated actor added a year to your life i would be about a century and a half old so it's a lot of different investigations a lot of different verticals from defense engineering financial institutions academic institutions health care and uh rocky was kind enough to actually tee up a couple of the investigations i've led in the last three years so

although i won't tell you which ones they were some of those have been very high profile investigations where a substantial amount was invested in security and a substantial amount of security failed so i feel like the two talks actually dovetail really nicely now i also teach i teach incident response but i've also spoken really prolifically about some of these computer network exploitation exploitation techniques uh leveraging things like wmi powershell and other native frameworks i've spoken at length about building incident response program so feel free to use some of that supplemental material as you're as you're going out and looking for additional sources of information to build your own programs and i've teed up some of those

presentations so when i release these slides later you're welcome to to go out and look at some of those other uh those other talks some of them are interesting others a little less so um and occasionally i tweeted people and that has mixed success so this is my manifesto this is really what it is so incident response is a human task it is absolutely a task for people you cannot trust a machine to do instant response for a lot of reasons we also want to make sure that we acknowledge that incidents will will wrap up we will end the response cycle we should begin preparing for the next one immediately we should always assume there is a next

one many of the organizations that i've dealt with that have been compromised and recompromised felt they were one and done oh yeah we had a breach yep okay we had outputs from that cool we made a lot of changes now we're good and that is a very dangerous place to be we also want to avoid becoming reliant on technology to do these things for us i think a lot of organizations buy the boxes they buy the big shiny box they spend a million dollars on boxes and they're like cool response done i got a machine definitely not going to go find a qualified candidate to become part of my program and be responsible for machine hurting for driving the

program forward and when you invest in boxes they will let you down boxes are actually really bad at incident response and i'm going to point out some of the ways that boxes are really pretty bad at instant response so intrusions happen attacks succeed and leadership leadership is really the group that struggles most with this because they're like i spent a million dollars in that box i spent a million dollars we had a breach we had an investigation we learned things why did i get re-compromised i felt like from a business risk perspective we were good but because they're buying boxes and not necessarily hiring qualified people they're not getting the message they're not being educated properly and so this

is really a pro a problem that we create um in our industry the fact that you can buy a box and the box solves problems and um it's also one that we are prepared and empowered to correct so all of us you know any organization we work for we can correct this trajectory this is within our power and hopefully today you'll get some ammunition for that um so this is kind of where i begin talking about my cheap shots i'm gonna keep it really quick because um this is the one that's most likely to be inflammatory um we have a saturated security ecosystem there's all kinds of boxes there's all kinds of tech out there those pieces

definitely fulfill certain types of capabilities so i thought i would define reactive or passive technologies and in short these are things that tell you about something that happens but that's all they tell you so if we look at some of these examples here you know firewall firewall says hey a bad thing a bad thing tried to happen i left it in a half open state that's a funny firewall joke there's going to be a couple of these um antivirus solutions hey someone created something bad on this endpoint so i deleted it there's no longer evidence of the thing i got rid of it okay yeah that's very helpful um ids ips a bad thing attempted to happen or succeeded in

happening so um i generated like 3 500 alerts i hope that's cool um and so whitelisting whitelistings approach is like hey something tried to execute on this endpoint and i'd never seen it before which is the very definition of badness to whitelisting so we're good we're done um multifactor it's like hey someone tried to log into this box but they didn't supply me with a numeric code so i ignored it because that's bad um and then vulnerability scanners are like i don't really know what bat is um self-signed certs are pretty terrible but like this remote code execution vulnerability it's probably fine um you know so this is the way that some of these technologies perceive bad things

and so if we look at this as incident responders with our informed lands this is a very different ecosystem because we approach these devices as sources of evidence these devices do not care about evidence and the evidence is all the context so if we look at a firewall you know hey what around this event happened that might be relevant whether it succeeded or failed because both criteria are incredibly important to us as responders maybe the firewall blocked a certain amount of traffic from a source ip address but before it did that it let through a whole lot of traffic that we're really interested in well now i got to go interrogate the firewall to get that metadata to begin

informing myself about whether i need to do anything else and because the technology blocks but it doesn't answer questions about the context around it you know again that that before and after i might not be properly informed so yeah the fire will block something cool we're done the firewall worked it did its job but i don't know about the jobs it failed to do um if we look at antivirus antivirus is everywhere it's a necessary evil we need to have antivirus everywhere um but let's say antivirus blocks something okay cool it did its job i've got okay windows credentials editor some random rat that's on the system antivirus caught it deleted it logged it okay well

how'd the attacker get into the environment did they get onto another system and then remotely interact with the target where this where this alert occurred we'll have to go investigate that it didn't solve any problem it blocked one thing but it gave me no context around the larger intrusion now i don't know where they came from i don't know whether maybe the attacker unpacked the malicious thing from another thing that antivirus didn't think was that bad um you know maybe the attacker downloaded some sort of archive unpacked all their tools which antivirus had never seen very signature driven and now i've got to go and investigate that so i've i've given myself an additional task

because antivirus did its job with ids i ips you know no one looks at 3500 alerts i'm going to stop right there tuning those devices is really critical and most organizations i deal with and i've dealt with hundreds they're not properly tuning any of this infrastructure from an ir perspective so they don't look at it through the ir lens they think oh i've got my my ids i've got my sim they're collecting all this stuff i don't know if i'm breached so i don't know if i'm collecting the right stuff and then after they learn the artifacts of a breach they might tune it a little bit they might get a little bit better but that's an incremental change they

don't substantially improve the reporting of those devices um and then whitelisting yeah we don't know where the attacker came from we don't know if this is admin misbehavior like maybe the admin really did think windows credentials editor was the best way to reset a password i've seen that um maybe renaming a native windows tool wasn't seen as as really malicious but whitelisting is like i got a name mismatch i gotta i gotta dig into this i gotta i gotta register an alert um and stuff like multi-factor you know how common are these occurring what's the frequency of occurrence of these events the device doesn't know to report it again this is kind of filling an

investigative workflow which is how i see these technologies succeeding but that's not the way organizations are using them most of my clients when they when they've dealt with this infrastructure they put it in place and it's kind of set it and forget it in fact i've seen companies that were breached that fired a bunch of people who became experienced during the course of the intrusion and then they came back later and got re-compromised they're like i don't know how this happened we spent two million dollars on fancy boxes i was like yeah well you fired the three people who were most empowered to solve this problem for you um so it becomes really challenging and as an investigator who sees the same

organization you know re-compromised multiple times i consistently see the same issues i see a lot of issues where folks are focusing on boxes investment and infrastructure and a lot of that i think is this this concept that all advanced attacks require an advanced technological solution where in reality i think a lot of these require a human solution now technology is crazy important i don't want to understate the value of technology but i want you to realize that technology solves only one problem for us it fills that pipeline of things it's a lead generation tool it blocks a thing it stops a thing it alerts on a thing and then we know that we have to go validate that activity we

have to begin kind of a mini response process and for organizations that want to reduce false positives they want to be able to manage these aspects of their vulnerability management program or their threat detection program that's a lot of effort and they're thinking about hey i need fewer people more tech and i'm going to flip that i'm going to say you need more people and less technology you need to get more bodies especially experienced bodies as rocky pointed out experienced people we're at negative unemployment we have a zero percent unemployment industry the problem is if you look at the most experienced people those people are such high value targets for every other company in the world

you're really facing crazy competition as a business very challenging to satisfy this there's lots of ways that we can accomplish that however so this is where i pause because i just kind of eviscerated the security industry um i don't think that this is necessarily a bad thing i think that if we can acknowledge the role of technology in our security programs we'll be better positioned to be successful i don't think that means less investment i don't think this necessarily means changing our approach to responding to threats because we can have a really solid methodology we can we can staff those differently but hopefully through a review of this material we get to a slightly better place

so let's call failure relative um all of these controls have a place firewalls you need fireworks i just i just read about a financial organization 80 million dollars because they didn't use firewalls they routed stuff but no firewall so that's a pretty substantial uh misinvestment that's that's one of those things where simple even a simple basic firewall would be better but i also want to point out that not all threats are created equal and this is because threat actors are learning from us the more information that we put out there to benefit each other to improve our own capabilities we actually enable threat actors a little bit more and some of the things we've observed as

investigators around the world is threat actors are getting really clever one threat actors are moving away from compiled binaries now this is a deliberate move to avoid endpoint detection so using interpreted scripts whether that's vbscript jsk jscript powershell whatever it is moving away from a compiled binary makes it substantially harder for whitelisting to be successful attackers have learned this because we keep talking about whitelisting things like using unique network indicators unique ip addresses fully qualified domain names for every victim org that means that if you develop intelligence in your organization and you share it through a threat sharing group threat actors know about this process and so for every target they create new nbis this makes it really hard to write

signatures especially if you've got a really simple detection framework now this doesn't necessarily apply to things like the c2 communication algorithm or ways that they're protecting and coding encryption etc that's a little bit different this is just for those network indicators with no other context they're compromising trusted infrastructure i was talking with a couple students earlier we've seen threat actors compromise organizations and use c2 through google code we've seen it through microsoft tech net we've seen it through other code sharing sites we've seen github abused for this purpose these are things that your organization will whitelist because you're going to have developers that need access to them you're going to have operations personnel that need

access you can't just block all traffic to those sites and attackers have learned this through a painful trial and error process and for us it's harder to see that but for them they're attacking dozens of organizations at a time and they're measuring their success and failure the same way that we might measure our success and failure on a single engagement because for them those campaigns are one engagement we're also seeing attackers go native use almost entirely native frameworks wmi powershell i've seen attackers exclusively use samba client an ldap client to manipulate systems in an environment and if you permit this for legitimate purposes attackers abuse that and they also know how your systems work i've

seen attackers compromise sccm because they know sccm sometimes leverages wmi they'll compromise the sccm management account so that it looks even more like normal traffic i've even seen them manipulate the names of those specific sccm tasks so that when they're pushing out malware or performing credential harvesting from a network perspective from an sccm server perspective it looks completely legit and unless you happen to be sitting in memory on the target systems which isn't always practical incredibly difficult to detect so attackers are learning and they're escalating on us but if we keep relying on the same failed technology to solve all these problems we're going to have a really bad time and then i had the same blade runner

quote down here as rocky use and i took it out while he was talking about the threat model process but it applies i mean i've seen a lot of stuff i've seen threat actors do some pretty crazy things i've seen a lot of inventiveness and innovation and i think that there are some ways that we can combat this so if these technology solutions don't solve a problem for us if these are not black and white prevention and detection well what good are they and the first thing is they save us time and money they catch some of that low-hanging fruit they block some things that allow us to detect they satisfy those requirements and they do a pretty good job at it if

we if we put them in place in a layered approach we also have to rely realize that a preventative control is a detective control if the firewall blocks something it's detected something it's given me a source of evidence to go query it's given me a notion that something went bumping at night and that allows me to mobilize myself to bring a whole team together pulling together multiple sources of evidence from multiple different systems hopefully to put together a picture of what occurred and maybe at that point i'm done i stand down because there's nothing detected there's nothing serious going on maybe it is in fact the false positive and what i've learned is i need to tune a

solution or maybe what i've done is i've detected the early salvo in what's about to be a much larger compromise so being prepared definitely definitely helps us it also allows us to extend our capabilities into environments where a single person just simply does not have the bandwidth you know i've done investigations of 100 000 endpoint environments you know quarter million endpoint environments if it was just me that would take forever just me with scripts that's you know that's a whole lifetime to get the details i need if i leverage tech in these different uh locations these different regions i can extend my capability multiplex my data collection and analysis approach and we'll talk a little bit about some

of the ways that we can use automation to kind of solve that problem and you know rocky brought up compliance compliance kind of an ugly word compliance is really important mostly because if you're in an industry that regulates data and you're a custodian of that data you are beholden to those check boxes that is unfortunate and i'll agree with rocky again that you can be in compliance and be insecure but if you are secure you very likely are also in compliance it's about going above and beyond using compliance is kind of the launch point um if you do detect intrusion these technologies can be sources of evidence you know think about netflow metadata it's not a pcap you can't parse out what

really happened but let's say you wanted to do statistical analysis of lateral movement well netflow is great for that you can keep netflow for two years takes no space at all you can apply basic data science principles to it heck you might want to make that part of your detection approach pro tip there um these are these are infrastructures that are already out there we've already invested in them well let's just co-opt them for our detection and response let's not wait for them to tell us about a bad thing let's see what they're already seeing that they're letting through see if maybe we can manipulate that and of course blocking and tackling these are these are basic security controls but

they give us a dialogue with leadership and you know rocky talked about metrics i'm going to talk about metrics too metrics are huge metrics are the common language of business risk if you are not reporting metrics on like at least a semi-annual basis your leadership probably is not well informed now it's impossible for us as folks on the ground to go to leadership and be like hey mandatory meeting you gotta you gotta receive these metrics leadership has to be invested but you're the best advocate for that because you actually understand what can happen if they're not informed you can communicate to them and using that common language of business risk they can at the very least make a choice

to be informed or or uninformed so targeted intrusions everybody talks about kill chain everybody talks about our model i like this model because it maps to human activity because i can look at what a human does and i can craft this into a narrative so this is this is pretty important but what i'd like you to take away is the closer we get to initial compromise the better we're getting if you take nothing else away from this diagram the closer you can get to day zero the better you are now chances are you're going to miss some things attackers are actually really versatile about how they compromise organizations most folks think it's just spearfishing or strategic web compromise

or maybe like a web attack i've seen threat actors compromise legitimate third parties at persistent vpn tunnels i've seen threat actors weaponize devices that were backdoored and then ship devices the old usb in a parking lot trick yeah it's a great trick because it works i've seen cases where attackers have compromised network hardware and then put it up for sale knowing that it would find its way into a target environment of interest now that is that is something that no organization could ever properly plan for you never just be like well we're going to just go direct to the manufacturer because maybe your budget doesn't allow that maybe you don't have that flexibility so knowing that you're going to be

compromised prepares you to try and find stuff early by looking frequently and having methodologies built around that technique would you in that case would you would flashing the firmware on the device back to factory take care of that so flashing the firmware is a great option but what if the device has multiple modules like look at a cisco appliance where the network traffic module has its own entire subsystem well you can totally flash the core device and that's never gonna touch those individual modules you're gonna have to go through a very painful process and of course you can go through that process and and still not be entirely confident that you are successful so it's it's a little

sticky and that's why i say that it's not possible to have every contingency covered if you assume you will be compromised you will make better decisions about how you look for evil and that's really what this is about is making good decisions about finding evil so i've got some really snarky case studies here these are all from actual investigations um 2015 i had a client who pushed antivirus out to a group of domain controllers for the first time ever these had been in production for years they had always assumed antivirus would affect performance um they were actually right but it was a bad decision not to push it because a couple years prior an attacker had dumped some hard dump some

software on these platforms which antivirus found it installed and started scanning which is what antivirus does did its job and it said hey i found this bad stuff now as we did our investigation we actually found out the attacker had been pretty active in the environment now if this company had decided to deploy any virus these dc's as part of their normal rollout process antivirus would have had the ability to inform them about the presence of this malicious software certainly they could have begun an instant response process because they forego that step they put them in kind of an uncomfortable place now this attacker was there for more than 14 months that's a long time you know having

antivirus is a common control really important let me back up one second here one of the interesting aspects of this i almost skipped past it because it's kind of the core here when they looked at the timeline around this and lateral movement to their dc's they looked at their firewall logs and they thought okay well one of our dc's was exposing rdp maybe there's something related there what they found is that there was a sql injection attempt around the same time and their firewall did in fact block it however about midway through the attacker took over and you could kind of tell the cadence of an automated sql injection scan versus the manual probing and those commands were base64 encoded

which interestingly enough the web app interpreted properly but the firewall did not see that as a sql injection attempt i would name the vendor but i'm pretty sure they would sue me um and this is one of those cases where yeah the firewall did its job somebody probably saw the alerts and thought nothing of it uh we actually couldn't find a single person at the company who remembered reviewing those alerts or seeing those alerts we didn't see a trouble ticket open and their ticketing system i think they use remedy went back like two years didn't see anything so this is one of those cases where oh the firewall did its job and we're happy because it did its job but if we only

dug a little bit deeper we would have started to see those base 64x xp command shell references would have maybe seen some rfi attempts gotten ourselves into that response process a little bit more fluidly now firewalls do satisfy a lot of requirements i'm going to call some of them out because i think firewalls are really important um you know we like our firewalls to you know digest some of these indicators like one of the features of a lot of these advanced firewalls is oh we have a threat feed we'll tell you about bad stuff that we find you know things like things our customers share with us or things that we digest from open source

intelligence things derived from malware analysis we've got a whole business unit that does that but firewalls also you know miss quite a lot just because they don't give us that great context um so you know maybe our firewalls can look up things like trusted and untrusted c2 you know maybe communication from you to microsoft update we're going to call that trusted maybe connection to dynamic dns result fqdns are bad maybe we even want to apply something like trusted and untrusted web domains to our firewalls these are all capabilities the firewall can apply that give us a good position gives us a better position to detect um you know malicious c2 but unfortunately most firewalls don't

do any kind of internal analysis even if you have an internal firewall chances are you're not looking for things like lateral movement rpc or dcom soap connections now some of these are frustrated by the fact that soap connections are encrypted but even non-encrypted traffic something that's essentially like human readable the firewall's not really looking at that it's just essentially doing like ip configuration or network configuration rule setting um but the problem with firewalls is that when we write rules about threat activity we are chasing a known threat like we write them as a response we're not really writing them preemptively you know this morning jared was talking about egress controls i'm a big fan of egress controls

firewalls that block outbound by default are huge how many people think a server should initiate outbound connections from their enterprise at will to any fully qualified domain name oh good i saw no hands cool you all passed past the friendship test um we want to make sure that if we have stuff in our environment it can't just get out to the internet because that is like the number one hallmark an initiated connection from a server outbound to uh to c2 that's a very bad thing you'll have exceptions and hopefully as you watch your environment over time you'll be able to identify them preemptively put in those exceptions so that you can get out to microsoft update that you can get out to

the various infrastructure sites that are necessary to do your job but i think a better solution would be set up a network specifically for that and then because you've isolated it you've given yourself an environment that's going to tell you about compromise because when those exceptions fail that prevention turns into detection and it allows you to act so cool firewall worked um but firewalls are fallible and we want to keep that in mind we'll be really focused on how limited a firewall is antivirus now antivirus can have a lot of uh similar parallels here in 2014 i had a client who wanted to compromise assessment compromise assessments kind of like hiring me to be your internal security

team to proactively identify threats full disclosure um and what we found was a bunch of de roos b variants from about 2011. well that's interesting um derusby which we call photo is a pretty common malware family we found a bunch of different variants in the environment each variant had its own md5 its unique file name its persistence mechanism was all done through search order hijacking but the actual persistent binary names you know were things that you might expect legitimate software to have they've done a little bit of studying you know what what's vulnerable to search order hijacking they name their dlls appropriately and when folks looked at these systems for normal i.t operations they didn't notice

anything weird now unfortunately this was detected through a compromise assessment so we knew that they were owned we were able to do an investigation to scope that compromise av had no ability to match on any of this metadata because it was all file based they even had a solution that claimed signature list detection or heuristic detection machine analysis buzzword soup stuff it did not work short story it failed um now we never got an answer from the vendor about why it wasn't successful but the bottom line here is we found malware in an environment for which there were no signatures no functional signatures though there have been signatures since um and while this remained untacted the

attacker had pretty much full ran the environment it was all installed in a privileged context now av is really super important having av on everything even linux systems even non-windows systems really important having av on your phone could be super important it's a computer if i could get av on my apple watch i probably would um and the rate the message to take away here is if commodity stuff is in your environment and is successful targeted stuff will be successful it will absolutely be successful as as a tool any virus it's not going to scope an intrusion for you because attackers are going to use things that antivirus doesn't identify as malicious antivirus like whitelisting

also succumbs to that blind spot when it comes to scripts or interpreted language because it doesn't know it's weird it's never seen it before it can't look at um a script that has two forms of obfuscation and say oh well clearly this is bad because i can't read it it just passes and so things like that put us at risk the call came from inside the house by the time av tells you about something you're already compromised and at that point it's really scoping your intrusion now sometimes you get lucky sometimes during that initial phase uh the attacker is using something old enough or um obvious enough that any virus does is have the ability to detect it but

just because it succeeds in blocking the malware on one or two systems doesn't mean that that antivirus was pushed everywhere maybe in your enterprise you have inconsistent deployment so we still have to do an investigation we still have to begin scoping that um ids ips and again this is there's kind of like a repeated message here of blind spots that we need to be aware of you know we had a client who came to us they were concerned that other industry organizations were being targeted and they said hey we want to know if that's us are we are we in this group too and what was interesting is they were talking with some other companies that

had been breached and some of the indicators of a prior phishing campaign came up they're like oh yeah we remember that we have some documentation around how we stopped that uh yeah that sounds eerily similar but we didn't think it was serious we didn't know it was targeted so we did like a mini investigation and cleaned up they actually did a really good job i was impressed at their response process but their ids ips solution it caught a lot of stuff that allowed them to scope infection but it didn't catch anything related to lateral movement in fact on one system the attacker had used rdp terminal services between two endpoints and the ids completely ignored that even though

one of the source systems was marked as infected because of that the attacker had a foothold on a new system they cleaned everything else up and the attacker knew ha ha now i know i've got a foothold they can catch my back doors but they don't see me on this one endpoint so they deployed a web shell this happened to be a dmz system that was out of scope for ids and through that one web shell they were able to remain persistent in the environment for over a year so this is one of those cases where certain amounts of regulated data were at risk there and that was not great for a certain regulatory body

so ids ips does a lot of things it's pretty important for us but one of the risks here is that the people who develop these technologies they might not know what you know they might not be investigators they might be application authors they might be folks with a computer science background who have had no formal security training they might have never even heard of some of these scenarios so how do they write a signature to detect it how do they write language to express that and the other thing is expression language is a function of the technology you know you go to any ids and write an alert well if that expression language isn't sophisticated enough to document the

things you care about the things that you know are bad well you've got technology that's incapable of helping you so you've got a huge blind spot that's where you have to rely on other technologies to help out um this is a i think this is a comment that's attributed to gus grissom this rocket was built by the lowest common denominator i think 10 seconds later he was incinerated um ids ips solutions solve one problem for us they don't solve all of our problems i've actually had clients tell me oh i don't need to worry about this threat our ids will catch that i saw the signature and then you have that conversation you're like well what's the signature

for and they show you this thing that contains four ip addresses and you kind of scratch your head because you're like okay what about all the other ttps associated with this actor like maybe the way they move laterally is really recognizable we can get metadata from our environment that informs us about that maybe their mission objective tells us about the threat actor and there's something that the ids can capture that will help us or maybe there's another associated system but because the the client had believed they were protected they were actually unwilling to hear that they might miss something um it also forgets the fact that attackers evolve like we look and say oh hey this

attacker they you know they tunneled uh rdp through a web shell that's apt whoever okay well what if somebody else read the same report you did and they decided to adopt that technique because they thought it was nifty or effective well that's a case where the attacker has learned and we're once again we're trying to we're trying to play catch up and then app whitelisting and i i actually can't knock apple whitelisting except we've had cases where attackers have used powershell scripts or used windows scripting hosts native ability to interpret jscript or bb script all of which you can and i've seen backdoors written in and apple whitelisting completely ignores it because if you enable

blocking of scripts you break a lot of the sysadmin functions that are necessary for your business and very rarely do folks go and audit the uh the stuff that's allowed through app whitelisting and both both steps are really important um i want to be sensitive to time so i can take some questions so i'm going to power through this um but if there's any questions about this particular slide please let me know uh-oh a thing happened come back um so we want to make sure that we understand the blind spots of our technologies because the blind spots are actually as important if not more so than what the technology does for us um and i wanted to kind of end this section

i consider this a high note might be might be debatable vendors want you to believe that if you buy their box or their solution you can live in a risk-free environment you can live in a threat-free you know wonderful utopia it's not a real place it is a pleasant fiction but it is still fiction the most important thing that you can acknowledge now is that you will not find things you don't look for these technologies are not the solution to that however we as human beings have the ability to leverage any tech in our environments whether it's security tech operational tech logging infrastructure data structures these things are our tools and we can build whatever we want

with them so it's actually cool we're gonna go we're gonna go on to the badlands but it's fine because we have what nothing can be what no technology can emulate we have expertise we have human beings who know stuff and through expertise we develop process and methodology you can take an incident responder who's even been through a single incident and they were going to know something more than maybe a person who's been you know watching glass and a sock that person has an instructional opportunity and we can build upon those to create efficiencies we can we can make this a little bit easier we can also track our success rocky talked about metrics i love metrics we

want to focus on ways to quantify what we do one this gives us a common business language we go to leaders and we can educate them about what we've been doing and how we've been doing it and how we've improved over time but it also gives us the opportunity to measure ourselves to self-assess in a neutral way because if you're using some kind of color scale some non-quantitative scale like oh we went from orange to green you have no idea if you've gotten better you have no idea and you have no way to estimate how much better you are we also want to make sure that the experts who are doing the work are selecting the technologies i've had

managers come to me when i'm on an investigation folks who represent leadership for my clients and say oh well we bought blah i was like oh what what's blah do and they walk through this list of amazing buzzword soup and say oh well did any of your team members select that technology they're like no i you know one of my buddies over wherever we play golf said i should get it i was like wait you just spent 200 000 on another shiny box we don't know where it's going to fit in the ecosystem we don't know what it's going to satisfy but worse we are now putting the weight of learning that technology and incorporating that

technology on somebody who might already be overloaded we might have a person who's already at the limit and now we just dropped another task on them when they could be spending that time looking for more threats scoping scoping existing intrusion and finally we don't stop we're never done the job is never over which is actually the great part about it is because it doesn't necessarily matter if you improve during this incident there's going to be another one you're going to have another opportunity to get a little bit better and if we're measuring our success we're going to know how much better the next time around so takes a village you got chocolate my peanut butter red

teaming is a huge boon for us because people with an offensive mindset who have an experience compromising networks they're going to understand attack methodology they're going to be there to allow you to test your controls but not just that test your response process it's the most important part you know if you're just doing pen tests to find vulnerabilities you're missing stuff because an attacker can leverage these native frameworks in your environment that leave very little forensic evidence and if you're not looking for that in a controlled setting well you sure as heck are not going to find it in an uncontrolled setting so getting these people on your teams very very useful um as a side note

sysadmins and investigators make awesome red teamers because they understand the outcomes of some of these different functions and so they look at it objectively and they can say oh well i found these things here's how i would clean up here's how i'd cover my trails and then the red team say oh and i learned a little something but the blue teamer can then say and here's how i'm going to preserve that evidence so in case you try and delete it i still have visibility so it gives you a great control sysadmins make great red teamers they have a security mindset kind of already they understand what normal is so abnormal to them is a thing they can key

off of and once you have teams built they play off each other very nicely in a non-adversarial way bringing in an external party is another good option but if you already got expertise in house these are people who know the lay of the land let's harvest that let's harvest that knowledge because the threat actors have to learn that we've already got people who know it let's take advantage of that phase two process so i started with people process almost the most important part you can live off the land if you know which phases of the attack life cycle leave artifacts and where you can start to collect those things analyze those things and that's going to empower you to find

future compromise in fact if you build methods around this around this approach it's going to empower you much more than investing in some technical solution buying another shiny box each one of these phases is important to a compromise and in some cases they're mandatory for the success of an intrusion but it doesn't mean that you have to say okay well we're going to start with initial compromise we're going to just master detection of that really look at the entire life cycle look at it holistically and say okay well what do we have for email logs what do we have for process execution do we have the ability to collect shim cash from all windows systems

do we have macs in the environment can we grab all launch statements how do we do that what are the technical controls necessary to support this process and of course automation is one of the best ways to do this automation allows us to work at scale in an enterprise to pull things back but also to start running through common forms of analysis so things like data analytics which are mostly a function of statistics and math we can automate that programmatically pull it back stack it let a computer do that that's not a human task i want to look at the output of that look at that least frequency look at that most frequency look at things like least

common substring for network activity see what does that mean for my environment am i finding things through these data science principles until you have the data you actually can't apply that knowledge and there's a human type of knowledge a person can say okay shim cache i see ds query ds get who am i and i see something called wce aux.dll oh geez windows credentials editor unpack library i better investigate and that's a human decision now you could technically automate that that very simple type of analysis but maybe your sysadmins routinely run the same three windows native commands and suddenly you're starting to see other commands like quinsta query q user things that your admins don't normally use and you use that

least frequency of occurrence to see oh that happened twice i've been four times everything else like 30 or 40 or 500. now i have something statistically significant to key off of and it's a great way to start detecting but again you have to start getting the data first before you can apply some of this and these are exercises that i recommend every organization practice these are classic preventative containment operations want to practice blocking an ip address validating that blocking it was successful use a dummy ip doesn't matter make sure you can do it and capture metrics about how long it took you want to know what that's going to be like because when you're fighting fire

this is a really important step sink holding dns records setting that to local host or some catch-all bit bucket or wherever you want it to be practice it test it how long does it take you to update those ttls like is that really short is it really long what's your process is it dns internal is it external do you have to contact a third party do you have an sla these are really important questions when you're dealing with a nation state that's stealing data so you want to be able to contain contain a system that's infected take a system that's infected maybe you don't know how it's infected yet but you want to make sure that beaconing stops

you want to know how long it's going to take to get that maybe you've got offices on the middle of the woods maybe you've got sites with no operations personnel what's your plan for for that contingency how do you how do you address that because again if an attacker is a foothold and you can't get somebody out to that site maybe it's a trailer in south dakota it's going to take an hour to get somebody out there what are your options because if you don't know what they are you're going to be in a bad place when there's an intrusion be able to go through all of your sources of evidence for some of these

common indicators like ip addresses or dns records have the data in one place and practice doing queries this is a thing that you should do periodically uh but especially as your organization changes as you absorb new entities as you change personnel size these are these are exclusion criteria for some of these uh tasks to be successful so we want to make sure that we we test against them match up a dhcp lease to uh a system like take a dhcp record and go back a certain amount of time be able to match them up because again it's a really important task if you can't go back in time to find some of these workstations that are infected that means they're

going to come back on the network and you may not have visibility where they come back maybe things like vpn split tunneling works against you in this case and so you want to know you want to have a contingency for that event um frequency of occurrence of ip addresses if it happens all the time if it's happening on our routine schedule well that might be beaconing that might be a thing you want to look at so apply that knowledge to the data you've collected um do an audit of accounts with multi-factor access look specifically for accounts that are in pin or token only mode disable these accounts if you can but check to see which ones have been

changed and how recently see if that is an active uh token account because we've seen attackers compromise two-factor solutions it's actually not hard if you've got access to a two-factor solution and it's a domain joint system pretty easy to compromise even things like exportable machine search pretty easy to get out even though they're marked as non-exportable plenty of tools that'll facilitate that for you we also want to make sure that we have the ability to do an enterprise password reset it's the number one thing that i recommend to my clients because it's the most painful you've got 10 000 service and application accounts those are in scope have a plan consider changes to your active

directory environment that facilitate it and of course be able to look at systems where your critical data is because if you don't know where it is if i say hey laptop x has been popped what's on it and they're like i don't know i don't even know who john smith is pretty sure he works here he's got a laptop well that's not a great place to be in especially if john smith is responsible for say all your social media accounts or maybe john smith is the guy who manages rsa maybe john smith is the guy who is the gateway to your accounting system and he is capable of executing ach transfers all these things are

really bad things to not know so start answering those questions now before you're breached and of course technology most organizations have firewalls they have web proxies that have some kind of click through they're going to have the ability to implement multi-factor these are all very common technological controls and they all have value but they don't they don't by themselves answer the question of have i contained an incident have i detected everything have i scoped it all they really do is fill that investigative pipeline we also want to focus on things like the ability the technological ability to reach out to systems to pull back artifacts because if we can't get those answers very quickly we're not going to

know what the significance of an event is we might not be able to detect if something's a false positive for maybe hours while an attacker has a foothold on a system well we're desperately trying to call somebody to get us logs or get us the contents of a directory that's a very painful place to be and kind of like my capstone here is you know technology is where we where we end but we start with people people first people who know things and we allow those people to develop methods and processes because that's really the value of incident response those are the things that detect and prevent compromise and you're never going to prevent compromise entirely it's it's a myth

i've entertained it many times i think it's kind of a nice fiction but i think if we can get very close to initial compromise and we can get uh answers from these systems quickly we're going to be in a much better place to respond cool and now you guys have heard me shoot my mouth off i encourage you to ask questions

yeah yeah so i'm a big fan of cancer cancer written by dave hall formerly of microsoft i believe in john turner um cancer is uh powershell and wmi based um there are a lot of risks associated with opening that up and just like rocky suggested i would say set aside a proxy or vlan where those requests are allowed to come from and they're blocked from everywhere else i really like grr power forensics is developed very recently i think that's jared atkinson primarily the developer there but power forensics is really cool because it allows you to at scale leverage uh the powershell capabilities of every endpoint um what's kind of interesting about these frameworks is they look a lot like

attack frameworks if you think about the way they're structured they're module based they allow us to remotely access systems they allow us to run queries which is similar to you know a lot of recon frameworks and i think that that parallel is appropriate so what are they doing yeah so the ability to enumerate the contents of the registry uh registered services running processes the contents of directories uh the ability to interact with other subsystems like the wmi subsystem to run queries very powerful capabilities um all of these solutions are free but just be aware that if you allow wmi everywhere if you allow powershell everywhere when threat actors leverage those capabilities you're gonna have a bad bad

time i've i've talked about it before i have a whole presentation on wmi uh forensics um the short version is there's not much what about something like swamp which is uh splunk um so i have a lot of feelings about aggregation and i feel like splunk is really a very nice framework but maybe you can't afford splunk there's elk stack i don't care where it goes put the data in one place and then develop processes that are repeatable that are documented everyone's cross-trained in that's the most important step whatever the aggregation platform is does not matter so a lot of this seems to play really well to enterprise class numbers how do you see this applying a few small

businesses so small businesses have uh a real problem and that's primarily that if you're under a certain size typically you have like one operations person but you don't have a security person you don't have multiple security persons but you also have a much smaller footprint i've worked at a number of those smaller orgs and the ability to interact with individual endpoints directly you've got a hundred systems well some of these solutions cater to that so grr very easy there's also awesome you can use cancer for windows and you can pull back logs to an elk stack instance that runs on virtual hardware like these are things that take time to set up and develop expertise in

but they're not out of reach they're not out of out of band the real issue is that small businesses are actually pretty frequently targeted i know organizations in the non-profit space that are 20 to 30 people well if you're involved in economic development in certain parts of the world there are nations that want to know what you know and they will do what they can to get into your environment now the risk there is smaller because if you're not losing regulated data if it doesn't financially impact you you might not feel the motivation to take action but some organizations feel like if they're compromised the people they represent are also put at risk and they take it

very seriously um there are a couple of uh groups of folks that uh you know put material out you've got sans top 20 you've got the asd top 35 that attempt to give some guidance but from a technical implementation perspective i think they really just need to own the process if you don't have at least one technical person it's going to be a real challenge

no so the problem with scripts is they're incredibly variable um even if you just look at simple vbscript or jscript you can have compression you can have string replacement you can have encoding those combinations are not great because they're used legitimately in a number of different functions so if you keyed off of just those attributes you'd want to blocking a lot of legitimate stuff additionally most scripts are being used for an operational purpose and the folks who are using those uh to support it operations they don't want to run the risk of breaking something so it's very it's very challenging some white listing solutions will allow you to say block all scripts outside of a certain

directory but not all of them offer that capability so it's definitely something to consider with vendor selection you know to focus on

function analysis