Threat Hunting: Defining the Process While Circumventing Corporate Obstacles

hey guys good afternoon ever can everyone hear me perfect thank you for coming this afternoon especially on a Friday today we're gonna be talking about threat hunting some of the obstacles we face up there we go in corporate environments and hopefully how to bypass them my name is max neck I'm a consultant security risk advisors I generally speaking focus on the endpoint controls as well as doing data analysis my name is Kip Kevin Foster I talk very loudly apparently I am also an endpoint specialist primarily focused on instant response as well as forensics and the purpose of this talk forensics is forensics at scale excuse me my name is Ryan Andrus I am generally an instant

responder and do a lot of data analysis as well as focus on the endpoint as well I also in heavily integrated with making sure the tools in the backend work as well for a lot of the threat hunting infrastructure that we work with in terms of a talk agenda we're gonna start off by defining what is threat hunting our own version of threat hunting the preparation and communication behind the steps the framework that we typically use or employed during threat hunts and in the post hunt activities in terms of how we define threat hunting we typically define it in a number of different ways or a number of different objectives but essentially we're looking to identify systems or accounts that are

compromised in the corporate network as well as look to build out or establish rules in areas of the environment that might have limited security controls or limited visibility in order to enhance or identify these types of things in the future additionally some of the strategies we look to employ as part of these activities or anomaly based or threat based Intel obviously that's Intel scans anomaly based are looking for unusual configurations or thing that might allow an attacker access to a system as well as behavioral so essentially taking the mitre framework looking for tactics or specifically the techniques across a number of systems or in scope systems additionally the why why do this essentially to identify systems that

might have pre-existing conditions or alternatively been compromised maybe before the security tools you put in place so they don't necessarily have the visibility or have seen those attacks as well as the blind spots a lot of times in some environments you have legacy systems or parts of the network where there's no visibility into something occurring so essentially trying to get visibility into those systems or those areas of the network as well as looking for secondary compromises a lot of times we see in organizations they're very good at performing the initial analysis say for example phishing attack but seeing the a lateral movement or movement away from that primary box is sometimes challenging to see where additional

systems or accounts might be compromised talking a little bit here about the high level just to walk everyone through it we typically see it in three core phases the preparation execution and post in post activities each phase generally speaking has a number of different steps incorporated within each phase with in preparation obviously we have identify the core team members what team members we typically use as part of a threat hunt as well as socializing with the key stakeholders so looking not only at your security your system admin stakeholders but also who are the business stakeholders that we need to incorporate as part of the threat hunting activities and get their buy-in again part of the

preparation steps is identifying a hypothesis and then as well as identifying the scope of systems where that data is gonna be available to prove that hypothesis correct and then as part of the execution phase let's go out and from those in scope systems gather the data in order to prove or disprove the hypothesis and then from there once we've gathered that data let's go ahead and perform some data analytics and some manual analysis in some cases with that data in order to prove that hypothesis true or false and then once we've got that data perform that analysis looking at the post activities what observations were made let's document them and then what rulesets improvement can we do an

environment those last two steps are critically important as part of the threat hunt because you need to get organization buy-in and not only to continue doing threat hunting but additionally understanding maybe what where you hunted what activities were associated with that hunts in order to atribute the actions that you did in terms of the threat hunt team selection there's a couple different things out there on the internet we've seen but generally speaking we go with three core team members a team lead developer and an incident responder team leader generally speaking works not only with internal team and provides direction but also externally and communicates with organizational stakeholders in order to get their buy-in developer looks to integrate the

tools that you have at your disposal as well as write any custom scripts necessary to perform data acquisition then the incident responder essentially the incident responders role is to look at what you're seeing within the data is it unusual in comparison to configurations in the environment is what's running should be running and at the end look to build out some anomaly detection one of the other core things we look to do is utilize some internal security team member from the organization in order to leverage their environmental knowledge that helps us get some more in visibility or understanding as to what we're seeing and quickly be able to take some action associated with it activity tracking an

organization is huge as part of these threat hunting activities and we generally speaking like to refer to this as a project because it's short in term in length but repeatable so one of the things we typically like to do is make sure that all the things we're looking to do are documented within IR toolset in order to answer the question is who did what on what systems what accounts did we touch what activities occurred and when did they occur in order to be able to attribute all the things we did to ourselves as well as answer a system admin when they say you took down my system on this date and in this area the

network and we can turn around say no we didn't which is huge additionally in terms of tracking this then starts to tell the story as to when you want to perform these in the future what did you do and be able to communicate with stakeholders and metrics and be able to track and measure success for long-term improvement of threat hunting projects what we look to do in terms of measuring success there's five core things in measuring success when we look to communicate with stakeholders so that's not only stakeholders at the business level but that's also at a system admin level one what's the hypothesis topic and what's the system scope and is that data available we need to make sure the

hypothesis topic is selected appropriately because if we select a topic and that data's not available all of a sudden we're gonna run into a number of different challenges additionally we want to define a repeatable process within the threat hunting methodology to ensure not only is the process repeatable but the results are consistent additionally we want to reduce the attack surface so one common example is if we're hunting through a network and find something unusual let's be able to then move forward and take that and incorporate it into visibility in the future so for example if you have like an old version of CC cleaner that's exploitable let's get that out of the network as well as to develop environmental

baselines so it's important to understand what you're looking at so you don't repeat the same analysis in the future and then the last one there is a dimension and rural generation so taking the results or the visibility you created in the network and ensure that in the future you can generate an active or an automated rule around that getting approval which is huge in an environment so one of the key things to do in in this case is speak the language of the individuals you're talking to and understand your audience so in this case we broke it down into two key groups executives and sis admins and typically we like to break those conversations out

into two different groups in order to target the conversation as well as the language we're looking to utilize essentially in execs we're gonna be able to do and perform something in your environment without capital cost utilizing tools in your environment already one our favorites PowerShell and we'll talk a little bit more about that additionally be able to talk to sis admins so obviously they have uptime goals and be able to let them know we're not planning on breaking or modifying your system we're gonna leverage internal tools that either SCCM or alternatively configuration management uses in the backend and as you note there at the bottom we're not looking to deploy an agent we're not looking for

change management approvals because we're not changing anything where does your leveraging existing protocols or control infrastructure in the last one there we're not gonna piss office it admins one of the next phases as well as communicating with stakeholders is what are we looking to do and essentially when we're looking to select a hypothesis we love the mitre attack framework it's essentially wonderful in terms of mapping different techniques back to the tactics of what an attacker is looking to do or Melissa take malicious actions in an environment so commonly when we're working with a client will look to walk through the mitre attack Mac matrix so technique and look to socialize that make sure the date is available then

from there look to try and gather that at scale talk about a hypothesis driven methodology so we take like to take an almost a scientific pert approach for proving or disproving the hypothesis essentially the first step is identifying what you want to test and again we go back to the mitre attack framework for that the second step is identifying a plan to prove or disprove that hypothesis and then from there being able to execute that plan you established on scoped systems in the environment and be able to acquire that data and the last step there is doing that analysis proving whether it's true or false and then being able to document that accordingly

alright so Matt just talked through kind of our high-level focus here how we communicate and get buy-in and organizations where there aren't necessarily you know clearly defined goals or there's hesitancy in beginning to poke around on the network a lot of times people express you know concerns you want to touch every system on our network you know it's the blue team perspective your red team's penetration tests are already currently doing that just kind of flipping the coin on them so as far as actual data collection goes here there's three primary groups that we look at first is endpoint which is what we're going to be focusing on for the remainder of this talk just because

this is the one that a lot of organizations have room for improvement or even collection in general or a starting collection in general not a lot of organizations are capturing endpoint logs very well currently at least ones we worked with network logs hopefully everyone in here is logging one of these items if not all of them and same thing goes for account logs there's a lot of a lot of really good data that can be mined here and you can start looking at things from a very interesting perspective so summary message is even if you are not capturing everything on this slide even if you only have one item you can start looking at it from a blue team

perspective and start trying to find bad or evil on it as far as network hunting is concerned a little bit of logs go a very very long way you can start running with some really interesting use cases such as suspicious geo locations port scanning you know various data exfiltration methods our personal favorite is DNS logs and Ryan's going to start talking about how you can start analyzing some of those if you're not currently capturing DNS logs we've had a lot of hesitancy enabling DNS debugging - excuse me start capturing those in environments I do want to give a shout-out to Bro which is in the very right hand column here it captures pretty much everything you

could possibly want under network level and depending on where it's positioned you can start actually looking for system system communications and doing lateral movement hunting one of my favorite bro cases to talk about is just capturing SSL and TLS certificate information and in general once you start grouping things together most of your systems should be using the same communications things that are interesting to look at our self-signed cert switch bro we'll just pluck out and you know explicitly tell you this is a self-signed cert when you look at a lot of the Red Team tools were pure cobalt strike by default they use self-signed certificates so if you can white whisk those out anytime you see a new one it's

probably something you want to go look at so going back to Matt's idea here about baseline in the environment really the core concept is you want to figure out what is normal and then start defining alerts on things that are abnormal when it comes to endpoint collection in a lot of organizations we end up having to use something called sim sweep if you've ever heard the name Matt Graber which if you've attended a PowerShell talk in the last five years I guarantee you have he has actually been doing some work on the blue team side of things he's often associated with red team tools he wrote some great wrapper kind of functions for a new while new as

of PowerShell version 3 Windows component called sim common information model which is the new and improved version of WMI if you've been around a Windows environment for long enough I'm sure you've heard of it it's actually existed since Windows NT 4.0 so what sim does is it pre-negotiate saw then' ocation for WMI communications which really speeds up the rate at which you can talk to systems traditionally through WMI there's a lot of back-and-forth there's a mutual authentication going on it's very chatty and it's very slow sim sweep does all that upfront in that way you can just interact directly with each system so really the benefit of sim sweep is it speeds things up considerably it is you know no longer is

slow as WMI is and it allows for scripted mass collection across an entire environment as far as some of the drawbacks go you are limited in what you can collect like I mentioned it's WMI based it needs to be in the WMI repository in order for you to collect that data so you get some flexibility you can start pulling back specific event logs if you're not centrally aggregating them you can look for specific registry keys or file paths as well as capture things like processes services scheduled tasks all the things you normally look for when you're trying to figure out is this system compromised unfortunately you can only capture metadata if you will around those

activities so you can see that a process is running on a system but if you want to go grab a hash of it it's kind of a limitation of W mine it's not in the WMI repository there's no great way to do that through SEM sweep or WMI one of the things that trips up clients here is the excuse me our PC ports so there's really two methods that you can use sim commands the first is through windows remoting which I'll touch on next slide and kind of the legacy protocol is RPC uses some obscure ports I think it's 45,000 and up so if you have internal firewalls in place that can potentially limit you from being able to use this

tool and start collecting information through sim suite fortunately that's not very common I guess unfortunately I wish we saw more internal firewalls but - speaking from our own experience we don't really that's never really been a holdup for us but one of the main drawbacks of sim suite is that it's a one-to-one data collection by default when you grab this from github and there's a link in the bottom there your collection system can only talk to one system at a time there are ways around that you can use PowerShell jobs or if you want to get really tricky there's some run space multi run space code available BOPE Rox has some interesting coat around that you can

write wrappers for it and you know do one-to-many data collection but if you describe this from github you have to do one-to-one as far as requirements to make this thing actually run there's really only two the first is the system you're collecting data with needs PowerShell version three like I said these commands commandlets weren't introduced until PowerShell version three but they are backwards compatible so the data collection system needs PowerShell three all the other systems going back to NT four do not so there's really only one system here that needs to be specific baseline requirements and the second thing is obviously you need access to the system you need local admin rights if you don't have them you

can't authenticate and you're not gonna be able to get anything back just one comment the PowerShell version three it's huge because most client environments we still work with still reporting over from windows 7 to Windows 10 and as default Windows 7 does have powershell 2.0 so you can run this in those environments that are not up to version 5 or later I mentioned you can pull back process metadata and you know services and things like that and well you don't get all the the juicy information like we're gonna talk about the next couple of slides in terms of hashes and you know specific things you can look for is IOC s you can see in

these screenshots there's still a lot of data SVC host if you ever see that running without that - K parameter you have an issue very easy alert to you know create and you can pull that information back with same sweep you can also check things for paths pads if you see SVC host executing anywhere out of system 32 you have an issue you don't need additional metadata to come to that conclusion that is something that should always be true and you need to go take a closer look at that system so when you're talking about hypothesis generation sim sweep really gives you visibility when you don't have any like I said not many people are collecting

endpoint logs and we encourage all the clients to but you know while you're trying to get to that point sim sweep is a good alternative the next step up up from there is really PowerShell remoting you can see kind of a theme we're here pS promoting offers a lot of benefits you can do one-to-many collection which really speeds up the rate that you can collect data red team loves it for kind of the same reasons you know blue team does as well one of the more interesting things here is it actually gives you direct access to the windows 32 api's which you need to be a little bit more of a developer in order to make use of

but fortunately there's a ton of community support for this one of my favorite recent kind of developments in the space is by Jared Atkinson who released get injected thread script which will actually walk through all the threads on his system map them back to the DLL that are loaded there and try and find on map code or also known as injected code it's not something you can do at sim suite but it's something that we ran a couple weeks ago across 15,000 systems in less than an hour and a half and quick we had a list of I think about 50 some systems that we needed to go take a second look at so it's incredibly

fast and if you spend some time on github you can really find some interesting scripts that people have written one of the problems here is it's not on by default and this is where you need to start taking a kind of a methodical approach and how you're communicating to stakeholders when we first do a thread hunt environment we get a lot of pushback about you know making modifications or changes to system oftentimes we'll use sim sleep as an initial starting point in order to say we were able to meet these use cases and find these interesting things if you're willing to work with us and you know PI wit Windows remoting and enable this across the environment this is the

next list of things that we can do to support that so really it's an iterative process and goes back to the communication that was talking about if you are not you know effectively communicating upwards about the value add for what it is that you're proposing you're not going to get anywhere or it's going to take you a very long time the screenshot here shows kind of a different take on the process level view like I said before on sim sweep you don't get hashes this you can clearly see maybe not at the top there there's an md5 hash there's also a sha-256 hash when it comes to looking this stuff up it's very simple to script this run it

you know once a week whatever it might be pull it back into a analytics engine like Ryan's going to talk about and then do automated lookups on virustotal once you get that process up and running you know it maintains itself anything that fires is something someone should go look at really the the most sophisticated version of this is continuous logging windows event forwarding it is something that seems like some people it's on the radar and a lot of others have never heard of it if you don't have a central way to collect in aggregate endpoint logs such as you know Splunk forward or whatever it might be windows event forwarding is free it's included in Windows and they have a ton

of great documentation around it Jessica Payne who is the author of that screenshot there has done a lot of work on continuous monitoring and intrusion detection excuse me and really we try and recommend everyone to its move in this direction hopefully through sim sweep and PowerShell remoting you've targeted specific use cases right you're not trying to capture everything here the left-hand side of that screenshot all the things too much data in no context you're better off picking and choosing we just want to look at Windows run keys we just want to look at Windows services don't log everything don't try and centralize it at least not initially if you do it's too much data you're not

going to be able to give it the attention it needs and you're probably gonna start finding yourself walking down rabbit holes there are thousands and thousands of logs to look through even on a small network when it comes to endpoint data so the basic concept is be smart about what you're trying to log have clear goals and trying to end your logging to them as far as benefits here what you've done is basically flip the script instead of having an acquisition system that's going out and contacting a bunch of remote systems to collect data those systems are now centrally you know themselves sending those logs up to a mothership it's aggregating it for you it's in one

convenient location and you can start throwing that into an analytics engine or doing whatever it is that you need to compliance requirements security cases i t what happened on the system before i went down you're gonna make a lot a lot of people happy if you start doing this fortunately one of the prereqs for this is actually windows remoting so when it comes to threat hunting when you enable the ability to interact with the system through powershell you are also turning on the same service for windows event forwarding so by enabling one capability you're setting yourself up to enable the next so we really see this as kind of an iterative cycle start small prove your

wins document everything make it clear that you're providing value and then move on to the next use case because they all feed into each other so in summation if you're like a lot of clients we work with and you're not currently looking at endpoint logs you should try some as we go it's really simple to get running the documentation on the github site is very very user friendly and it pretty much just works you can modify that and add new functions if you want to but matt has done a fantastic job of pulling back some of the most useful information you want to look at the next step from there is powershell remoting which you know is

sim sleep on steroids for kind of lack of a better term it scales very quickly and it gives you a lot more flexibility if you can write a PowerShell script for it you can basically collect it and there's a whole bunch of stuff available on github to start kind of reiterate that point and another note there is if you're sleeping on PowerShell still it's kind of time to you know adapt Microsoft is pushing it heavily when you look at some of the stuff they're doing in Asia as well as their server frameworks they actually I think they killed the project but the Nano server which was going to be a headless server pure PowerShell it's definitely the way of the future

and lastly here is Windows event forwarding this is the goal that I think everyone should try and get to and whether or not you or whether you're using Windows for it or you know another end point collection mechanism maybe you have simay door whatever it might be you should definitely be making an effort to centralize all of your logs [Applause] so moving into the data analysis and project tracking first gonna talk about a few tool sets that we use actually within our infrastructure and when we go to client sites to do thread hunting so starting off with one of the first ones is miss or malware information sharing platform what it essentially is a threat

Intel aggregator whether it's a open source or you know paid private feed misfit probably and will likely aggregate those feeds together for an IOC database that's hosted big things here missive is also integrated heavily which with the hive and cortex which I'm going to talk on a little bit as well and it allows a panda richer threat Intel so virustotal showed in domain tools it has these tools built in the background so that once you have your i/o sees in there or even if you're doing an instant response you find something and put it into MMus you can then query the pre-built API keys that are in there so you know add data enrichment to some of the IOC's or files

in there so what's the value and why do it bulk ioc queries long gone are the days with now with mis where you know you have one individual who is hosting your IOC s on a text document or Excel sheet I don't know how many client environments I walk into I was like hey where's your repository of ifcs and some guy gives me a text file and it's one person can only edit it one at a time or if it's hosted in a network share not everybody has access with mis you also get the ability to do threat actor tracking so whether you're assigning tags to different IOC s different types of ransomware in an environment or in

the wild that you kinda want to track how is this payload developing over time what is it doing to this Windows system over time how is it trying to hide itself you can track threat actors as it goes on as well as mysticism agnostic any data format that I've come across right now mis allows you to export JSON CSV sticks doesn't matter and even if you haven't seen a data format that Mis supports somebody's probably building it in Python right now and it's probably out there the last thing I want to touch on is organization segregation with mis when your company gets big enough or if your organization is large enough I've seen some people segregate you like Sox

or NOx they can both contribute IOC s where both teams can see it and if you don't want those databases kind of messed up by one person having full admin rights and changing everything you have they can go through our vac and approval processes or proposals of like okay we've cited it here's some changes that we've seen as it's developing so big win for you know hosting your IOC s now everybody can kind of concurrently look at them and also feed them out via api's through Mis database into your sim or you know other tools as well moving in the data analytics our aggregation a log aggregation platform that we like to use is ELQ so if you guys are not familiar

ELQ is kind of everywhere right now elasticsearch is gonna operate as your database log stash is your log shipper data manipulation we're gonna get into that a little bit and then Cabana as the visual aspect of what you kind of do visually within elasticsearch the reasons that we love ELQ or at least I love L come speak for myself but is you get data manipulation via visualization whether this is trying to do it with you know executives showing what you're doing reporting wise or even another analysts of you know here's a quick data table look at this as you can see in the bottom right hand corner that it's whether it's a bar chart heat map data

table there's multiple different visualizations you can create when it comes to those visualizations as well it gives you advanced filtering so white listing or black listing different file paths if you want to retain or reduce I don't want to see you say Program Files right now I only want to look in system 32 that's it you're allowed to do that getting a little ahead but if you want to have the streaming data into ELQ you can use other tools like a last alert to kind of blacklist or whitelist file passion hashes so obviously data reduction within blacklist and send an alert every time you see you know specie host outside of system 32 obviously least frequency of occurs is a data

analysis technique that we are going to leverage so process stacking a lot of times we want to see these outliers in a homogeneous environment you're gonna see you know out of 10,000 systems of 10,000 systems are running the same applications and files lets you know the do least frequency of occurrence and let's see these outliers or are these files approved at the environment or is this actually evil you know it gives you a quick win to drill in to see what's operating outside the next one rapid query results elk is one of the fastest data aggregators or log aggregators that I've actually worked with and while this seems like a very simple thing flexible query options like string queries file

paths app data 90% of malware operates out of these directories don't need privilege to run but this same query and some other tools like ESM is like a paragraph long of regex it's kind of painful so you know string queries along with a lot of other queries that you can do makes it rather simple to just hone in on different files or processes 11:00 shine or fuzzy queries so service hosts now are providing a plain height or plain sight so if it's just SV host trying to hide within the same directory or even outside of that directory you can kind of look specifically for you know these common files as well and this can be

scripted as well on the lack logstash side as well and then going to locks - plug-in filters so Kevin touched on sim suite PowerShell you know some other tools and different scripts that you can pull across different environments or if you're building in your own but unifying your data fields when you're trying to build reports or hunt through the data is crucial so for example with just even the different time times that we have used some sweep or even just PowerShell at different environments image path paths upper case and lower case elk sees them is differently as well as just file paths you know it building a parser to associate this is like executable path

going forward everything once it's logging in elasticsearch is going to be executable paths and then we can correlate stack the data as well further on this parsing out like file extensions some of the scripts do that some of them do not so you can parse out the file extensions as well as you know user fields when it comes to stacking user fields it's painful so if you apply a parser to it you know see users and then you just make replace that with a wild-card and then you can also just filter it out as well so now when I go to view this data I can stack it and see the outliers as well so data enrichment

on ingestion locks - can make rest queries on invent and jet in ingestion and Kevin mentioned this a little bit go but to tools domain stats and frequency server but mainly with domain stats so definitely with DNS logs is it an Alexis top 1 million so two things that I know we've done you can take the forensics approach and gather all data just throw all your domains against it and just query it in elasticsearch because then you can start to build visualizations on geographical locations or just you know types of names at different data visualizations whatever you prefer or the other is I only want to look at anything that's not Alexus talk 1 million so the only

adjustment that you would make to this this filter on the right there is add a tag and say these there is or you know found or not found and then drop if it is found essentially frequency server test the Englishness of a domain so essentially was this domain generated or was this an individual creating it essentially well whether pseudo-random or an individual creating as well and then finally on the slide virustotal queries or realistically any threat Intel application that allows you to kind of do rest queries in this example we actually have a proxy server where when the data is ingested with the scripts that we pull whether it's a raw CSV hash or overall CSV of just hashes

or the actual data coming in with the hosts associated with it it's going to pull out the sha-256 up there and run it against virustotal and then populate it with an elastic search with virustotal and then you know dot all the information and i'll show you a little example in a second so this is really helpful both private API keys and public API keys work just with the public you got to add a 15 second throttle to it so that you don't need up that call API key rather quickly essentially one other thing I will note though is what we use here and in this example is we have IP there we have a proxy server where every time we run a

hash it saves the data and then shoots it into our elastic search this way that we only have to run one key once and all the hashes are saved there going forward kind of effective we don't eat up our API cost per day so so don't expect you to read this at all this is a JSON from raw input this was a raw CSV of just hashes and this is one hash actually so what it did is it pulled out the sha-256 hash and all of this is the virustotal information so the file names positives was it clean was it not clean this is the Royal JSON in the table of you if you see in the top left corner

elasticsearch and cabaña will make it a little bit easier to view or you could put it into a visualization so for that shot 256 what file names did virustotal come back on and then you can filter based on rating because it's gonna sign it whether it's unknown clean I don't want to see those there's no no quick wins at least right there let's see malicious and suspicious and drilling did it further and then from there we can add another sub bucket within the visualizations of what was associated with or how many hosts have this actual shawl on it so moving forward into the high oven cortex the hive is going to be our threat hunt

process tracker so and cortex is going to obviously help us with analytics as well so starting with the hive it's a live stream of team member activity it gives us accountability going forward especially with the the they touched on as well with you know being able to track you know what were you touching in at this time and what users what hosts as well as project tasks assigned for visibility and accountability associate with these tasks you can also assign metrics or statistics around it so reporting for c-level executives or other data analysts who are being brought in from the client side so I want to keep this picture up here because I'm referenced it already but

miss cortex and the hive are heavily integrated the developers going with these projects are actually going back and forth I thought actually believe they had a conference last week or last weekend releasing a new update actually with hive this will be this week so but one of the main reason or uses that we use the hive is obviously for observation tracking it allows you to populate all those hashes that we always just talked about running against virustotal in here for you know statistics based type or statistics based on type IOC or you know tags so if I want to say this is the East client if we're doing say different domains which we have done

like West domain East domain or the East client West client you can break them up by different tags so that when you go to do reporting it's a little easier to kind of filter on these the next topic obviously there is filtering observables out of the 276 that are within this I'm filtering on only the ones that came back with an IOC hit that's of a hash type it allows you to kind of search your data essentially with elastic search or Cabana level you know aggregate are filtering if you will so moving on to that data analysis and automation so this is where cortex comes in looking at an observable this is just one picked

out essentially when you do log observables I'll take a step back for a second but obviously adding your tags was this sighted different hunts you can keep going forward was it sighted again and then description obviously it's test just for this you know scenario but you know what host what user what environment was this found on links on the right there when we do data acquisition we did it once and then we did it twice if it shows up in another case it's gonna link right to that case of how it showed up analysis on the bottom is where cortex come in comes in cortex has around 30 plus pre-built analyzers and it's gonna link your miss

instance your threat Intel or whether it's paid or private and any tools you have to make data analysis rather quick so the main goal of cortex is data reduction be a threat Intel correlation anytime that we can get a machine to do most of our work I'm all fans for it for automation but essentially cortex as I was saying has multiple analyzers it breaks it up between the different data types as well on the right you have the vendors or the different tools that it's gonna be pulling from the hive helps here because it has pre-built templates it makes when you run the observables through the hive utilizing the cortex analyzers it's going to populate it into a nice pretty

template almost like you're looking at virustotal itself makes it very easy to look at on top of this and I'm gonna go back to here is delegate and let the machines do your work so when I was running those hashes essentially at certain clients through I just dumped in 10,000 hashes bulk imported to the hive run the cortex analyzer do another 50,000 IPs or domains I want to run run it against our threat Intel so you know it's doing all the work I come back and I can filter on that data of what actually got hits you know or you know what it's gonna return so that we can dig into this deeper so in when

it gets actually to data analysis and analytics just talking about the tool sets right there at least that we were using data reduction is key we've touched on it at least twice or three times now but targeted threat hunt topics persistence and then registry Keys just registry keys alone and even then you know run keys versus log on scripts versus services and tasks when you run auto runs against one host or endpoint you know you get thousands of results or logs that come back you run this against ten thousand you're now looking at millions of logs so when it's just Auto runs let's let's focus in on logon scripts or let's focus on run keys

so setting the scope and realistic standards is going to help your threat hunt be a lot more effective going forward as well as you know reporting it up to upper management least frequency of occurrence so on the right here obviously automated semi automated manual this is kind the forensics best practice when looking at an endpoint because you're going to be looking at a wealth of data upfront we're kind of adopting this kind of this methodology in the fact of looking at data so stacking all the things as I said with elastic search already homogeneous environment every application or most applications or files are going to be consistent across say all 10,000 hosts the outliers are

going to be easy to pick out again find evil or know what is normal in your environment whether this is Linux or Windows this has to be essential and this comes down to just knowing your environment being in it long enough is service host running out of a path outside of system 32 is their single name characters dot exe kind of suspicious or pseudo random letters and numbers legit files operating and illegitimate file paths so a lot of knowing what is normal and not normal is a lot of what data analysis techniques we're going to be utilizing but an elk it obviously makes a little bit quicker because you get to do some of these

string queries like a you know top 20 processes that are out of system 32 let's just quick look at them and see if they're outside of it and obviously threat Intel correlation talked about this obviously with cortex the hive and miss you quick wins dump all your data in see if anything comes back and then move on from there so a little example of data analysis and analytics within process stacking or process dump here obviously we just dumped in a unfortunately I'll say this is not a homogeneous environment however we did segregate it for least frequency of occurrence so what process is found only across like one host or two hosts or you know one

count and while this can be kind of hard to kind of look through you kind of have to validate as well there's two things that kind of pop out to me at least in this example it's coupons users love their coupons or at least they don't realize they do and these would be browser hijackers I've seen them a lot and we need to get these remediate off the hosts scrolling down a little bit more not even half a scroll and there's two more things CC cleaner 64 which Matt actually mentioned before let's validate that this version is not the exploitable one you know going through and just making sure that hey I remember this from the news or I remember the smaller

IC database whether it's an miss for the hive we got to validate whether this can be a ability for us and then mixed race 64 dot exe or Mik tre HP audio driver for some of you that not aware of this but essentially it writes every keystroke to the computer in clear-text making it a classified as a key logger so real bad we don't want that we need to patch that and get that removed to make sure it's not doing that anymore so now that we have observations going back to the hive observation and result tracking like I said with mis where it's a hosted instance the hive is as well so we get to you know host all of our

observables so that all of our team members can view these concurrently and we don't have to wait for one person posting a master file with Excel he didn't update it yet he sent me the wrong version now it's all right there people can see the live tracking updates as well and on top of this the hive allows you to assign the tags like I said before for hash IP domain etc you get to get these pretty graphs which can be helpful to you know digest the data as well as show these reports to going forward as well so looking at accountability for team members and status updates for live streams you can see on the right that's actually me

dumping in around 10,000 hashes and probably into the next step after that was running them through some threat Intel correlation but you see every time I interact with the hive whether it's you know adding and observable adding a log to a different task that I was assigned or anything your team members are fully aware of what you're doing a little bit to the bottom there we see the tasks so you can break up your projects and obviously have the metrics around these as well so that each step of the process if somebody's like hey what's the status update here let me generate the reports of what metrics we have already so drilling into like one task that's a sign that's the

first one might be a little hard to see sorry about that but essentially you drill into a tasks and it provides a little bit more metadata what's the description of this task what's being asked and on the right-hand side here you get the add a log so whether that's you know every hour or at the end of the day for when we're on client site or at the end of the day in general I'm updating my team of hey this is what I did this is what's going on and in this example we're just distinguishing these are the hunt topics we're going to be covering over the course of the week so that everybody is up-to-date and they

know what's going on with each task so coming back measurements of success linking this all back around hypothesis topic and system scoping we got a pit we have to pick threat on topics that are actually executable like Matt was referencing so is data based on data availability establish and define repeatable threat hunt methodology with some of the tools that's we were just identifying the hive you have tasks that make this repeatable you can create these templates okay a new persistence topic or you know another TTP that we want to identify our TTT would map back to the attack of mitre attack framework and we just pop out the template and do the same tasks over and over again

identify and reduce environment attack surface any observables that are recorded whether or not they are just you know misconfigurations or actually malware you know we're gonna be reducing that and it's trackable so hey we found this this time but next time we come around why wasn't this fixed develop environmental baselines when it comes back to this again this is the process stacking in the homogeneous environment from the example I showed it can be a little hard to sift through data but you know approving what is approved business software or you know maybe implementing a system like cots square you know right off the shelf it's only approved business software within the environment and then the last but not least and one

of the most important and front maturity is automation and rule generation so going forward we're talk about Windows events generation or Windows event forwarding sigh and you know just porting all this in are the information into a sim or a data analytics platform having it streaming we got to create rules so hey we looked at registry run keys already next time that a registry run key is created in like a certain file I want to see that alert I can validate it real quick just want to see what run key was created so that going forward it's not necessarily malicious or bad behavior and that's all sorry talk a little fast I know we had a

little bit on time but there's still a lot around this topic so feel free to ask questions come up to us we love doing this stuff and love talking about as well so [Applause]

you got the mic so when you come into a customer environment are you writing this data to your ELQ infrastructure as well as the customer sim environment and is that kind of intensive on read writes yeah so good question good question when it comes down to a customer environment obviously that's going to be distinguished before the engagement even starts you know we're gonna say hey we need to stand up or we want to stand up this architecture whether or not you guys can host it for us that'd be great if not then we can do it on local systems as long as we get the approval it's all about just communicating what is expected and what we are going to be

doing and being very clear of how we're handling this data as well one of the key things I think around that too is time we'll see availability time and resources on their side we're happy to bring our own systems in and just run it ourselves in order to do that and meet their goals or timelines but if that's not something we can achieve in terms of working with them to stand up in the client infrastructure will probably just support it ourselves so also when you walk into a customer for the first time how long does it take you to figure out what normal is great question I think my opinion personal opinion is it highly

depends on the hypothesis topic you're looking to hunt for in the environment obviously Ryan mentioned one thing around run once keys if the scopa system say is user workstations only and it's a homogeneous environment that's far quicker than say an environment where they've had multiple mergers and acquisitions and a client wants to scope them all together so I typically like to say great you have maybe two or three organizations as a part of your overall company let's scope them differently why because we're gonna get much more continuous data or like data that we can stack as well as perform analysis on rather than grouping all images together at once you guys have Matt touched a little bit - but also

segregating that data based on you know am I only looking at developer systems am I only looking at server systems so when we start to segregate it that way it can be rather quick learning what's normal and what is supposed to be the normal within a client but it also heavily depends on you know do they utilize a cots do they utilize business proof software or is everybody kind of have free rein and local admin just one more additional point there I think the overarching opinion or overarching theme here is data reduction techniques how can we get as much data out of what we're analyzing as much as possible in order to identify say something unusual

all right gentlemen thank you very much then another round of applause for our speakers we've got a challenge