BSidesGVL 2021 - Nick Sweet - "Pretty Easy Threat Hunting"

all right thanks everybody uh like ryan said i'm nick sweet i work for a birdium um i live in knoxville tennessee so not too far away from you guys in south carolina um so a little bit about myself let's just say i'll lead our solutions architecture team at averidium we're a professional service managed service security company we've been in business for over about 22 years when you look at kind of a lot of our founding companies myself you know i've been in the security industry going on about 13 14 years i started out in the department of energy uh working for operation national laboratory where did a lot of security um and you know kind of government

contracts you know at the time we had three of the the world's fastest super computers so did a lot of security around that dod projects um and if you're not familiar with ornls were part of the manhattan project during world war ii um when it did a lot of the you know building the bombs and stuff but now it's a really large research laboratory so got exposed to a lot of good science research and things if you guys are into that sort of thing did a stand at the university of tennessee helping build their security program moved on to nielsen where i did a lot of you know security operations pin testing doing the red teaming and

stuff there um and now i used to and then when i moved to averitium i let our managed security practice so security operations um you know making sure that you know basically a lot of the stuff that ryan you just talked about you know operationally making sure we had the right processes procedures in place to help protect our customers on a day-to-day basis so i'm really excited to be here today um and as we go through here i don't know if people are allowed to ask questions or whatever you know feel free to jump in um i'd love for this to be interactive uh you know it gets gets kind of lonely up here without any interaction so you

guys feel free to jump in interrupt ask questions and i'll go ahead and get started so today this is a quick outline if you got to see the abstract what we're talking about today is is really just nothing new but hopefully it'll give you a few tidbits or things that you can take back to your organization to help with you know security operations threat hunting instant response i mean just general investigation within your organizations because i know everybody here probably comes from different size organizations you know not everybody works for a fortune 500 uh where you might have a hundred people on your security team plus or you know we probably have some people in here that you're the i.t

person the security person and the cso and you know you wear all the hats within the organization um so hopefully there's some things in here that are beneficial to you know that full wide range of audience because i think some of the concepts that you'll see in here can be leveraged across the board um so we'll just do a quick history intro i kind of hit some history about myself there i'll introduce some things that you know are probably not new to everybody here you've probably seen them before in a lot of other talks that you've been to and then we'll move into some you know why notebooks there's been a lot of work you know cyber war dog does a lot of

great things around threat hunting playbooks and stuff but i want to put just a little spin on that and and hopefully bring it up a level um a lot of times you know when you're in security operations you're you know the technical guy in the room you spend a lot of time you know in the command line or the sim or your technologies and sometimes you might lose the you know kind of what's the business driver here how can i relate this to you know what the company needs and can i leverage my technical expertise to you know show leadership why we need to do certain things so um we'll go into a little detail on that so

moving in um so what are the concepts that you know we like to use at a verdium and you see this a lot is transitioning from that old aspect around you know traditional sec ops and really into more of a cyber fusion approach and you know what we'll go into here a little bit is you know we have two main themes and it's really around know thyself and know thy enemy today we're really going to focus on know thy enemy you know because that's really where the threat honey comes in you need to know a little bit of yourself right because that really drives where you can do you know some specific threat hunts but

when you're doing the threat hunting right i mean that's the key word is the threat right what is the threat actor doing what are they looking for what are the types of information you know that they're looking for to be able to to protect and um i've only seen cover slider presentation yeah nick we're only seeing uh looks like the powerpoint um view it's not even in presenter mode so we just see the avertium slide it's okay uh maybe stop sharing and reshare it looks like you may be frozen i don't see a cursor moving on your screen either okay we'll try that again we'll see the cyber fusion slide now all right perfect hey thanks uh

tim there for let me know um so yeah so this is what we're going into here just you know now now that you have the visual is really you know this concept of pulling all the pieces together into this more of a cyber fusion and having the different parts of security really working together not having your governance and your risk and compliance and policy team doing their thing in one area having you know your red team do one thing your blue team do another it's really how do you come together and have this kind of fusion structure and by doing that one of the things that we like to focus on is is really around how do we assess and

measure maturity it can be you know in the first where like know myself you know you might be using a security framework like the nist csf you could be using you know the cis top 20 or i guess it's the top 18 now i'm obviously it's going to take me a long time to get used to that but you know really focusing in on like what are the security controls and things and really measuring to that going forward and this is you know pretty standard maturity model and you're going to see how we try to apply this not only from say a holistic program level standpoint but how can we apply this at a more tactical technical

level around how you're doing your thread heading how you're diving in and really relating that up into you know business decisions business risk and the things that your executives you know really care about um you know being in the industry here for 13 14 years you know it's it's one of those things to where you know if you want to be a really good security professional you know for the most part at least if you want to move up i'm not going to say let me just rephrase that but as you move up in the organization you really have to start understanding what's important in the business and a lot of times you're having to sell you're either selling up

you're selling to other people on your team you know that could be another security engineer that you're working with that could be the cis admin that could be the application owner there's a lot of things that go into you know kind of the the political landscape within the organization so hopefully some of these things that we go over will help with that so starting with some frameworks um you know i think you know if you go to any security talk nowadays especially within security operations i'm sure moderate attack you've probably heard a million times we could probably turn this into a drinking game but you know it's still a really important thing when it comes to

measuring success within your security operations center and like what should i be looking for when i'm out there spending my time you know digging through the sim digging through my logs or my endpoint or my edr or whatever that is to really get a sense of what i should be looking for and they can't be you know we can't um you know with that adding to that though what are some other things that we can really add to that from a a security perspective so jumping in from a threat hunt perspective right so trying to level set as you know if you're new to security operations you're not quite sure you know like so what is

this threat hunting that we're talking about today you know so threat hunting is you know just really the process of proactively going and looking for stuff you know it's in the key thing when you're when you're going to go look for stuff right is you need to have data to be able to sift through so you'll see here this is just a high level process diagram um if you go to the internet you know this is pretty straightforward uh you know nothing necessarily groundbreaking and it's a pretty straightforward concept right is instead of sitting around and waiting for your tools and technologies to to tell you to do something it's like why don't we take

the time to actually go into our technologies and see what we can find based on certain intelligence and stuff that we see and when we think about you know what we can look for you know at a verdi and we think of it as really as in three different buckets you have your more what i like to call your your more temporal stuff the things that are maybe of urgency maybe you've got a threat intelligence report from some uh threat vendor that you all have or maybe some you know uh c cert that went out that said hey you know this particular apt group is looking and exploiting x y and z you know you think

of your solar gates think of you know the dark side with colonial pipeline think of you know some of the more recent ones you know there's there's a lot of different you know things that you can turn on the news and say wait a second what's going on then you have the bucket more around what we kind of caught this is the catch-all right this is where you're using miter attack to where regardless you know of who the apt group is behind it and having very specific procedures you know that they're going to be using is you know if you just hire a pen tester or if you're doing red teaming yourself internally these are the steps

and techniques that you might leverage within your environment you know if y'all were on the talk you know earlier with you know ryan you know he went over you know low bins and you know living off the land techniques and and certain things there right is you know regardless of who the you know bad guy is you know it could be a good guy or a bad guy if someone's doing offensive security there's certain things that you leverage to try to get into those environments so really doing just those more generic uh kind of catch-all type of hunts and then the final one the way we look at it is really identifying iocs and this one you know i you know it could be

considered you know threat hunting you know sometimes this is more just a continuing investigation but when you do find particular iocs you know maybe you were um you know doing an investigation you saw one of your workstations or servers that that got hit with you know a piece of ransomware or piece of malware or whatever else and you find those registry keys or the ip addresses that maybe it was reaching out to and you start doing a deeper dive investigation across your entire environment for where else you know that might happen right is maybe it was a phishing email that someone clicked on and you found it on one system but did other people get that

same phishing email did they click on it so using some of those iocs you can do some of those threat hunts across your environment you know and really see you know what was the the scale of that particular attack and so once we establish those hypotheses right then we perform those hunts you know and and then you have to utilize your technologies in your environment and like i said earlier you know you know you there's a lot of great tools out there but not everybody has access to the latest greatest cdrs you know you don't have access to you know maybe a sim maybe you're just sending stuff into a syslog server and you have to comb through and have custom

scripts that you comb through those logs to try to find things you know it's not everybody has the liberties to have you know an unlimited budget to spend on security so whatever that is is right you need you need to understand what your capabilities are and then be able to go and then look within those particular environments and then once you do those hunts right refine those detections you know as as you go in and you you start out with that first hypothesis you might get tons and tons of information back you know the the classic example right is your power shell you know powershell is something that people use within your environment and if you go and just say

hey show me everywhere power shells being run in my environment you might get hundreds and thousands and and thousands of different um you know techniques that end up you know coming back or you know results that come back and you might have to refine that detection over and over again but the good thing is is once you refine those detections over time those those detections and analytics get better and then at that point right you detect those threats once you've defined and build out those detections and then you move into the actual triage remediates right this is once you find those within your environment what are your playbooks how do you get those out and how do you make sure that you know

they don't get in again by using lessons learned for the future and so taking a step further right so we had mentioned earlier it's like so where do you focus when it comes to thread hunting um you know you talked about those developing those hypotheses and if you're not familiar with this this is a pyramid of pain um i think it's a it's a bianco that that uh came up with this that you know if you're really gonna focus your time you know where should you do it at and that's where the ttps come in you know you could spend all day looking for ip addresses hash values domain names but how easy is it for a bad guy to actually go in and

change those things right is hey i have a piece of ransomware malware it drops to a box and you know how many times does that hash value you know change once it gets written and a lot of times people don't write to disk anymore right they're staying in memory so a lot of these techniques you know just don't really work i mean that's why you see a lot of av solutions you know dying is because that detection mechanism is not something that really works and then maybe you're a good blue team or you keep to do you you go take that piece of malware you detonate it and you find the ip addresses that you

know were in there you know you go put a you know a firewall block in place and next thing you know you still see traffic going out because guess what there was actually some domain names that those were those ip addresses were registered to so you continue to do your research and all right well maybe you have some type of proxy or dns filtering in place and so as you can see as you continue to do this it just really turns into a whack-a-mo type of approach so if you really focus in on the top end of this around ttps the tactics techniques and procedures regardless of what domain it's reaching out to your ip address

it's reaching out to if i'm looking for very specific procedures and techniques it really gives you a better detection mechanism within your environment so hopefully that makes sense like i said this is you know really a lot of the lead up pieces to this um so and now taking that so using the moderate tax so one of the things that we like to do at averting and helping our customers is really help them get an understanding of where they should look at right so using miter attack using that maturity matrix that we talked about earlier right around cmm is where does a customer start out you know it's like hey you know what are my

detection mechanisms you know a lot of times when you're thinking about you know you know attack the attack framework and sim monitoring a lot of that's very reactive and detection based and maybe start at the beginning right it's like of these you know hundreds and hundreds of techniques and sub techniques which ones actually make sense to in my environment so have an initial risk score so go in and actually identify which ones make sense to your environment and then do some you know kind of what are our protections there are we able to prevent against those particular attacks and those particular techniques then we get into the detection piece right how are we able to

detect that how are we able to do we have rules in the sim do we have a sim do we are even logging for that particular technique so really getting an understanding of what that detection score looks like and then finally how are you able to respond do you have the appropriate playbooks do you have the right workflows do you know who to call who to reach out to do you have the skill set internally do you need to have a partner or a dfr retainer with somebody that come in and help when that incident happens and then finally how do i map that back to a to like a threat intelligence group if i know i'm in a certain industry um

you know or a certain uh where or a certain part of the you know the world you know if you work for a very global you know a large global organization there's certain threat groups that you might you know need from just like a geopolitical standpoint right if you're working you know if you're doing business in china you might have things there or russia or europe or whatever there's certain things that you might need to think about from a threat group's perspective and based on all that information that can then give you kind of a residuals risk score of where i should be focusing and then with that right this is where you want to start your focus around you

know where you're doing your hunting because then you're going to have you know kind of leftover ttps that this is where i need to focus because i i definitely have some type of risk here uh to be able to focus on in the future so taking that step further now so that was the lead in a little bit of history uh let's say hopefully you know if you read the abstract a lot of that you know i wanted would just to be really just introductory um and hopefully now i'm gonna introduce you know some maybe newer concepts if you're not using notebooks or anything within your organization today um and i know you know uh ryan had mentioned you know

using like onenote uh you know jupiter is almost like our onenote uh to an extent it just really provides us you know the more technical deep dives that sometimes you can't do within a onenote or you know a text document or something like this it really builds some interactive capabilities for when you know you're trying to look and search for things so the first thing when we're thinking about this is you know when we're building these notebooks is i really want to kind of orientate you all into kind of like building a pipeline and having this very be a modular approach and and not only having these be something that are consumed by your analyst team

or consumed by your engineering team is if you build a modular approach to these these are things that you can use to help your team go and ask for things within your organization and those things usually being technologies tools budget more people or whatever so if you can come up with you know using some type of framework like this to where you can share this with the executive team this really helps you tell the story of what you're doing day in and day out and so why don't we make this stuff pretty you know i think you know for us is you know spending that extra little bit of time really cleaning it up not just doing the

hunt within the sim and just leaving it there and opening a ticket is can we abstract that up an additional layer and spend that extra 50 you know effort to really turn this into a work deliverable that can really help your team within your organization or your customers if you're actually a service provider so what does that look like you know so within you know i just pulled a few of these here you know within jupiter there's a lot of a lot of great things from like you know plotly and cuff links and you know you can now embed dash and do you know dashboarding and stuff within jupiter notebooks and really gives you a full indian way to represent

data within your environment and what's really good about this is the now you can start sending this up within your organization and they can actually look at this and be like all right what am i getting out of this all right you know i can see this information this can be shared you know laterally to your sis admins this could be sent to your you know application owners when you're doing those threat hunts and it really gives them a visual of what you were seeing when you were doing those threat hunts and the good thing about this right is when you're starting to think about this is now how do i relate this up you know

not just laterally within my environment how do i look you know maybe up within the organization and so being able to relate this back to the business is going to be something that's you know really important right so the first the last slide really focused on the tactical the technical right so yes that's really good that's your traditional threat hunt you know hey i was looking for a certain ttp i was sifting through some logs i saw that this particular ip address was performing some password spray attacks or some brute forcing or you know whatever that might be but what's the so what well if you know you know like i guess certain apt group that leverages

those techniques you know i have here an example from a threat report that we did earlier in the year and the reason i chose this one is you know the dark side of the colonial pipeline is you know what's really good is is trying to relate it to you know your executives i i would think you if you're especially in the southeast right i think the colonial pipeline that hit people at the pumps right where gas prices went up and it was one of those things where there was you know worry if you know hey am i going to have gas next week you know as i'm driving to work or if hey i'm trying

to take my kids to soccer practice or something so finding certain groups not necessarily to make it you know scare tactics but just to make it relatable so that the executives can understand why you're doing what you're doing as a threat hunter is really important so be able to tell that story you know look for recent attacks that were you know publicized within the media then go and find which threat groups were the ones behind that look and see what you know attack you know techniques they were using and then go look for those in your environment that way you can then relate that and tell a better story and then when you're telling that story

executives they speak in dollars right so be able to look at hey what's the average ransom demand what's business downtime what does that look like from a business risk and reputation standpoint so really being able to relate that back to the business is really key when you're building this so i'm going to show you here on the next slide just a quick format of what this is but this is just hopefully gets you thinking around all right when i'm writing my hypothesis to go do a threat hunt that i'm not just writing this one-liner i'm not just writing this one little you know detector or analytic or whatever you call your particular you know hypothesis that you know

there's a reason behind this and then being able to then relate that up to the business so going in is like how do we operationalize this you know i think you know that's well and good right i think you know we would love to be able to do that but here's just a quick little framework and you can take this ad to this this is you know i try to simplify it as much as possible you know for the audience today um and i know you know going back to earlier not every organization is going to have a full team dedicated to to be able to do this you know you know for us you know this

is what we do from a day-to-day basis you know our cyber threat intelligence team we have different people that work on different parts of this you know we have technical writers you know we have people that can actually write the analytics and and really build these polished things so i get it you know this can you know can be a lot but you know hopefully just gives you you know some concepts to think about is if there is that one that pops up sometimes and be like you know what i remember what nick said at you know besides greenville this would be a perfect opportunity to not let me just stop at this individ individualized threat hunt let me take

this put this into a report because i know my boss or my boss's boss could then leverage this to then go to the the cfo to help supplement and help me buy that new edr solution that i was wanting or help me you know get that sim i've been asking for for the past three years or whatever that might be so hopefully this just gives you some things to think about so starting at the top right and this is you know pretty straightforward uh document info uh nothing groundbreaking here right so just you know what are we talking about here you know day of the report you know who prepared it you know did we do the hunt you know this is

really thinking about a modular threat intelligence report so you can start with hey did we do a hunt what was the objective of this what was the abstract if you do tlps you know a traffic light um protocol and right you know is this is this you know sensitive to us only can this be shared you know you can put a lot of different things that go into that that document info you know one of the things we like to use a lot of times is the data being used right is there you know as you're trying to share this maybe to your you know maybe you have a work group or some discord that you work

with people on on the outside or you know just some friends of yours you want to share this report too is you know what are they looking for when they're doing that um the next section is really around the thread info right so that summary so we talked about that slide earlier around dark side is you know what are we talking about what tactics and techniques did darkseid use as part of that you know any recommendations that you can make the references you know really do a really good job of laying out what that threat actor or whatever they were doing and then the final piece is the actual hunt data so once you've you know kind

of laid this out you have your document info you have a really good narrative right explaining what you're looking for and why it's important this is where you can start getting into building the hunt pieces is this having a notebook that you can actually go in and do an actual threat hunt against you know this is for somebody that maybe you're an instant responder maybe you don't you know maybe you are a you know security admin and you don't have a sim and you just have maybe a syslog server you're just maybe you just have logs sitting on systems and you have to go run scripts to go do these grabs and do this you know

whatever that is you know there's ways of doing that and really mapping that out from what data fields do i need what technology do i need to use are there certain api hooks that i need to pull into maybe i'm in aws and i need to go out and you know do pools and maybe store information within s3 buckets and be able to do flat file analysis or or whatever that might look like and then be able to pull all that data together correlate it together and then be able to do metrics and reporting against that so what does that look in practice so here's just you know quick examples right so you know having a report data

making sure who the audience is you know that's really important you know if you're doing a threat intelligence report and it's you know really focused around you know just maybe be informative to the executives maybe you don't have the threat hunt step at the bottom you can still write these threat reports and have this same type of structure and kind of give that objective you know what you're looking for what the data is required and then here's you know what we were talking about earlier right so here's that information around the example here's you know a quick screenshot here's some references and executives love this stuff they talk you know security used to be one of those things

where most the time you know it was it was you know one of these things that it was a cost center but more and more organizations are starting to see where security is helping you know it's almost like your insurance policy it's keeping them out of the the news it's helping the business up and running and they use this as a way to talk to their you know fellow friends like they're ceo ceos all the time that when we share these documents they're sharing it with their fellow ceos and we get calls you know wanting this same type of stuff so you know a lot of cool things that you can do here so going a little deeper right so just

giving you some kind of you know what does that document look like you know what were the important pieces what are the recommendations what were those minor attack mappings back to that and really giving you a sense of you know what was the attack actor doing so now examples of you know what does that threat hunt look like you know so you know we mentioned earlier around you know you know threat hunting um you know it's like within jupiter there's a lot of capabilities here you know you can do a lot of this in onenote you can do a lot of this in word or you know insert your favorite text editor but within jupiter the reason we like

that is when we're writing these narratives and writing these within a notebook is we're not just setting there you can actually go and take action on that so you can write embedded you know python commands and searches and stuff here where you can do ml ai models you can go and search and do parsing of logs you know using data frames to be able to go out and do the searching and stuff this is definitely not meant to be a deep dive in the jupiter notebooks and and you know data science and all of that but there's a lot of cool things that you can do within these to be able to do those thread hunts and this i

think is even more important you know for people that maybe don't have a sim or maybe you're an instant responders these make really good instant response playbooks to where maybe you have different you know playbooks based on if i'm doing an investigation on a windows machine or if i'm doing an investigation on a linux box or maybe i'm looking at a you know an application you know maybe one of your applications are beginning to hit really hard and you're trying to figure out you know who's the you know ip address are they what kind of attacks have they been doing so you can build a lot of custom parsers a lot of different things here to do those detections

and here's an example right of the you know here's a some apache logs right so being able to go in and doing that deep dive around hey parsing that you know trying to find those user agent strings or who the referrer was or who are those remote hosts and you can really build those graphs out to be able to do that threat hunt directly from this notebook so just like you would type in a word document or just like you would type in any other text editor you're writing these but then you're also within the same document writing the raw code to be able to go and do those threat hunts for this particular information

and so to wrap it up uh because i want to leave some time here at the end for some questions so this is where you can really work on you know you know building that pipeline so you saw that kind of modular approach and like i said you know not every organization is going to have this um but you know sometimes you don't have to do this all the time right i mean if you're that one guy that does security use this as just a tool maybe every once in a while to be able to tell a story that you're trying to tell so what does that pipeline look right look like you know starting at the bottom

right if you have an intelligence team you know they identify that threat maybe you see it on twitter maybe you see it in some email that you're subscribed to from uh you know just uh you know some type of threat intelligence source is then you can create that threat report start at the top level you know think of the business risk tell that story of what's important there then you can start building that technical hunt using notebooks if you want to you you can still use your normal tools this is just trying to introduce you know kind of maybe a new concept you know go and then start looking at your sim looking at your edr looking at your

firewalls looking at your ids ips whatever tooling that you might have there and really building those pieces out and then be able to take that report bundle all that up together and then once you do that hunt put the evidence of what you found there right show put those in tables you know put that in an appendix you know maybe the executives don't care but as it moves up to the organization you know they can scroll or go as deep as they want to right you know and be able to do that and ultimately just be able to show value within the organization so let your threat intelligence team let your threat hunters now be

somebody that the organization sees as an important piece of the organization not some hackers that are sitting in the basement what was the old quote the 600 pound hacker right that lives in their mom basement we don't we all know we're not that right so if you're from a threat hunter you like doing opera you know like doing the blue team stuff is try to be you know show your value within the organization right um and and make you raise your hand and be able to to be able to do those things so i think i'm coming up at least about like five minutes here so um i appreciate it uh appreciate the time today hope you all learned some new

things here um and i will open this up to questions