BSides Rochester 2019 - Threat Hunting and Other Arcane Magic

up alright I still want to yell and have it all be loud alright alright so thanks for coming didn't talk about threat hunting everything that's going to kind of build up to a very efficient for hunting program whether it's you know one or a group and I hope everyone has fun alright so right first you know who am i why am I allowed to talk to you about this and actually be right I mean I'm just a security guy but doing this for you know just a little bit I have actually been a security engineer for about the past two and a half maybe three years and before that I was an infrastructure so I was you know systems

you know building out out and you know everything before that is history that I don't care I speak above so alright so you know what is threat hunting a lot of people have their ideas you know some are right some are wrong I I feel that really nobody is wrong when they're talking about threat hunting as long as you're really like proactive in searching for threats in your environment I would say that that is the essence of threat hunting so as long as you're doing that then you're doing it but to really be effective in doing that you're gonna need a couple of things but we're gonna split up threat hunting into two kind of types so you have your your

theory or your Intel base so these this is short term this is things you you know read on the internet have artifacts that you can actually look for in your environment and then the other type is more of a long term so it's project base you you come up with this plan and you know it can be described as the digital forensics base as well where you're actually looking for the things the artifacts on your network that the tools aren't bringing to you so another thing is you got to be careful if you're gonna start thread hunting you have to be ready if you find something on your network you do have to know how to

respond to that and you have to have a process in place when you do actually find something or if so we'll go into the types of threat hunting so your your Intel based your short-term where you actually have artifacts that are known that maybe some of your tools don't automatically alert on you know whether it's hashes or network traffic or other Intel based things so I mean it's the hunt is usually about a day it could be a couple of days but it's really not further than that more than that you're probably gonna have to have a plan your project your forensics based is gonna be long term you're talking probably a week maybe more something's gonna take up a

lot of your time your your team and to be effective at it you you do have to have the people on your team or you do have to have this knowledge of the tactics that are actually used by adversaries in your network so that you can actually for that scenario find that activity so you're not looking for no neo sees you're trying to make AI OCS off of the activity on your network that the greater community doesn't know about yet so this is gonna be planned as documented and you know for some environments maybe you need approval to spend that much time on something like this because it's gonna take you your focus away from other activities that

you might maybe normally need to do alright so how do you find him right it's a good question how do you start throat honey you can't just you know search for the the threats and have them show up in a tool you have to actually really know what you're looking for and so what will kind of go into that right you know just you can't just jump into it so you have to really ask yourself where is this data where are you gonna get the information from on the scenarios or the events that are happening your network that you wouldn't know about from the tools that you already have and then that data you have

to understand where that data is coming from because you're gonna look at a set of data and you're not gonna know where it's coming from you're not gonna know where to respond to and you should have a central point otherwise you're gonna waste a lot of time jumping from different you know tools the tools or consoles and to try and get all of this despaired data you really do want to bring that back to you so that you only have one console to sit in it's been all your time rather than spend time blogging into 50 different consoles right so if you don't really have everything that you need you have to spend some time you

have to enhance you have to add these log sources grab this data bring it to you centrally so a couple of questions you have to ask yourself do you have the right tools in your network do you have the right data do you have the right people right if if you have you're gonna build out a team where maybe it's just you and you want more people I do you have the right people to do this hunting and then sure maybe you have the right people but do they have the right training do they know how to use the skills that they already have all right so you go to the tools big part of those

tools you buy these tools whether they're detection tools or even if their mitigation tools if they're alerting for you or they're already mitigating some of the threats it's going to reduce the data set that you have to look for you'll be more effective and those tools should be able to log things so if they're not gonna alert on something they should record it and they should bring it to you so you've got to collect those events that data and then it has to be readable right you can have these events and it's just a big garbage jump of data and you have to sift through it to actually find the good bit of information that's gonna tell you about

the event otherwise you're just gonna get fate to eat and it's not gonna care and also next step for the data when you have the tool that are collecting that data like I said they have to be descriptive the events have to be informative you don't want application logs filling up your sim or whatever you are if you're bringing those blocks too and it's not gonna tell you about the security event you're gonna need to know those events have to answer the questions who what where and when if your events don't answer that question it's not gonna be helpful for you and you won't be effective your team will not be effective your hunt is not going

to produce it right you see who what where when why the why comes from analysts my people don't care about the why just want to know what actually happened then respond near people you're gonna have a if you're gonna have a hunt team you're gonna have four these threats they've got to have this investigatory mindset to actually think about how someone's going to act in your network maliciously and then find that activity so you would look at something and say well how would I go about laterally moving or any other example you would think that an adversary would actually take inside of your environment another thing would be they could think better like an attacker but as far as threat

hunt teams go that's helpful not necessary so you don't have to waste your time trying to find those people that can also be developed in the right people when you start getting mature and you want to build out your your red team maybe they come from your hunt team and they get to the offensive point then you want the skill and then yeah you have those people they have that skill set they have that mindset they can think that way but do they know how to you need to make sure that these people that you have this talent that you have is trained in the way to actually think that way and apply it to a search to a

hunt that they're going to be doing in your environment or that you will be doing in your environment right maybe you have the skill set to maybe you need the training and then it's gonna be a detriment that they won't understand the data you have to make sure that you know in the training your people know about your data how its structured and where it's coming from and they will be very effective in the hunt you will be effective so you hear about all this stuff Union people you've got you know tools you got data all right you can't just go out and you can't just get all the things you can buy all the blinky

boxes you fill up your data center your network closet what-have-you you can log everything you can hire all sorts of people fill your chairs of warm bodies but it's not going to matter if it's not refined and the people don't know how to use their skills so you kind of take a aspect of will say incremental progression kind of a term came up with I was game of Phi and I wasn't good because that's not really how it's supposed to be done so I like to I had to saw that you think of it like a video game where are you kind of going through a tutorial really a tutorial for anything and it's going to

give you an item an ability just something for you to use they don't give you them all the same time because then you're not going to really know what to do with all the things so they give you the one thing then they kind of teach you how to use it you have to learn it you got to know what you got to love it you use it for awhile figure out how it works for you you know when you click the buttons how does it react I think you could move on to the next thing all right you got a to net item and you master the item but you don't have to master an item before you moved on the

next one as long as you know how to use what you have right now for the most part you can start gathering some more and really build up what you've got and then you achieve the next one you could just right you can buy everything you can just you know just install it and you just have it all but then you know your people are just not gonna know what to do themselves they're gonna be a dog flying a plane and you look at all these dials and I don't know what it says just be character is them they spend so much time trying to figure out I'm the job not being effective or maybe you work

for a company just bought all the things and sat you down in front of a screen and they say go and just to where so about knowing what you have so here are a couple of examples right on your end point that you have in your environment do you have the antivirus like about logging these this data I do you have the antivirus are you logging these PowerShell logs or process logs you know your event logs are gonna help you to get an idea of the users time they spend during the day you got to make sure you know when you go into an event log forwarding or you collect these event logs from these endpoints you don't want

to collect all the logs it's gonna be it's gonna just fill up your disk space and you know it's gonna confuse everybody you make sure they're pertinent you know even in this charity log you there's only certain event IDs that you really want to bring to you - they're gonna be helpful you can always maybe go out get them later if you need to act in some forensic way your IDs and your IPS sees a lot of stuff it's a lot of lurtz that happen there but they should also record the data maybe some of that traffic they see come through that might be suspicious and not really worth an alert maybe you actually want to bring

that to you to central logging location where someone can actually look through the data and say hey this is suspicious or this is odd or maybe didn't even fight suspicious maybe this traffic came through and and it's actually malicious when you know that just the Intel wasn't there you know your Active Directory it kind of goes back to the endpoint logs Active Directory has just a wealth of information right if you are using Active Directory some sort of directory service you're gonna get your authentication there people are gonna off they in a case against that and it will tell you the source of that authentication you'll be able to get that picture of where people are in your

environment your firewalls there's a lot of debate on how much you should log on a firewall they see a lot of traffic especially if you have a larger organization and they sit at your edge but that information can be extremely valuable if you actually want to know where people are talking on the on the network where people are going where they're trying to transfer data this also helps if you got your edge firewalls you know who's trying to excellent rate the data where are they trying to exfiltrate exfiltrated to and over what service you know you might want to bring that back to you but again depends on how much you have if you're doing try

and log all the things from a firewall you're gonna fill up your dis space you remember the spacing you lose everything else is valuable just a thought et our endpoint detection response it's not for everybody a lot of people don't even really know what it is yet but it is very valuable it's not something you want to log though but it is actually helpful in trying to get a picture of what's going on your buyer mint it sits on your endpoints it records pretty much every bit of data and it's extremely useful it gives you that picture so that might be something where you do have to make that extra jump and get that data

all right so this is gonna be something kind of kind of helpful try not to laugh too hard uh uh something near and dear to me so your secured toilet I swear there's an actual like diagram that goes along with this but it's a good way to think about it and it's a posted show how you should operate in order to quickly mature your organization and your security organization right so you have these like three tenets right the intelligence you have to detect you've got the response so the idea is so just because it's in a toilet doesn't make a pad we're trying to get rid of all the bad stuff and stick it down the hole so you

have you have these three tenants and you kind of work your way from the outside you cut you swirl around a few times as you get closer and closer and closer you get to that sweet spot in the middle where everything happens so right you kind of you get to start out like maybe you start out intelligence or detect right I had this discussion with some you have your security operations center just the people who detect things people watching the alerts coming in right you have your intelligence you have this inventory intelligence we're going to take this from the top in then you kind of have this response side and you really should start with you know

your policies and your standards right you got to set that like how are you going to operate you if they actually explain that so you know what is real and what is not or what is anomalous and what is not and right all of these are going to feed into each other right they're gonna make this swirl they're gonna go around they're gonna feed into each other intelligence detect a response to intelligence again and keep going around and you're gonna keep moving down so you got you know you can get information from your users you know what's going on there again you have that intelligence from your user based inside of your organization you're gonna get into

vulnerability management on the detect side know what's vulnerable know what you have and how open it is to attack you know come around you get your incident response team so you have a detecting now you can respond to it you've got all this data you can actually respond to that intelligently because you're actually recording that data and you have it on your teams and then as you go around you got this configuration telogen maybe you create a baseline for your environment now you can tell what the nominal is there it's gonna be extremely helpful in a hunt that's all about anomalies because your tools aren't telling you that it is there and then you're gonna go around you have

your pass configuration management you got threat intelligence you're gonna bring that in right you have to have a certain amount of maturity when you get to Vaughn to these points all right you're detect team's gonna you maybe can start building out a red team an offensive team inside your environment you can constantly test you on the inside you know and then feed that back to your responders how are they gonna respond to something that you found in your environment your intelligence team I found this in our environment what are you gonna do about it and then our detect teams like this is their watch it and that's the idea so this kind of jokingly displays the

idea of a fusion center I don't know has anyone actually heard of a fusion center ok so that's actually really good when you when you kind of look a fusion center up I've seen a lot of people reference a physical geographic location where you take a bunch of disparate teams and you bring them into the same room and they get to talk each other and scream at each other across the desks but I don't really view it that way it doesn't have to be Geographic you just have to have open communication between your teams because maybe you have a large organization maybe you have a building here in a building in another zip code maybe they're in another state

maybe they're across the country maybe they're in a different country as long as they have that open communication they might as well just be sitting next to each other it's the same it's all about open communication they got to be respond quickly so your intelligence team has to talk to your detect team has to talk to your response team and there's bastia has talked to the other teams right and as long as they're talking together they're going to quickly mature in all of their areas maybe it's not just security teams maybe you have infrastructure teams you got your engineering teams they hey we found this engineering team go fix it so everything should be feeding each

other this is gonna get you to that maturity point where you can actually start threat hunting where you can build that baseline you can actually say when something is anomalous that is something you need to focus on so where do you go to hunt all right let's next question well you go where the logs are alright so ineffective threat um team's got to go where the logs are and you don't want to have to go many places for it you're gonna waste a lot of time like I said earlier you shouldn't really have to leave they should be all coming to you you know they're gonna have those situations where you do need to jump out

of that you know single pane of glass and then go to this other tool or console okay so maybe you are recording it you've got all these logs or everywhere what do you do with them to bring them to you make all the things do the work for you right there's a couple ways to go about this depending how much money you have how much the company is the organization maybe you can only get log aggregation right you're bringing the logs in to a central point it matters the logs stacks them up and you can kind of search them through there but it's it's not great especially for a security organization so once you've got

all your security events and your logs and all that you want you bring them in maybe you go for a sim alright security information and event management um so you get that aggregation it all goes there it all gets aggregated but you also get that correlation that's that's a big thing you get that threat intelligence right you get that intelligence that it brings down automatically whether they're free your paid services you get the intelligence from and it applies it against your event all the correlation it says this event happened here here and here and I know that's a bad thing through this intelligence and I'm gonna alert you because of that and then you get

alerts out of that with all of this applied knowledge and that's great it doesn't make you inefficient of you doesn't make you not need a threat hunt team but it reduces the data set you would have to look at to be effective or maybe you have an alert and it seems benign but it's actually malicious activity because it alerts over events over here you found in the logs okay so you got all your vlogs you got your tools get your people it's done right it's not done you know for your hunt team to be effective and I have a plan what's your plan so when we're talking about the long-term threat hunts that you'll be

doing you're gonna need a plan you're actually gonna have to scope this out you're have to follow this when you're doing your hunt especially if it's something you're gonna need sign-off from your management to do you're gonna spend your time on it's gonna take a lot of time so what's gonna be your focal point there's all these parts of your environment you can focus on I like to focus on the endpoint itself that's where the user is right that's their the closest person to the attack they're probably the ones that are going to get the email they're gonna click on that's gonna infect your environment or a phone call so yeah I want to keep these other

last two bolt points is uh there's notes but you can all see them it doesn't matter right so I like to think of it as their every external threat to your organization if they are successful will become an internal threat they're gonna assume it uses identity now it's gonna be on you to find out what users bad right the idea that an attacker has to win a single time so you have to be right as a defender every single time and something comes against you until that one time where you missed it and it came inside however it flips now the jobs easier you have to win once to find that attacker and kick them out they

have to cover their tracks constantly they're always looking over their shoulder guaranteed they'll make a mistake hopefully you get them fully out and not just out of the one system that they're in and then they just pop up on the other box and then you've got to do this all over again like a little hamster all right so I've been talking for a little bit tell you all this stuff that you're gonna need right what does that mean to people who are actually gonna be doing this work teach me so we'll kind of go over kind of as as the hunter what you'll kind of need to be looking for in a network not gonna go

too deep but I'll kind of point you in some of the right direction right so first step is really create this threat model how do you think you're going to be attacked because you have to have scenarios that you're gonna be looking for in your environment you have to know what those scenarios are so one of the good ways to do it I mean you could just come up with it on your own how would you attack your organization it's always a good question to ask where are the holes but another thing to do is actually use an example so us sarah has a bunch of alerts so we'll use you know ta one eight zero seven for a I'm sure

everyone knows exactly what that is so this was a event were the they labeled is Russian government cyber activity and they were targeting critical infrastructure so mostly like energy kind of companies and we'll kind of go through some of that but they were actually going for energy infrastructure and this these events actually happened they got into a company you can read about it on us cert and probably other places and a good way to keep track of these events is to model them off of a structured framework so I to kind of the bigger ones they're gonna be the the Lockheed Martin kill chain or mitre tack framework they're both good they kind of allow you to kind of focus on a single

point of the attack and can pick that apart and find the events that way instead of looking at as a whole and trying to feel where do I go all right so a couple of key tactics we're so the utilize spear spear fishing so heavily focused very small groups usually hard to tell the email if it's actually malicious or not because they've actually crafted it against the users or sending it to they did use compromised Crenshaw's that was based off of that spear phishing attack and you know they did a bunch of stuff to maintain access you know connected to command and controls is another thing enumerate Network so they had to find what was out there when they

were on your network cuz they don't know and you know they did end up trying you know cover the tracks to you know leave a clean clean environment all right so the spearfishing campaign so right we just talked about its highly targeted you they actually pick hand-picked people and they try and figure out what they do your organization they're gonna tell her these emails against them in this case it was actually it was around the end of the year so they actually sent everyone invites to the New Year's Eve party for the company you know a lot of people thought that was really fun so they clicked it and they opened the attachment it was a Word document it was

attached in the word document it was actually supposed to pull the dot em so it's macaron navel were macro enabled template for the document and what it did instead of being something local they actually referenced an external SMB share that your machine would automatically try and go to this SMB share and authenticate against it with the local credentials so the users credentials so it was actually hash of the users crotchal and they would just they would take it on their machine and then they would crack that password and they'd have a plaintext password some of them were easy and they would log in to the external systems right so that's automatically they've got your password

you know if they got one password they only got a couple of hashes they can still pass national your fireman's bad gotta be careful of that but how are we gonna kind of detect this stuff when it's coming in so the the first line there right so look for kindly in the body of an email now it's kind of a joke I put up here one of the people I work with actually came up with that idea and I said oh that's garbage that's not that's not real and so I actually did it I searched and I would say about 90% of my results that came back we're malicious there's only a couple people who actually use the word

kindly it's so it's actually pretty efficient right so you can manually search all the all the emails it if you do are you aren't able to search through emails in your organization you can kind of get an idea of what's a normal male flow for people and what isn't even in a larger organization you kind of skim through or do you say oh you know you get things actually say invoice because a lot of malicious documents like should be invoice for you and then maybe you can tell how that's worded to actually say this is anomalous it looks weird maybe it is a real it's legitimate but at least gives you something to focus on now you got to be careful

because some companies don't like you going through email even it's just looking at the subjects right that's good you can get a lot of trouble for that you got to be careful to know your policy but another another chance is to kind of get you know after the original delivery these documents came in and they started to authenticate SMB externally that's very weird you should be able to detect that so you have these these edge logs these you know network logs that say you have SMB come out of your network and you should probably look into that where do they come from you might want to change your password

all right so the compromised credentials were used at the edge for single-factor logons now this is gonna be dangerous for anybody you should have if you're gonna externally that should be at least two factor right you should have all their protections in place that are not gonna just let somebody username/password in now to kind of detect that if you do happen to allow single factor log on externally you know whatever it is maybe it's OWA mates web mail for people who are outside the environment or some sort of portal if you can take those authentication sources that people are thin eekcast this is gonna be your ad authentication right you see there they're authenticating to you from outside of

the company but they also authenticate it inside that day there's something weird going on sure maybe they were on launch but I can take the chance that's part of that if you're gonna be on your hunt that's something you're gonna want to look into you're gonna spend your time on now you be a and user behavior analytics is gonna help out with this it'll kind of give you a maybe it doesn't alert for you but it will give you that weird activity for a user that hey they you know they would have had to travel at light speed to log in from these two locations right non-low slogans even even internally may be there if you have your network broken up

into zones maybe they're logging in from a Citrix environment or some sort of shared desktop environment and then they're also logging in from you know the server sides you know in your data center it's weird too right you've got you got all these logins from the disparate places like they should probably only be working in one location a laptop or share desktop environment maybe you want to look into that user so they a new brand of the environment once they got in they actually worked away inside the internal network and they started moving around they had they they were able to dump the admin credentials they got around they added admin users for their persistence but they were able

to use PS exact now you wouldn't just be able to search the PS exactly XE cuz they renamed it but everything around it still the same maybe you see that PS exec surface startup on a machine maybe you're recording that and you're gonna see those events where PS exec comes in and actually runs a command on a machine maybe that's normal for your fire maybe it's not maybe you know Susie Q is not supposed to be using PS exact that'd be weird right they numerated all your dns that'll be a little more difficult but if you lock down your dns you wouldn't really have the problem in the first place but that is beyond hunt so they numerated all the

users they got a feel for the actual naming standards you know all the machines they actually took screenshots so if you were actually recording processes you would actually be able to see that happen so if you're actually recording the processes that are running and the command line that's an extra setting if you actually recorded that data you would be able to see these things they use lots of batch scripts they use you know they did use some PowerShell if you were logging those if you were logging PowerShell you were logging all the command you know all the process creates with command line attached you'd be able to search for this or maybe you had some other tools

that are recording that data maybe have something on the endpoint maybe it's a antivirus or EDR that's actually reporting that data you can search through and then out of those batch scripts you're gonna see these commands to add registry entries they were enabling RDP all over the place they were added they were installing VNC they were they were changing the registry to store the credentials plaintext you should be able to but it's incredibly noisy you should be able to see that it'll be something getting logs so that goes into the maintained access another part here is their scripts to add administrators on the machines it tried to do it in several different languages so you know administrators group in

different languages is actually spelled differently for several different languages so you should see that command fail several times until they hit the language you were using on the machine so you say Oh someone tried to add someone to the local administrator group four times and failed that's going to be anomalous that's definitely something you want to look into so they disable the firewall it should be rather noisy there's events for that they enabled RDP they made some registry entries to do that as well and they installed web shells on the external facing web servers and even the exchange servers and they added VNC so yeah so to check that right you're long as you're going to show if you're

doing those security logs they're gonna see that someone tried to make changes to a privileged group the administrators group and the other failed or succeeded and that should definitely be something on your radar right disable the firewall maybe you also have something that actually baselines your configuration on your endpoints that's gonna be a tip right what's your drift on your configuration with your baseline that's gonna be something you want to hammer on though they enabled RDP right scan configuration drift right if you're actually doing that in the registry you can actually check that with a tool and you know your process logs that's gonna show that change as well now the web shells it's a little a little more

tricky because you'll have to actually read through you know some of the logs right they installed some web shells on the web server so if you actually are capturing those web logs then you should be able to see those connections to those shells that was like those files on your web server and that would be odd so you can actually probably use entropy on your IAS logs to say you know we have 10 million connections to you know index.html but then you also had 5 connections to web shell JSP or whatever the name would be that's gonna be weird look into that right so those web connections and then you probably see that traffic also maybe it's on a web

servers see that on the network blogs as well VNC if you don't actually use the answer your network it is popular but if you don't right if you prefer another you know utility you see those ports run through on your network that'd be anomalous maybe pick down that traffic right they went through and covered their tracks now they uninstalled all the applications they installed so they would actually add you know Python and BMC if it didn't exist and bunch of other things so I mean if maybe you don't log that maybe there's no way to tell maybe it isn't your base configuration well you could be able to see that through the drift they deleted

the logs like even though they deleted logs that leaves back the log says you deleted it and then they cleaned up the users that they created so Lisa the application logs you probably aren't centralizing your application logs but maybe you think that's valuable you know when you see an application installer installed or maybe it's in your baseline configuration that you're watching it's adrift they're deleting the logs right you're gonna get that our security logs that's gonna be the law that gets to leave is security so you don't see any of that if you're actually constantly logging that centrally it doesn't matter if they deleted them you recorded them they got sent so they can delete them

all day it won't matter but you should also see that when someone clears that log there's no one should be clearing that log and then you know they cleaned up users it's kind of interesting they actually created a user - as the administrator user but they created a second user that was the cleanup user so you've got the you got this like backup user and then you've got this you know clean up user and you know that would be a little odd you see it start performing all these actions on these machines that should be a clue however if they clear the security logs you're not recording them centrally or they're not coming to somewhere that you're gonna actively be

then you know you won't see those logs but you'll see the clear log alright so the command and control right so they didn't have communications over the command and control in this case right they didn't go over and like a four for three they can go over a cryptid HTTP board or anything like that to make that communication happen but they did have encrypted communication over port 80 so they didn't HTTP and the way that they hit the log the command and control server was a get request with a specific user agent string now if you're actually capturing web traffic and you actually have the headers recorded you can use entropy to say well the standard what

have you chrome law or chrome request or an internet explorer request where the header is very specific you take all those out and now you're left with all the things that we're on maybe they're PowerShell get request maybe wood whatever but I mean at least that'll pick out something anomalous in your environment and you'll be able to focus on that threat hunting is all about the anomalies because your tools didn't detect and you have to go out and find that all right so actually I went through this lot faster than I imagined so I that is the end of the what I had so if anyone does have any questions I don't know if they're

gonna come up and ask me or we just want to open forum this then yeah anyone who wants to go anybody have any questions yes

so there's actually a lot of ways you can take event logs from a Windows endpoint right if you have event logs and to actually record those and send them along so the way that I've decided to do it was use Windows Event log forwarding which is actually pretty pretty easy to set up once you get around the whole idea of how it works so those actually would just get sent on maybe like five-minute intervals and then those get quarterly you just send them to another server that does have to be central but I use them to like a collection service it might be protected and give those filled up if you want to script that you can absolutely get the

logs out and you can either save them to a flat file somewhere or send them along so you have a lot of choices but right if you're just gonna leave it to the system if someone goes down there and clears the log then they're lost right you want to actually pull them out in some way and record that

so it's gonna be all the surrounding data so you mean like as the hunter how do they decide whether it's malicious or not so exactly yep yeah so like I said the people have to fake a certain way but it's also going to have to try and know your environment so you'll be able to pick out an anomaly so something that's that's strange right so maybe let's say it's a covert channel and it goes over a port that you use in your vironment all the time but maybe this connection from this machine or set of machines is actually transferring ten times the data that's normally transferred in those connections so that's an anomaly so then that's gives

you something to focus on and pivot to and then you can you can drill down it's going to be a pretty hard process to actually say it's malicious or not but you're gonna take other events from another location so right you get that anomalous traffic you go to the source and you can see at that source what's odd and fall that out whoo and follow that out and then and then kind of track it that way and maybe you end up a dead end maybe she's a rabbit hole but that's going to be your job to kind of pick that out

yes so management behind is always magic right so that they're going to be adverse to change and spending money so you have to there's a couple ways depending on how your management structure works you show that efficacy that you're at the point where it's valuable so you say we've got these great people we've got this great tool we can do all these things automatically and we have this time that we can spend and help us look for malicious things in our environment so the whole point is I actually kind of forgot to mention it the whole point of threat hunting is to reduce the dwell time in your environment so I don't know everyone knows as well time is but it's a time

between the point at which an attacker gets into your environment to the time you actually know they're there they're doing something malicious and that time is about 99 days the whole idea is to shorten that down as much as you can so you can get them before they either take your data or destroy your environment so it's you got to kind of feel it out with your management right so someone at the tops not going to say we need threat hunting you have to kind of make your case and it's gonna be big and you have to say that you're ready for that

yes yep and you have the right people to find things that the tools because they'll never be 100 percent right you never have a hundred percent coverage in a tool that to bridge the gap from the percentage you have covered by the tool to the rest of it your people can fill and we have the time and the resources to do so and it's going to reduce your risk water

yeah so you can modify or you can run in run applications a certain way where they log separately log differently you can actually bypass some of that logging and right that's where it becomes tough right now you're it's a very advanced case right now you have attack was actually doing that and that should still show as anomalous unless they're very advanced where they're actually seeing what's normal for your environment they've actually spent the time and they're actually injecting those logs in right but it should seem extra maybe even duplicate of if they're if they're using the same write obfuscation or or what-have-you yeah

like a generics hacker not like

yeah Yeah right so you're gonna always be at a disadvantage and at a really advanced hacker who's actually done their homework it's gonna be extremely hard to catch you might never catch them but like if you have the right people and you're collecting the right data sure they're mate they're modifying the logs but you have all this other data too right because they're gonna most modifying event logs they're not gonna be buying Network logs I mean maybe that is possible but even like then then you know you've lost the two points there right so I mean you're never gonna get to the hundred percent like I was saying but at least you have something it's better than not doing it

yeah yes

yeah so you can you can do whatever you want the trouble is when you're when you do more of analog acrid a log aggregation so you've got whatever whatever law collection you want to do or log transfer right as long as you're bringing it to a place where you can search it that's fine that's good as you prepare all their systems they're not gonna apply intelligence against it right unless you have a service right when they say data Lake you know you can have a service where all your logs go to and then they'll apply their intelligence to it I don't know if I'm actually answering question but if you actually have a system that's made for

that that correlation is the big part where it's gonna do a lot of work for you but if you're you know elasticsearch or what have you maybe got your elk stack where they call it now where it's just grabbing the logs for you and like maybe it graphs it out because right so while the graphing is not correlation it gives you trend so if you actually have lots of data that you can trend and they can create a like a trend line against then that actually helps you find an anomaly as well but it's a lot more manual and it's really just searching yep yes

yeah so I picked conversations with people about when to approach users some people are very concerned about you know insider threat or maybe their concern of like like you've just jump the gun and now they're like why are you investigating me I didn't do anything I think the best rule of thumb is to gather enough data where you're sure enough where you can actually tell your management you justify why you approach to user so it also depends on the size your company like maybe maybe it's smaller or maybe your tight net and you actually know the person from across the way and you say what are you doing so you kind of feel it out right before you

actually start approaching user what I would do first is gather all my data around the event in question and get my own picture of what the user did and then after I've made my determination ask the user as long as it was appropriate for the investigation and say what were you doing here and just leave them leave it open-ended for them and then they get their question answers my suspicion then then I'm all set and if it's malicious then it's a talk with but if you think it's malicious right you don't want to engage the user immediately you want to work it up your chain and you say like I think this is happening what do we do and then get

that because then you might have to maybe you have to switch over to forensics maybe it's a legal case right so then you have to be careful where you step yes

yeah

yep

yeah if you have you have tons of data you've got people all over the place you got fifteen fifty thousand however many right billion doesn't matter you have to pick somewhere to start from so maybe your high risk users or maybe you know maybe you're gonna look your focus on the crown jewels out like your sensitive data that you're trying to protect and then move out from there and see like who's trying to touch that data or see these people who have this privileged access you know are are they doing something odd right because you that is a really easy path to compromise your domain access users maybe it's your finance department what-have-you but yeah I mean it's

definitely something like in your plan you're gonna say I'm gonna start my investigation centralize around this maybe it's a group of people maybe it's your data maybe it's just a zone and then you work out from there

yeah I wouldn't start with just going at a sim like it's all nalli involved or whatever had what it's free right right the big thing is to have the data and know it's there and then a way to search it so threat hunting you kind of have a lot of stuff that's required to get up to the point where you're looking for this anomalous activity in your environment but if you're trying to use it like maybe you don't have a tool that alerts maybe you're trying to look for that data so the experience getting to this point where you're ready to do that is all about experience right if you're trying to get into that you're gonna

you're gonna try it you want to view events right where do these events come from like saying you're on you have a Windows machine just look through your windows logs how do they look what do they tell you what can you tell can you answer the question about what you did say you start some programs you did some commands or you launched something or you deleted something and then how would you go about finding that activity and then you can kind of apply that to like a hunt right you're gonna find that was weird for me how do I find that for someone that I don't know and so you kind of kind of do

it and play with it it takes time so yeah generate data and then find it actually I don't wanna drop names that Splunk has this thing called boss of the sock so if you'll bots be OTS well you look that up that's actually would be really good for forgetting that kind of experience I think

yeah so like in the short-term like if you're doing a small small one and you're actually just looking for artifacts then you kind of pick those up you got to make sure they recording those or you got a document this whole time it comes more important when you're doing a long term project based one because you have to prove out your time but even in the short term one you know I found this here you know write that down you're gonna have to record that because that's gonna probably be an investigation in your environment and you're gonna need that that proof that that Intel that data to actually back you up on your justification on why you

said that we need to act on this because it's gonna in it's going to end up in a response so you have to have the data to support that response someone else

[Laughter] yeah you mean just like for the collection or oh yeah I mean I don't want to like really name tools but you know like I said I mean Splunk is tools that get all your logs in for Windows events I said Windows Event log forwarding so that's all built-in it's all you don't have to add any extras no agent nothing like that I'll use Windows remote management so if you have Windows endpoints that's just all gonna push it wherever you tell it and then to identify a force and then you can pull it off however you want maybe a logarithm or something like that with Linux I'd you know you're gonna syslog that all somewhere you want to just get

that to a central point however you want to use the log aggregation like maybe you do have like well you have your Alex deck right you have all the things and you can actually push those and then maybe you know give to that graph and then there's other solutions that have agents right that'll do it for you so I said logarithm is a free option or you can't remember it but it's a similar kind of things free agent that comes on this mahn is another way to do that that's really big for Windows things I love system on it's not really built-in but it's made by Microsoft and so I don't have a problem installing it but

you got to be careful with the data that it that it builds up so so I mean with actually all that that I just mentioned is actually a pretty good way to actually send the log somewhere where they'll be safe essentially on an interval and then you can just go and pick them up however you want using whatever tool yeah we're good okay so I mean you can find me in the hall [Applause]