PowerShell-Fu — Hunting on the Endpoint

all right how's everyone doing today my name's Chris Garretts I'm going to be talking about hunting on the endpoint using some built-in Powershell uh commands built a framework I'm actually going to be open sourcing this I did not post it yet because of Pride I've got to clean up some of the code that looks messy so but that should be posted by tonight or tomorrow um so everything you see here will be available um and you can go ahead and download that see see what we're doing um my background uh I was in the Air Force for 10 years I did instant response for the Air Force C for about five of that those years um I helped

establish the Enterprise hunt team there um our our entire job we had a team that would go out and look for compromises on our Network um after that ended up founding a company called infosite uh to build a product and a tool set to do hunting on the endpoint um so you can follow me here but one of the key things here is Powers shells was really great and I I started using Powershell uh very early um I was a electrical engineer in school but I never actually did professional coding I never did professional programming um when it came to getting stuff done Powershell was really the way I went with it because it enables so much power uh especially on

Windows systems which you know government all they have is Windows um it enables so much power to do automation to uh deal with data sets to deal with Excel spreadsheets things like that so that's kind kind of where I got started with Powershell was just being able to um parse things parse uh uh spreadsheets was how I started um that led into hunting where I had to get additional information from the the computer uh from the host and the tools I had wasn't giving me the information so Powershell unlocked that and some recent advances in Powershell uh which I'll show you have unlocked even more to make it easier to get anything you want off of a

system so all talked about threat hunting um what is it there's a lot of marketing out there today I two years ago when I got out of the military and created info site we uh got told no one's ever heard of that thing unless you've been in the military now everyone's heard of it and everyone has marketing on it so that's fun um so what is it it's a proactive search for threats hiding within a network you control that's really what what it's all about it's not um it's not a firewall it's not a um it's not using antivirus because what most tools on your network are trying to do is prevent an attack from getting in you want to stop crypto

Locker from getting in you want to stop an attack from getting in that's the number one priority unfortunately that's never 100% especially when you have Insider threats when you have people getting in your network around your automation how do you find that stuff can you go back on your logs was it logged if it's been there for six months do your logs go back far enough you know all these things are are big problems when it comes to persistent compromises and that's really the the big problem is persistent compromises most the tools today are to stop exploitation to stop installation of malware um your Ona access scans and things that that look at malware either on Transit or as it's

executing very little very few tools and I've evaluated a lot very few of them actually look in memory look in an operating system to find out if you already have a rot kit installed there's some open source tools out there but for most part our products don't do that even in the Enterprise um so the big problem is many are breached and don't know it you see the headlines every day um they're going they're being breached for a very long time just the the DNC hacks you know that was a year OPM Hack That was a Year everybody that every breach report that comes out it's usually been there for a year and they didn't know it and I've

got a long list of these things and how long it took for them to actually find it how are they finding them today usually it's it's Brian Krebs writing an article or uh the FBI telling you nine months later that they saw some stuff go to Ukraine um but that's legitimately almost all of them are found that way so hunting really bridges that Gap let's go find some some breaches um this debate comes up a lot is hunt and digital forensics is that the same thing sort of they use a lot of the same techniques but it is different because we're going to be more scalable um a lot of the the products that are in

Ed today for forensics and response are not very scalable when it comes to looking for a a wide net if you know what you're looking for and you have a breadcrumb trail perfect if you're analyzing one host there's a lot of tools to do that but what if you have a th000 or 10,000 how am I going to do that you need to be um very scalable and you need to reduce the complexity otherwise you're just not going to be able to do this very often um there's another thing when it comes to hunting is principle of diminishing returns a lot of people when they get started with hunting is they're looking for something specific like

let's go look for you know signs of mimik cats being run um and they go look for that but if you're just looking for one thing you're going to have a problem when it comes to hey is that thing even there did they use that tool or did they use a different tool so if you're just looking for something I call that focused or ioc based hunts you're looking for something that you know you want to look for it's some kind of hypothesis you know I'm looking for this specific type of malware I don't recommend that because it's just not scalable and there's no Roi in it um so the methodology we're going to show you

today is basically collect a lot of stuff from the endpoint compare it against you know every data source we have to see if we can find things and then filter to the top through uh through anomaly detection what's interesting to look at I guarantee if you have some reverse engineering skills or if you have some malware analysis skills if you're looking at malware it's going to be pretty obvious to you the problem is you have 10,000 hosts and there's a hundreds of thousands of processes and and drivers and everything running in your environment how do you filter that down to something manageable so in the Hunter's tool bag there's a couple different ways to do

this um there's endpoint Solutions there's data Centric Solutions if you're in a large Enterprise you're going to do a data Centric usually almost all of the information online right now about hunting is data Centric you've got some kind of database a Hadoop cluster or something that has a massive load of logs and event data and packets how am I going to hunt in there how am I going to um build queries that let me find additional context in there and find find those uh those bad guys rolling around in my network um you're not going to have that access to that data and that data is not going to go back far enough if you're not if you don't have a

limited budget which most people don't so uh the solution we're going to show you is an endpoint solution script we're going to be able to find everything on a host regardless of what security stack is already there and what you've collected and then of course malware analysis tools when you do find something suspicious and weird um actually verifying that that's actually malware you need to be able to do that as well so again that's just two different ways big Enterprise event data Centric um the rest of us we're going to do an endpoint validation strategy validate what's on these systems if I don't trust this computer how do I begin to trust it so without further Ado I uced power

uh PS Hunt is the uh threat hunting module that we're going to be releasing today it is a Powershell module that allows you to um scan systems gives you a couple different options to either look for information that you want to look for or just survey systems for everything um from what's running what's triggered to run to any manipulation that we're looking for so we'll go through that the different components of this um there's about six different components I break it down into scanners and surveys so a scanner is a script or a utility where I'm going to be looking for something remotely I'm going to be using wmi I'm going to be using remote

uh registry to say I want to find this one registry key across my entire environment that's going to be very fast and it's going to be threaded so I can look for the one item if I know I've got a problem and this specific malware uses a name pipe you can enumerate name pipes and that's for uh for clarification that is when any malare that does lateral movement with uh um SMB which is very modern everybody's doing it these days um if they use a specific name pipe you can enumerate that name pipe remotely using Wy so that's a scanner a survey is a script or a module or an executable then I'm going to drop on that box and

execute and let it run that's where we're going to get more information wmi and everything else and and remote registry is not going to give you the kind of information we need at the level we need uh so survey is something that gets dropped down that box and we have uh one major one that uh that I developed that lets you get all the information you need from that host Discovery what assets are out there building a Target list um your utilities for transport and execution um and then how do you analyze the data from that survey and then how do you analyze files that you identify so all of these components go into the

the PS hunt framework uh scanners again this is going to be used to rapidly scan systems basically I'm doing wmi queries I'm doing register queries those are the main ones looking for that one thing surveys are again dropped on the host some of the challenges is a how do I get it on a lot of hosts we've got the transport and execution methods for that um B is how do I get it back how do I get that information that I just dropped on disk or wants to return to me from memory so if this is going to run for a little while because most of my surveys do a couple minutes uh one way to be

scalable is you can either go back and pick it up from your main box that you just deployed it from or you can have it automatically send it up to a a an http P server or an FTP server so that's how you do it scalably I usually host one on the box that I'm scanning from so that it always returns to a folder that I I have there's another way to do this uh so the surveys themselves can be quite large if you're packaging um other executables so our the implementation here is going to use some syst internals tools so I actually use Sig check for instance because doing signature checking and power shells hard um so I

use that that's an ex extra 500 kiloby so there's two ways to do that if you have an executable if you want to get down to Kernel level you're going to need an executable Carell won't let you get down to Kernel um you can do it two ways one I use a function called download file that's basically says I have my script it's very small when I deploy it out that host is going to download from the internet an additional module so I actually download uh Sig check from Microsoft uh system journals live so they host it it downloads to the box and gets executed if it's not already on the box another way is if you

don't have access to the internet and you don't want to you know make 10,000 boxes call out to Microsoft Live um you can actually convert that binary into a Bas 64 string put that in your script and now your script has that executable to be dropped on dis when it runs um that's some really cool ways to do it packages everything up into one Powell script so how are we going to get it there um I implemented five different ways to do uh transport execution uh transports usually done with SMB server message block basically just copy file um for those power shell folks that know that uh wmi Powershell remoting scheduled tasks and service manager all

of those are implemented into one wrapper which uh I'll show in a second so all of them are wrapped into one Powershell script if I want to deploy a survey it's going to basically reach out figure out which ones are available on that host if it's a Linux host it'll tell you to go away um if it's if it's got wmi exposed then it's going to use wmi um if that doesn't work it'll it'll back down to scheduled tasks so all those use a different port um I would do this all with power strol remoting but I've almost never seen a Network that has it enabled they're very rare so I am a big Advocate that you enable it power

remoting should be your primary um remote admin protocol it is awesome it lets you do anything you you need to do on a network and managing it do doing Discovery and access we have to get access to the system on these ports so depending on what protocol we're using uh we need we need these Port access so there's a couple Discovery Tools in there that allow you to automatically do this if you go ahead and just hit the um the uh invokes remote task which is the the task that does it it will automatically do all this for you you don't have to remember it um but it'll look for TCB ports it'll look for assets out out in your network

to be able to find it and determine if it's the right operating system things like that the Windows host survey is pretty complicated lot of code in but each of the functions is fairly well uh and doesn't have a lot of dependencies so if you want to go online use it you need a function that you know in numerates all the name pipes I got a function in there for that so you can just take that out of there um so what we're going to do is get active processes modules drivers any floating or injected DLS in memory um we're going to look for active connections any auto starts the implementation I have for auto starts is

just Auto runs so it'll download S internals Auto runs it's not perfect but it does the job um and accounts and key event logs for execution so the idea for the survey is to collect enough comprehensive information to give me what's the state of this system just to uh normally in Powershell you can do a command like get process and get process will give you the process list unfortunately it doesn't have all the information we need for hunting it doesn't have the hash it doesn't have sign signature information um it doesn't have the owner of that process so there's a bunch of things that are missing from the standard utilities that are in Powershell so I

built a wrapper for that to put all that information in there so the surveys is collecting with I get process list it'll get the process list from wmi it'll get it from uh additional information such as the loaded modules it'll get the hashes it'll do signature checking and validation uh as well as get the owner so everything from processes modules drivers Auto runs everything is going to have a similar um layout to it to where all that information is available for me and when you actually see malware there's going to be a lot of things missing and there's going to be a lot of things weird and so I'll show you that a a little bit later

here persistance mechanisms right now uh assist internals Auto runs is going to be what we're using we'll download that execute it and then wrap some additional information in there as well so just as an example um here's my totally not malware freeonline game.exe uh this is running and it's going to run key so you're going to be able to enumerate all those with auto runs uh this is the other thing when you get this inform this kind of information on every file you're going to find things missing most people who write software for a living for like companies to use forget to put a publisher name sometimes they forget you know to put a version name in there

um but Mal writers almost never bother so you a lot of times you don't see any of that in there unless they've troed another executable uh another thing that I always find if it's not at troed is the internal name is never the same as the actual file name because when they're deploying malware they they'll try to hide it as a different file name hide it in with the system so that internal name is almost never the same so those are the types of things you can do with that data set is just look for that that kind of um pattern so memory resonant malware this is the cool part so Matt Graber if you've ever used power exploit or any of

those other tools uh he built a module called PS reflect and it changed my life with Powershell because I no longer have to compile C code I no longer have to do um I no longer have to use net to get access to the native Windows API so just directly from Powershell just by including his PS reflect module into our survey um anywhere else I'm using I have direct access to Native Wind 32 apis I can troll through memory I can look for things um so the implementation I have for this to look for this is I'm going to do a virtual query walk across process memory and I'm just looking for portable executables and then protected

memory spaces uh that's um a pretty effective way to do it there are ways around it but that pretty much finds 98% of the malware out there that uses process injection and almost all malware does in fact I don't have any this is all user mode so um a question that might come up is what about kernel mode rootkits how do you find those well modern rootkit design uh actually actually doesn't have all the functionality in the kernel module so every rootkit that I've seen from like Euro buros to everything they don't put all their functionality in it if they got networking functionality or parsing or command interpretation all of that is in a user mode DL that the kernel module

injects into memory so you can find most kernel mode rootkits by just looking for their uh injected modules in memory using this so how do you analyze this data once you get it so we got a few functions um I have lists of reputation so if I've done a virus total lookup I store that data if I've got um I've downloaded the N database I have a list of hashes from the NIS database just to give me my white list and my Black List um I load that right now into a global variable so it's all in memory to do lookups against uh this might be better to do with like a like an SQL light or something site uh

but for now I just do it with pure power shell and loaded to memory so make sure you have you know a gig of extra memory because I'm storing a lot of lists in there um so the objective is to compare everything we find in these surveys against the white list and The Black List query virus total um we can put additional modules in there to query you know another set of thread and tell sources but for now we just got virus total in there um and we'll the implementation I have available if you only have a free API key from virus total you can only do one look up uh every 15 seconds that's they're

throttling so usually when I'm doing it with a free key uh I'll just run this overnight so you can do about at that rate you can do about 5,000 hashes lookups a day with vir toal with a free key um and then if you've got thousand hosts you need to group them so that you can look at all the data at once you don't want to look at one host at a time I got another function called group host objects so the host object is what I call the results of that survey um we'll group that together so we can look at all the data once just with uniqu set if you've got 10 if you got a thousand

Windows 7 boxes you'll have the same Internet Explorer on most of them so uniqu that set and and grouping it uh makes a lot of sense for that so let's put it all together how are we going to do this so when we actually deploy we're going to be sending out our surveys to the host it's going to get what's running what's triggered to run and any indicators compromis that I'm looking for we'll collect that either through uh going to grab it or waiting for it to return to my FTP server then I'm going to do some lookups I'm doing that on the back end the cess internals tools actually do lookups themselves I have

that disabled because I don't want every box doing that uh so I aggregate all the data unique it and then I'll do my virus total lookups and my hash reputation lookups after that you get a certain number of suspicious executables unknowns no one's ever seen this before you're going to get a set of those those are the what you actually triage you look for uh anomalies and and unknowns you do some data stacking to be able to find out hey this is an interesting guy let's pull a sample of that file back from one of those hosts and let's submit that to virus total let's submit that into um let's pop it into PE studio so

um some of these open source tools that allow you to do that uh poow shell Arsenal has some cool re tools that you can use um and then of course Dynamic analysis if you want to pop it in a sandbox so mare.com is free you can do cuckoo that's free that's not part of the um PS hunt module yet that's kind of uh manual right now but it will be implemented later so how do we find bad things um misspellings and lack of spaces all right active processes modules drivers all these things are going to be normalized in the same data set so we can stack them we can look at them so a lot of malware today if it's

Advanced it's not going to try to hide from you so there are things that are hide root kits that'll hide user mode root kits that are hide um a lot of malare today hides in PL site so you know you can still enumerate it using these these uh these API calls um the initial technique of force is to hash everything compare it um look for digital signatures make sure it's the correct digital signature and a good CA uh certificate Authority um and then compare that against virus total that'll clear all our good knowns known Goods known bads from the unknowns stack the remaining data and do anomaly and outlier analysis look for things that are weird look for things that are out

of place look for a due frequency analysis and say I've got a th boxes that that have that version of explorer.exe but you've got you know this other one running on this one box and that's interesting because it doesn't look like a Microsoft file so anomaly analysis and outlier analysis using that data set and Powershell is awesome for that because you can filter things you can structure it in different ways it's awesome uh and then of course when that when you've filtered that list down to something manageable start analyzing those files and do it with static and dynamic analysis digital signatures most malware not digitally signed especially when they're popping different DLS around um and the ones that are often times they

load a rogue certificate Authority into your Windows host so U one of the things that the survey does is collect all of the certificate authorities that are loaded in The Trusted host store on every computer um and then we'll be able to check that they all have serial numbers they're all searchable online and you can tell you know who certificate Authority is this um I used to think that there was a good set of certificate authorities there isn't there's no online like white list of certificate authorities it's good other than Microsoft set um and most of the experience I have is a lot of people stand up their own certificate authorities especially in an Enterprise

they actually have their own and they're not even like it fails every check you do because they've implemented it wrong um so just be aware of that but often times you can find um Rogue CA that someone's just stood up signed their malware so if you're not doing your signature uh checking correctly it'll come up as signed on that that host and of course if someone compromises a legitimate CA you're screwed with this technique so it's not going to work and that's what happened with like bit9 ca in 2013 persistance mechanisms um trolling through scheduled tasks jobs pers um registry persistance uh looking for anything referencing an executable or a script so once we have that reference

we're going to collect the same data that we would normally collect on our processes so all those things I told you about on the get process list I'm doing that with anything referenced in the registry as well especially if it's it's referenced in it's on disk boo process redirection I don't have this done yet this is um it's messy so I don't have it in there yet but um one way you can do that there's a module called um Power forensics that you should download it has the ability to get a lot of low-level things one of those is you can grab the MBR and if it's redirecting to a different bootloader you can find um

boot kits as well memory injection the way that it's always done every type of memory injection out there whether it's DL reflective D injection or process holl they all have one thing in common and that's unprotected memory regions that are contingents they're always in one stack um you can't do reflective DL injection in Heap you can't do it in and and different things like that so it's almost always going to be in one contingence memory section there there is some malware that will like move itself around um like it'll pack itself and unpack itself in memory that's a little more advanced this will you'll miss it with this technique but for the most part um you know 90 plus per of

malware out there will have all their malware in contingent sections so just look for unprotected memory regions um legitimately loaded dlls their PE header is supposed to be readon especially if you have um dep enabled so when you look for read execute or read write execute and you find P headers in memory those are almost always not supposed to be there or you've got like net doing just in time compile in memory so those are your false positives you'll see some of those um but oftentimes they won't have PE headers so we'll just walk PE headers and uh in volatility they do this at the kernel level we're doing it at user mode so uh it's still very very effective

most of the malware that I've tried this on it still finds the PE headers um unless they've mangled their PE headers and it's just more complicated but it does happen so that's it for now more to come some of the things that I'm working on um enumerating hooks doing custom Yara scans uh being able to deploy that widely um did a lot of cleanup of this code this code set was actually what I showed our investors when I started my company because we were doing this as a service for folks going out and doing these scans um so we're open sourcing this now it's it's an old uh old tool set set an old uh code base about 2

years old but I've updated it and I will continue to U maintain it um but we'll be adding more uh techniques in there to find things on the endpoint so I'm at 30 minutes now so I guess we have 15 minutes for questions

rightone question

ask any questions if you're not using Powershell you should and if you manage the network please enable PS remoting it's a simple command are youting your slides um yeah I gotta figure out where I'm gonna put that I'll put probably put it on SlideShare so cool how many people use Powershell awesome cool well I'll be uh I'll be around yeah so I use this to do active directory assessments um but we haven't scripted it so have you thought about doing this with privilege so you can kind of scan out through the network and see what kind of ab is doing and how it's being managed onc environment yeah unfortunately right now for my open source development I don't have a big

domain to test against so I've got like a little virtual box containers with 20 boxes so I don't get a dohole out of that um but that is something that I would do if I had a playground but um yeah all this stuff is done with privilege privilege you have to have an account that has privileges um and the active directory stuff uh there's there's a couple tool sets out there that some of the pen testers have like power viiew and a couple other tools that enumerate active directory like accounts um using that for hunting isn't as straightforward as I originally thought but it can be done