Memory Analysis During Incident Response

all right can everyone hear me great all right so i'm brett cunningham and i'm going to talk about memory analysis during incident response thanks patrick i got the giggles back there so what is memory analysis it's basically taking a look at ram taking a look at the artifacts that are it's just in temporary memory and in this talk i'm going to speak to mostly what my abstract was about i didn't have enough time to completely hit all of it but we're going to take a look at we're going to take a look at basically moving beyond the basics which generally speaking is just using strings also take a look at what artifacts that we can

look at more intelligently that reside in memory data structures specifically that help help you put the full picture together and how they pertain to the incident so we've got things like network connections typing on the command line and the residue of injected processes so anyone here do incident response besides that back corner all right and those two got stragglers all right so we're gonna go with the absolute beginning where when i first started i didn't know what i was doing um this is even a step beforehand so what happens during incident response is you want to dump the full contents of everything in ram all right hb gary has a tool that does this called

fdpro and it will dump it into its compressed file format i had no idea what i was doing so when i was told to run strings against that i ran strings against that actual compressed file format nothing came out all right so what i'm doing here is the strings minus el this command history is the raw dump of everything in memory the el is a combination to include unicode in there and then i'm grepping for telnet so here why i'm grepping for telnet is to see if the attacker had put any typed any commands on the command line uh with telnet and we see here that they telling it to this ip address on port one two three four all right

problem here is that we have all this extra information that really doesn't make sense it's left over from um it's left over from the telnet program running and just other just other junk so so to go a little bit more intelligently with that we want to take a look at data structures and can anyone here describe what a data structure is in a very simple term all right i kind of struggled a little bit when i first learned about it but it's used by programmers to create a little bit more advanced variable type so if i want to create a variable called movies it's going to have all these members that belong with it so we're going to have a variable of

a variable called title is going to be of type string an actor string so the name of the movie is going to be a name which is generally what a string is in the year it's going to be of type integer and integer is uh typically just a number all right the price is going to be a float that's a number that can get some decimals after it all right so we end up declaring a data structure by saying movies star wars so star wars is going to be the name of our new data structure so that's the name of the data structure the data structure type is movies if you want to know what the background

of movies is you have to come back up to where you declared what the structure of your data data structure is going to look like so to put a little bit into perspective if you wanted to create that movie data structure and then assign the title it would look like movie dot title equal encino man the main actor being pauly shore the year and the price so this right here would be the data structure type of movie all right so in windows whenever you create a new connection specifically this one's a tcp connection it creates the structure type of tcpt object and underneath here it defines everything that belongs to that that data structure here we can see easy ones that most of

us i think are familiar with remote ip address local ip address belong to this tcp object the remote port and the local port belong to it and then we've also got the pid number

so i'm going to use a program called volatility to extract these data structures volatility is a command line program that helps you run through memory analysis more efficiently and quickly than using something like a debugger and more intelligently than something like running strings here you see it's just as a memory analysis framework that looks a little cut off there we go all right and we're going to run conscan so this is going to look for any tcp object in memory all right here it gives us the offset into the ram dump that it found the object we have as we looked at in the slide we have the local um well we had the remote ip

local ip then we had the remote port the local port and then we got the pin number that it belongs to so the pid being the unique process identifier that you typically see in something like task manager

and what's interesting is if we take a look at conscan what volatility essentially does is it opens up that ram dump which i did in a hex editor and it will go through and look for specific look for that data structure that's defined and here we're just reading it in hex but you can see that we have the declaration for a tcpt object

so it's iterating through this file looking for basically the start of the structure and then everything's defined in the structure right here and it starts ripping out all the details and that's simply when you go through and type in a command like this that's what it's basically doing in the background so it saves you a lot of time second command we're going to take a look at if you remember we were doing strings in telnet do we know if the attacker went through and typed in any other commands the only way to take a look at that is by replacing telnet there with other commands so you can end up building this huge library of commands and trying to run it against

a file but you saw how much junk we had there volatility has what's called command history yep command history was reverse engineered out of windows um in order for uh the volatility crew to be able to extract the data structure so there's no real pretty way to show this um unless i made my own diagram but obviously i didn't but we are taking a look at this data structure called command history and all the members of it are listed right here so flags being one list entry application and so on all right

so it's command scan there we go and we when we run command scan it goes through and we'll pop out the commands that the attacker typed right here so we saw the first telnet to one two three four if you guys looked carefully when i showed you that huge blurb of data there was also the telnet to 81 but i mean it was hard to see that when you get lost in all that data we also see you know an ftp connection outbound um trying to enumerate a little bit on the domain change directories and then finally that's me coming in and dumping dumping ram into a file uh win32dd belongs to moonsol's memory uh and basically it's been a fantastic

way to get a memory dump yes yes it should always be that case i'm not sure if that's true but these are in particular all right you should see that it's typically always with csrss.exe because that's the um that's the executable in windows that handles the win32 console application

and just to briefly run through what it would look like if we were able to fully do this by the way um i'm gonna sidetrack real quick here with um with task manager this is stored in a data structure in memory what the processes are and one of the it's just called e-process one of the fields in the e-process is the name of the process the unique identifier and so on the commit usage and all that so if it helps just understand what a data structure is i mean this is kind of the best graphical view i can lay out so when going through and looking for looking for data structures you can essentially use something like when debug open up

the memory capture and list all processes we can list all the processes with that command of pound process zero and then we know that cs rss it always handles command histories that's always the process that that data structure is going to belong to so we can find upwards in here is that true on the powershell side as well i'm not there yet yeah so anything with windows 7 i haven't been able to confirm that so this is for windows xp and server 2003

if we find upwards we can locate our process so the image name is csrss and what we're going for is the process id which it's shown there we can then switch to the process by typing pound or bang process and then pass it the ready here we go okay we switched there and now it's showing us all the threads that belong to this process

yeah yeah this is one example of a thread so each one of those belong to the parent of csrss and if if the structure wasn't reverse engineered if microsoft had it outlined we'd be able to run through all the threads which i'll show you later on a little bit better example yeah i'll show you later on a little bit better example but we'll be able to go through each thread and extract the data structures out of there

so that plug-in was written by a guy named michael heilay hallway i think and if you guys recognize that name it's because he wrote the malware analyst cookbook well part of it he's done a lot of work with the volatility project and made things really simple so we're going to sidetrack here and go to timeliner timeliner was written by a gal named um jamie jamie levy uh her online handle is just glitta g-l-e-e-d-a and what she's trying to go for is pulling out pulling out time stamps of interesting artifacts from memory the supported timestamps she has is the system date process timestamp so when a process is created thread time stamps network time stamps so the thread belongs to the process

network timestamps are anytime that you create something like that tcpt object that we talked about earlier registry key last write times which are interesting especially when you have a registry key that was written to one of the auto runs so anything like current version run if a program was you know included into that and then some of the event logs for just xp in 2003 and then any pe timestamps so some of the metadata that you get from windows executables such as an exe or a dll how she did this was that she borrowed some commands from volatility already that we're already in process so we have we have uh it's called ps list and ps scan i should

show you ps list goes through finds a process and then tries to enumerate all the other processes so it's essentially like um task manager and you see here that we have the time that it started ps scan is essentially the same thing it takes a little bit longer but it goes through and it ignores information from the operating system and really goes through the entire file and tries to see if there's any other processes so what that's really helpful for is in trying to look for some kind of root kit something that's trying to abstract itself from being found in task manager so so she went through with timeliner and used the e-process create times e process exit times

for thread scan it's the crate and exit times for uh the plug-in sockets and netscan use sockets create time um more formatting required don't remember what i mean by that um

so we're going to take a look at the output this is how you run it just call timeliner pass it to volatility and give it the image name and the output file uh one thing of note here is timeliner does not run on the latest development branch it's locked in at 2.0 otherwise you get some pretty nasty errors and just let it run it takes forever so i had gone ahead and yep ran it so here we see a couple of the different modules or when i went through and said and she borrowed you know eprocess.createtime and all that we see here we get the time frame what kind of data structure was dumped here the name of it and then

some of the additional information you see here we've also got socket types the vent log and event log is really short so i think it's i think what's happening is when the event log is written it hangs out in memory until that area needs to be used again so it's freed up but when memory is freed it doesn't immediately get overwritten so she identified the structure of that and is just capturing it because you see that this event log i mean generally speaking it will go on for a long time and that was only what two pages we've also got uh thread and it will part of its data is showing the uh the parent id that it belongs to

so the process id and pe time stamps last one should be way down here the registry i'm not entirely sure how much of the registry it covers but you see we're six percent through the file yep okay so registry keys just started we're six percent through the file and it continues to go down forever so what i need to do next um is either ask her or just compare to see if there's any what the differences are and why we're doing this is during any kind of incident response memory is the very first thing that you want to capture it's the thing that's the most volatile it's going to change anything that you do on the computer is going to change

the memory so you grab that and it's going to be real interesting to see how much data we can extract from the memory before the rest of the long before we're waiting for the rest of the logs to come in okay yep so the last thing that we're going to take a look at is injected processes

and how we do that is by looking at what's called the virtual address descriptor yeah the virtual address descriptor is a tree-like structure that it's basically a train that tree that hangs upside down you have your vad route so the very beginning and it continues to branch out what this describes is the pages that are actually being used in memory um windows waits until the absolute last minute before it creates uh any memory page so if a program says hey i'm gonna need you know 512 megs of memory for my program and it asks that up front windows doesn't do it until that program actually attempts to write there so the vat is describing what is

actually being used

yeah so in bad parsing we are walking the vet so we're going through each each process is tree so every single process has a vat and what we're going to take a look at are the two windows api calls that are infamous for process injection it's virtual alloc ex and write process memory and what those do is they create a vad page of the type mmvad short and it has the permissions of page read write execute and no cache which i didn't put that on here but we'll see that in a second other valid pages page types can be mmvad or mmvad long so you have three types and what you just have to know is that

virtual iloc ex and right process memory since they are used for process injection the left behind artifacts that they have is that it's of m bad short and page execute read write plus no cash

so i've been experimenting with volatility's shell lately and it's been cool

we enter into the shell by calling volatility and just saying valshell and it will give us an interactive command line program has anyone used something like python's interpreter or irb for ruby all right it's very much like this and it it's in python it uses volatilities modules and classes that they defined

so p.s will give us a process listing i've already kind of cheated because you know this is going to suck if we go through every single one i know that explorer.exe is one that i'm interested in since this is python we could just iterate through every single every single process here and look what we're looking for so i'm going to change to the context of explore.exe by saying change contacts and then giving it the pin number of explorer.exe 1096. okay and then volatility is kind of cool because it will give you it will describe the e process structure e process structure when you call upon it so i just asked volatility i go dt and then give me the process chuck the

e-process structure you can do this with other data structure types that windows has defined in here you have create time exit time those belong to e-process you know active process links quota and all that the interesting one that we're going to come down to is the vad root this is a pointer to a pointer of where in memory the first committed page is well the data structure of vadru which will then lead us to where the first committed page is

for this process we switched using this command we changed the context of explorer.exe so i want python or i want volatility to fill in the values of the current processes data structure so by issuing dt then self.eproc this says for self meaning the current process give me the e-process structure with all the values filled in here you see the create time exit time hasn't occurred yet um and then we come down and we see vad root bad root is a pointer so i want volatility to now give me give me the information of the data structure of vat root for the current process so current process all right then it goes to vad root and it's now printing out with this it's

now printing out the data structure and the values of it here we can see the tag is vad s and then really the right child and left child are the ones that we're going to take a look at right child and left child show us

this will describe to you the tree structure what i'm doing right here is straight out volatility's um you know handbook or whatnot but when a command gets too long go ahead and throw the value of that into a variable the variable we're going to call vad root probably be appropriate um great so now we're going to ask it to describe to us what's at vad root at the right child so at the right child it goes and you see that it's another data structure that looks like that root so these are all vad nodes that's what the data structure is we can continue walking the tree by saying go to the next right child so we went

from the parent we had a decision to go to left child right child now we went to the right child for you guys um and we got another decision to go the right our left child we can keep going down to the right child right child and so on until we finally hit no other no other committed pages and that will be of the value zero so that's just how you navigate the vat

why that's important is we're going to now take a look at what the member the data member u has in it so when we ask what's in you know we navigated the page um to find some node and then we ask what's in you we see that it's long flags and bad flags we're particularly interested in vad flags that flags is now going to show us the characteristics of that committed page and memory we see a lot of like the commit charge physical mapping private memory some of this information you see and some of the more advanced use for a task manager what we're really interested in is the protection level the protection level is four here

that doesn't really mean a whole lot until you come back and you look up memory protection constants so what does four mean four here says page execute read write so that means that that page in memory is able to be read written to and if it can be executed so if that contains code that the code will be executed

now we talked about the virtual alloc function and it's ability or okay this is what the virtual alloc function looks like when you call it and we have this called uh fl protect fl protect if we come down

um will lead us to this page that says memory protection contents memory protection contents we saw page execute read write but we're also curious about the artifacts that are left over um in memory so if you have not only the page execute read write plus the page no cache we see here that in microsoft's documentation that this is often page no cache is often accompanied by um virtual alloc ex and quite simply if we

if we use volatility's vat info so this will give us the info i pass it along the pid of explorer.exe pass along the pit of explorer.exe so i'm asking it to give me the information of every single vad node that belongs to explorer.exe so it walks that tree for me which we just did manually and now i'm looking for protection type 6. all right the grep with the ba and or the b2 and the a2 means give me two lines before giving me two lines after so this puts in a much easier view what i want to see so it tells us that in explorer.exe we have 1096 gives us the beginning and the end of

the memory or beginning and end of the memory page the type of that it is so vad s means um that mmvad short that we saw earlier and i said i said here if it's a mmvad short and page rewrite execute that's you know a leftover artifact from using virtual alloc exo now we just identified all the pages that have the tag of s protection s and in short uh that is what malphine does

all right we can run this command and it goes through it iterates every single process it iterates or it looks up the vad root then walks that vad root to grab all the pages looks at the protection level and the type and so malphine goes through and it's trying to look for injected processes it also does a lot of other things but we'll skip that today

the output from it is a dump of all those memory pages if we go into there and just take a look you can see it's just data all right so this data really doesn't mean a whole lot um but what's nice about malphine is that it will help interpret it for you it's going to assume some things about the architecture type and be able to interpret it for you so it found a concerned process or found a a page in memory that it's concerned about it tells you the process it gives you some of the attributes about it such as page execute read write protection level six so that means page execute read write plus that no cache we

look that up in the memory constants and then it'll try and start spinning out the code segments this should right here if i recall be the page one of the pages of the injected code and memory so

all right

yep so that's that's absolutely it right there um we went through uh there we saw the process that was injected into we saw the characteristics of those memory pages that that made it interesting for malphine at the beginning we just took a look at the data structures for connection scan for the command history and i hope you saw that it was a hundred percent more reliable not sure i should throw that out but uh a lot more reliable than using something like strings so now i'm gonna ask for any questions and we got jack back there thank you so um did it list the bad number or the page where that executive process lived and then so could you run strings

against that page to i guess better determine what was injected so it gives you the offset here the address um i suppose you could go through dump that uh region in memory using something just like a hex editor carve it out and then uh run strings acro against that i guess it would be one way to target that specific area of memory yeah that's that's a great question that's actually one that's one method of being able to reconstruct binaries so if you have a trojan that was deleted off disk but it's still running in memory that's one one way to reconstruct parts of it so you can then give it to your reverse engineer guys they can go

through and tell you what they can glean from that little bit of information so great question thanks

okay all right great i think uh command history written by uh jamie levy i think that's still only for xp in 2003 but what tim back there is saying is that the data structure type type changed a little bit but it's not too far off from what's currently in there and should be entirely feasible to still grab it uh yeah it should be i mean that's how that's basically how windows finds its way around uh memory that's actually been committed so yes would process injections show up in a situation where you've got uh detailed process tracking auditing on the box in the sense that i mean one of the things we do on higher security machines is enable detailed

process track so anytime process starts on the machine or you know does whatever certain operations you're going to have that which seems like it could be a shortcut for some of the early stuff here but i don't know what the ability for example and that kind of stuff is to sneak past detailed process tracking or whether detailed process tracking might be pulling out i'm not sure how that works but i'd love to look it up and see so yeah let's talk afterwards any other questions all right great thank you

i did

all right