AJ Leece: Malware Analysis for Incident Response

so welcome everybody uh to malware analysis for incident response uh my name is aj i will be your host for today uh so today's talk uh definitely uh we'll get to it i just really want to say thank you so much to the besides organizers i can personally attest to how much effort goes into these conferences having been a part of uh some of the development in previous years and what you see is is bookended by just an enormous amount of effort from everybody associated uh any volunteer that's here has put in considerable amounts of time uh as you have no doubt seen already or heard the ctf is like unbelievable i mean year over

year that consistently gets much better uh and then put compounding the fact that we have uh obviously a pandemic still in our midst you know doing this in a virtual environment definitely comes with its fair share of headaches you know i like to maintain that everybody has a plan until 250 people log in all at once and that's probably a little what we're experiencing so big thanks to everybody for the patience on that but really really big thank you to the b-sides organizers if you get the chance please give them a shout out let them know that they're doing excellent work i genuinely appreciate it the support is really excellent and i hope you all get something out of

it uh so thanks for the uh for attending uh we will uh we're getting going on the biological malware version of b-sides here uh so let's get to it so a little about me some shameless self-promotion uh there's another bit of letters over here somewhere i just forgot to add them i've been in the business for a little while now uh in many different capacities previously consulting uh internal operations uh client-facing operations spent some time as a qsa uh you know a little everything and uh definitely focusing the majority of my effort as of late uh in the incident response field uh so i was a teacher in the iss program at state briefly uh i was there for a couple semesters

and now i'm uh off to something else that i'm working on so it's good uh one of the big things i always noticed inside of organizations was that incident response uh definitely wasn't handled as effectively as it could be in some cases uh and there's definitely a few reasons for that we're gonna get to some of the way the incident response process is supposed to work uh and you'll actually have an opportunity to sort of test some of these concepts a little later today and tomorrow i'm giving an incident response tabletop workshop uh where we will be playing a game that i developed as well so we're having an incident it's in a game like fashion uh and it's

it's going to be pretty interesting so that's that's all the hints that i want to give to that if you want to come check it out be sure to do so it's going to be a good time uh so let's talk a little bit about the malware problems uh at first what we're actually looking up against uh i'm going to talk a bit about the ir process and how it's intended to play out and what an organization can do to sort of uh increase that we'll discuss a few benefits of malware analysis so what is the net positive to the organization for having somebody on staff who can conduct this work i'll tell you a little bit about how to

set up your labs it won't be a very comprehensive lab setup uh reason being is the actual lab setup isn't overly difficult so if you have any questions at the end of it don't hesitate to ask but i'll certainly talk you through that process it is assuming you have some basic uh fundamentals in vmware uh but that said it's nothing i couldn't catch you up on in a hurry if we're doing lobby con afterwards so don't don't worry about that and then i will align the analysis efforts to the ir process so that you can start to piece together how you would use these uh artifacts in an incident response process and how to make your ir a little bit

more effective a few things to consider and then i'll open the floor up for some q a or cut you free for some time back so based on the verizon data breach report they had 509 reported malware incidents in there at least in the the stats that i was looking at of those nearly 40 percent were attributed to password collection as everybody is probably well aware plenty of attackers usually seek to steal credentials usually as a means of persistence or furthering their campaigns so that's often targeted by a large portion of attackers for sure makes sense to weaponize that in a malware form if you can get to it in addition to that 25 percent of those

were attributed to information theft uh so simply collecting the data from the workstation uploading it to command and control server and then moving on uh with the rest of the campaign and of those 20 were classified as ransomware so uh the majority of us i think at some point whether we care to admit it or not have worked through a ransomware incident uh they certainly are no fun uh they were definitely never my favorite things uh but uh it's it's out there it's happening a lot uh and definitely given the remote nature of work and how some of these processes have likely changed uh information security is is very important and a lot of organizations i think are gonna start to

run into problems here one of the things i've noticed is when it comes to a malware outbreak most organizations tend to rely on tools uh or capabilities that you know maybe are just a little bit inefficient or a little slow on the uptake one of the commonalities i've seen across most companies is we're going to let av do the scanning and hope that it returns something usually is the first method of identification uh that doesn't always work especially if your antivirus hasn't heard of the sample that uh got loose in your environment uh or if the signatures have changed or if the models haven't been trained accordingly so uh it's not a guarantee that your in-house tools are gonna find

it uh the other thing to consider is malware is so publicly accessible uh many criminal organizations have gone corporate now they have the ability to stand up a call center to give you the encryption key uh that they themselves use to encrypt your data so as long as the payment clears everything's good so there's very low barriers to entry for anybody looking to start a malware campaign and make any money at it the other side of it is companies have to pay a lot of money in malware incident response oftentimes the organization's down critical parts of the business are down for extended periods of time uh just the pure labor hour cost can be insane

just because so many people have to be involved usually in intermediate situation from a pure labor standpoint that gets very expensive uh so there are definitely it's it's not in the organization's favor in this case uh the criminals definitely get the upper hand on this one so it sort of behooves us to have a better process for finding some of this and being a little more effective in our response so at a quick glance uh ir breaks down into very distinct categories you have your preparation phase this is basically the precursor to an incident uh this is you know ideally you should be building the ark before the rain comes this is all of the work that you would

be doing to get your teams ready so i'll touch on these in a few seconds here from there you have identification so assuming an incident has been called and you have alerting and everything in place at this point now it becomes about identifying all of the compromised assets inside of your organization so you can effectively contain them at this point you're trying to stop the bleeding that containment effort becomes very important because if you miss something you actually run into situations where that asset can continue to infect new ones uh all of your your various containment eradication work tends to go out the window so making sure that everything is segmented either physically uh or logically

through some kind of uh control lists is is very important again we'll dig into those in just a few minutes uh and then once you have all of that isolated and contained you want to go ahead and eradicate the threat so attackers maintain some kind of persistence on systems that's just the way they they operate uh so you want to remove that persistence so that you can go ahead and recover safely without creating any kind of additional back doors that you may or may not be aware of so definitely spend the time to find all of the methods of persistence here you never know you think a device is ready to go you bring it back on the

network and something that missed these first two go as goes ahead and reinfects that one that you just brought online that definitely prolongs uh you know the the length of your incident uh once you've eradicated everything you now go through the recovery process bringing the assets back online uh so that the business can resume and then from there is lessons learned this is actually pretty important i do touch on this a little bit but i cannot overstate how valuable this piece is to the ir process as well so it might look like waterfall the idea being that you go through each of these steps in order in reality it usually plays out a little more like agile where you might

find something work to contain it early on and then carry on back through your identification process each organization is a little bit different and how they go about orchestrating this based on the assets affected the team members they have that's going to be unique to them but in this case you definitely want to at least follow these steps at a very conceptual level so that you're making sure you have understood the entire scope of the problem you've contained everything effectively you've removed any methods of persistence you've brought the assets back online and that's the business and then you go ahead and learn from it and implement any changes you don't rise to the occasion you fall

to your lowest level of preparation chris voss this is a he's a former fbi negotiator lead negotiator for fbi counter-terrorism i think it was he wrote a book never split the difference i highly recommend this book it's just really interesting reading has value uh not only if you are a hostage negotiator which i think there's probably none of us in the room but it actually has really interesting uh knowledge to gain from this highly recommend that book but it's absolutely the case when it comes to ir the level of preparation you and your teams have is the best that you can deliver at that time uh so the more prepared your team is the more effective your response is

in that case if you're using any tools to help you in the process if you have any scripts uh if there's any environments if you're doing any kind of forensics or capture environments or in the case of malware analysis your malware analysis environment you really want to make sure that those are operational and working before an incident happens uh in the midst of an incident is not the time to troubleshoot why a tool isn't working depending on what it does you may want to try and use something else if you can or you know have some clear documentation another thing that i found with a lot of incident response uh is communication you absolutely need those clear

communication pathways uh you want to have physical phone trees phone numbers uh because sharepoint goes down what if your sharepoint was one of the assets that got compromised and now it's offline if all of your documentation lives there and it's important for you to have that in an incident that needs to be available somewhere so in that case you want to make sure that you have some additional information available for your team so that if something does go down that they are relying on that isn't what hangs up the rest of your incident progress uh similarly if your attacker has breached your voip or email environments uh you definitely want to have some kind of

out-of-band communication right make sure that you have uh some additional phones uh or some separate phones uh i once worked in an organization they called it the bath phone it's just a small handful of uh flip phones that they used in the event the voip system went down so that the environment could stay online so that worked very well you definitely want to make sure you have some method of staying in touch with your incident response teams you also have to validate that any equipment is in good working order so if you are doing forensic imaging if you are doing um malware analysis or any other kind of technical involvement you want to make sure that is working as much as you can

beforehand you want to be able to restore that quickly if something goes down and especially in the context of malware analysis you do need to be able to restore fast in the event that you have infected your system because you will willingly infect your system for the purpose of analysis so i'll get to that in a second uh and have a clean shirt incidents take a long time uh make sure that hygiene is a thing uh it definitely doesn't take long for a conference room to gather a bit of a funk so you know just show up in clean clothes your colleagues will appreciate that so let's say we have an incident we've declared that you know based on

the indicators that we've seen and the alerts that we've observed or the behavior that we've observed that okay there has been intent to do harm here a system has been compromised it is now time for us to start working the identification process so with regards to malware there's definitely a few places you can look indicators of compromise will show up in your log data if you are monitoring traffic if you have um like a network tab or some kind of pcap going on some kind of packet capturing uh those files usually have some pretty interesting content in it if you can acquire it this this is a little bit rare in a lot of organizations unless they're

purposefully capturing that network monitoring it doesn't take long for this to get noisy and tough to sift through so this might not be available to you all the time but if you have a known infected system and you have the ability to capture packets and watch what's happening across the wire this can help you out quite a bit uh tons of malware does live in the registry specifically file list malware will actually load itself right into a windows registry almost in every step that it can so paying attention to registry integrity monitoring again this isn't not this isn't always a proactive approach that organizations have uh the overhead on it i imagine would be quite immense but in your analysis

environment you can actually leverage this quite well so the way that works is you have a registry monitoring tool you take a snapshot of it in a clean state you run your sample you take a second snapshot compare the difference and then at that point you now have some registry indicators that you can start to to sift through inside of the organization as a whole uh if you have uh file integrity monitoring running there may be some alerts that get generated if a specific file is trying to be accessed in which case you could monitor that tool for any of those uh file accesses if something is trying to write to a file same idea if it's

monitoring for that integrity changes you might be able to pick that up inside of your environment again film is is notoriously noisy uh there are no shortage of organizations that have it in place and it tends to alert on everything except what you're actually looking for so your mileage may vary if you have film installations but it's always worth the shot i mentioned antivirus earlier on if you do have endpoint av i don't think i've ever encountered a single organization that didn't have some kind of endpoint antivirus running on it so that's a good step that's definitely something valuable the big problem with antivirus is if if the signature is not in the threat intelligence database uh

and it doesn't catch the behavior as it's running you may not get an indicator from your endpoint tools that there's actually a problem on the device so you don't want to rely on it as much as you might think you'd want to because it can be bypassed we hear about av bypasses all the time uh and and that's just the reality of the situation we're in so leverage your av as much as you can but in this case consider it a bit of a trust but verify situation and of course any traffic monitoring so if you have again network network layer monitoring if you have any uh firewall logging going on any of that information all of that can help

you in the analysis process if you don't have any of that and you're in the middle of an incident with this analysis process you could potentially drive uh some logs that you should turn on and start to pay attention to which can definitely help out logging is is super helpful uh it's more helpful if it's in place before the infection so when i mention preparation one of the things you may want to consider doing if you don't already have some logging in place on some critical assets or some of your outbound connections where a malicious file would now have to pass through in order to call out so firewalls proxy proxy logs if you have them

definitely something is better than nothing just get something going if you can because that definitely makes your life a lot easier on the fly if you're in the middle of an incident and you do need to get some logging stood up uh there are options for you to do that but you'll want to have practice that before an incident happens i can tell you setting up logging uh in a hurry is a really difficult thing to do so um you know that just something to be careful of the the main goal the main outcome is identify as much of the scope of compromise as you possibly can the idea here is you want a tally of all

of the assets that have been hit by the sample so that you can properly contain them and start to work the incident so when we talk containment we are effectively removing the assets from the network you can physically disconnect these if they are physical assets those are becoming uh fewer and further between as we migrate to cloud but there are some out there so if you have uh desktop servers anything that you do have a physical access to you can just yank the cable and that should get you there you can segment them into their own quarantine vlan supremely common works really well and then it gives you the opportunity to keep all that data inside of one

specific area make it so that it can't connect outbound to any of your production or sensitive environments and then it gets you in a position where you can you know use some additional console connections to either get in and start to work the eradication steps or at least leave them segmented out of the network so that they're not causing you any more pain similarly if you have any internal firewalls or any layer 3 you can use acls so access control lists will prevent that connection from coming out the nice thing is with most firewalls they do log whenever an acl has been triggered so now you can actually see if a specific device has been compromised because it's trying

to phone out to a known c2 or c2 endpoint that you've identified through your analysis you definitely want to avoid powering down if you can reason being is not every organization wants to sign on for a full forensic analysis uh consultant usually it's somebody who comes in independently if you don't have it in house it becomes a very expensive engagement and if you don't have a lot of the really important artifacts lined up ahead of time it might just be a very expensive investigation that may not get you any further that said if there is a desire to have forensics come on site and do some work definitely make sure those systems are up and running you don't want to power

them down there's a lot of excellent volatile data that lives in ram and you more than likely will lose it as soon as you power it down and then any of the really interesting infections uh or some of the maybe persistent connections that would be there get lost so avoid powering these down if at all possible ultimately what you want from containment you want all of your infected systems to be adequately socially distanced from all of your non-affected systems you don't want to create a bigger problem than you have to solve it definitely compounds the issue if you have your infected assets on the same network as your production ones and they're now reinfecting these systems

that makes for a really long night however disconnecting these boxes will almost entirely cause an outage of services it's very rare that they wouldn't unless it's somehow a system that isn't necessarily used for anything of any importance uh so you will almost always need some kind of sign off from data ownership or the business itself uh this is actually this can be difficult to get when you are in the throes of an incident uh i i was once working an incident where it was three o'clock on a wednesday and we couldn't get sign off from the business to to take down the server uh because it was still working so they said no we we just want to keep this

going uh we'll come back to this a little later that's a very real conversation that you may have and it might be frustrating but the business will win every time just be ready for that not a bad thing that's just the way it is so once we've gone ahead and contained everything it's time to get rid of any of the persistence and threat actor presence so there's a few ways that attackers specifically in a malware event will maintain compromise scheduled tasks and cron jobs that's often overlooked uh many administrators will look past the scheduled tasks uh there's tons of them that do show up in windows by default so it is important to have at least an

understanding of which ones are authorized which ones are not so that you can go ahead and find them obviously attackers with password dumping capabilities will uh try and compromise those credentials assume whatever system they were on that that sam database is probably the credentials within it are probably compromised so definitely make it a part of your containment eradication efforts to disable those accounts or reset those passwords so that whoever is authorized to actually hold those accounts can come back and use it registry modifications as i mentioned uh fileless malware lives in the registry lots of malicious samples will also dig into the registry as a means of persistence so just because you have cleared the file

and stopped the traffic doesn't necessarily mean that you've actually eradicated the threat there might be some registry hooks in there that now come back uh when the system reboots there's your malware all over again uh similarly with the account credential theft if they have administrator level credentials that they've compromised your attacker might create some new ones whether they do it uh by hand or whether they do it through a malicious sample that they now pass back through c2 whatever that looks like new accounts may be created and they definitely look for that administrative level so you want to go through all of your admin accounts at the very least as part of your containment and

eradication exercises to make sure that everyone that's there is supposed to be there so authorized by the business and that the levels of permissions are are what they're expected any ones that aren't what you're expecting or if it's uh just kind of a suspicious one your best bet is to disable it uh and then wait until somebody comes either saying my account doesn't work or follow up because administrator accounts can run roughshod over your network one of the most common methods that you'll find uh malware leverages persistence is simply dropping files on the system uh some of the example of the examples i'm about to show you they drop files they reach out to c2 server

it downloads the payload onto the workstation itself for the endpoint and then leaves it there in a file buried somewhere that nobody's going to look for and unless you actually know what to look for by name and by file path finding those on your own can be notoriously difficult even in very very small environments it really doesn't take much to hide a file inside of an operating system similarly you'll find some malware creates a service so one of the things you should be doing as part of your preparation is baselining services on your environment if you can that definitely takes a lot of effort administratively so you know maybe hire an intern or something if you're trying

to do it on a budget but you definitely want to go into these systems and understand what services are supposed to be there uh those applications and and what they're doing so that if a new service spins up all of a sudden you know that that's probably an infected system and it's time for you to start working that process alternate data streams for anybody who hasn't seen these these are processes that actually contain some hidden data behind them inside the process itself so finding these alternate data streams i have to look the syntax up every single time i rarely ever do it but you would be surprised at how many systems uh benign processes on your system

are running some kind of alternate data stream one of the things you want to look for inside of an endpoint is these alternate data streams is there any additional calling out that you might have missed is there anything that might be hidden inside of the payload itself whatever you can do to find this is definitely in your best interest you want to be thorough this is an important step because if you miss something you bring it back online your incident starts all the way over again and in a lot of cases you might accidentally tip off your attacker uh that you're on to them if you've gotten this far so they'll now either ramp up their

attack or change their methods and then you're pretty much back to square one again so you definitely want to take some time and validate that you caught everything at this stage if you can and recovery now it's time to bring the business back online so we've we've gone through and identified all of the systems that have been compromised we've contained them into their own separate environment we've found all of the er the threat actors methods of persistence we've cleared them out now it's time to bring the servers systems and points what have you back online so if the malware that you were experiencing in your environment made use of a vulnerability you want to patch

that if you can make sure you patch it before you bring it online and it isn't enough just to install the patch you need to validate that the patch is effective a few ways you can do this you can vulnerability scan with an automated tool like nessus inside your environment that will go ahead and usually tell you if there's a volume present or something else you need to address but if you don't have anything like that uh you may need to do some digging at the endpoint level just to make sure that the the vulnerable condition has been fixed before you do anything else uh if you have any missing data or compromised data or encrypted data uh

go ahead and reload that back into these systems now as long as your backups are solid and there was actually one incident i remember where the malware was backed up with the backups for a considerable amount of time so much so that that resulted in quite a bit of data loss so you definitely want to as part of the preparation efforts you want to test your system backups regularly make sure that they are intact that they are known secure because if you have to restore that data you want to do it not only as quickly as possible but as effectively as possible because large-scale data loss depending on the scope of that data and what you're required to to disclose

could be bad so you definitely want to make sure as part of your prep your backups are solid and malware free any services that you've turned off whether it be for troubleshooting or investigation turn those back on make sure you're paying close attention to application dependencies some services may need to be started up in a specific order so if you have a business critical app that needs to run in a specific way make sure it's documented so that all of your administrators can go ahead turn this back up nice and quick and bring the business back online with any new setups they have to do if you aren't backing up configurations to critical systems it might be a good idea to start doing

that at some point soon some configurations are long and unwieldy you want to be able to bring those back very quickly as well malware can often get into some of these files and either create backdoors through that depending on the permissions they have available to them or simply just corrupt them entirely so you want to make sure that your configurations can come back as intended otherwise if you do it incorrectly you might introduce some new vulnerabilities that all of a sudden now have to get dealt with uh if somebody gets there first any security tools av is another common one that shows up here av might be missing signatures you might have an agent that's out of date you want to

make sure that that's up to date before the system comes back online if you have any security controls in your environment you want to make sure that those survived as well because a lot of malware at least at the endpoint level will try and stop those or corrupt them in some capacity so if you're using an endpoint av and the agent is out of date make sure you you go ahead and install that if you can but again paying very close attention to dependencies because there have been stories of malware endpoint updates where it broke something critical on the system i'm sure everybody has a story about that at some point last and certainly not least

whoever owns the data in the environment whatever business unit is in charge of that sorry brief cat based interruption uh you want to acquire sign off that stipulates that the data is intact the system is properly configured everything is online as expected ultimately the data ownership is the one that signs off on this uh so somebody with that authority can go ahead and do that once they've had a chance to review it so don't skip on this step this is important it might seem onerous and whoever might have to pick up the phone and validate that in the middle of the night might not be overly pleased uh you're you're sort of get out of jail

free card is i just want to make sure the business is working before business hours tomorrow morning that's a really hard one to get upset about uh if anybody does get upset with you about that they can fight me in in real life i'll take that on for you so lessons learned uh this is like i said a very important meeting this is where some of the new business decisions happen so you want to make sure you attend this if you work the incident if you can uh the way this works if you were the incident responder presenting what happened you just want to present the facts of the incident let the facts stand out on their own you don't

necessarily want to throw anyone or any process or anything under the bus that's definitely not a good thing to to sort of paraphrase you know happens right like things are going to happen in in any corporate environment malware happens even to the most well-intentioned people so uh just present the facts of what happened let them speak for themselves you want to identify any of the affected assets reason being is somebody in the seats of management inside of your organization might not be aware of uh how involved this incident may have been so if the assets that were affected were really high value or of a particular critical nature that can sometimes get really good attention to all of a sudden find budget

for your new changes that you're trying to propose if you have evidence confirming that compromise make sure it gets in there screenshots and visual evidence are always good the recommendations that is such an important component as well quite often overlooked but really really critical reason being is one of the questions you're going to get in these meetings is how do we avoid this next time you want to preempt that entirely by discussing the recommendations create a business case for each one and specifically evaluate the cost of control versus the cost of doing nothing if time permits uh reason being is establishing that business case will not only help you get budget in a place where you

really didn't think it would show up but it also helps to understand the value of a security program inside of an organization so that's the ir process in a shell let's talk a little about malware analysis so with good malware analysis you can usually uncover some kind of indicator of compromise that you can use to accelerate the identification process so remember when i mentioned that if you're trying to just mine through a bunch of data or you're trying to look through you know your av that can take quite a bit of time if you're leveraging some malware analysis efforts you actually have this ability to sort of expedite that process in a lot of cases

you might be able to find some level of c2 identification so some kind of c2 endpoint that you can use to start hunting down affected assets so the reason why i mentioned things like proxy logs and firewall logs if anything had to transit through those assets at any point you now have the the necessary information to find out if something is malicious uh or if it has been affected if there's any persistence mechanisms during your analysis effort you might be able to uncover those so watching your analysis machine for all of the usual places your uh your tasks uh any crown jobs if you're doing anything in unix environments services that gets spun up processes that get created

all of that information can start to show up in your analysis pretty quickly and it will often work faster than your antivirus tools can so again saving lots of time more effective ir that's the goal here uh and incidentally if you uh some information stealing malware depending on how well you can reverse engineer this you might be in a position to see the type of data that it's targeting which now becomes a really excellent phone call to anybody who has to go ahead and disclose this so for anybody in a pci space you might have breach disclosure requirements with the various card brands depending on where your business operates you might have mandatory breach disco

disclosure laws uh depending on the type of data that's compromised you might have mandatory breach disclosure as well so having an understanding of what data was sought and potentially compromised helps you start to navigate that process nice and early uh a few drawbacks to malware analysis uh it can be time consuming the the long and short of the malware that's out there is designed to trick you malware authors will write all kinds of of little red herrings and uh rabbit holes for you to fall down purposefully to waste your time some of them might actually go so far as to further infect your system try to break out of your analysis environment so you have to be careful it is time

consuming it can be very easy to be led astray malware analysis is a very niche skill it doesn't come with a lot of sock offerings and it doesn't necessarily find itself inside a lot of organizations that said it is something you can cultivate on your own you can practice it internally in your own environment uh with very very specific segmentation and setup environments which i'll get to in just a second and of course nobody ever wants to make the incident worse if you are the malware analyst and you haven't taken steps to segment your environment properly you are about to make the incident just a whole lot worse so you definitely want to avoid getting

wrapped up in that and becoming your own liability if you can so your analysis environment uh as much as segmentation is an option inside of an organization for whatever reason it is not an option for malware analysis you are obligated to segment your environment you can do this a few ways you can do it through some vlan segmentation if you have some purpose-built infrastructure if you have some spare hosts hanging around that you can afford to air gap and consoling with a separate monitoring keyboard that's also an option uh the host only route if you're running vms on a laptop works just fine as well uh just keeping in mind that the endpoint host that you are working with

or sorry your laptop you want to make sure that there's no business critical data on that you want to make sure that it isn't likely to cause further infection you want to disconnect that as much as you can or any of the above if you have some spare infrastructure and uh somebody who's willing to set it up for you in a specific manner you can actually go ahead and set up a pretty awesome malware lab without a ton of extra work so if you have an isolated network that you can leverage inside of your organization that definitely opens up a few doors for you for sure you need to be able to quickly restore your environment so what i mean by that

is if you will infect systems this is the deal you are purposefully infecting endpoints that you control that you're capturing the events specifically to find whether or not there has been a problem uh you definitely want to make sure that you can quickly restore that reason being is if you can't and you have to rebuild it you're now taking more time at a point in the incident where you probably can't afford to so quick and easy restoration definitely is good snapshots for virtual machines make this way way easy so feel free to use that that that works really well and make sure that all of your tools on your systems are set up there's a few dependencies for a lot of

tools uh and i'm i'm about to get to that with my preferred setup but all of your tools should be up and running uh setup configured tested validated before your samples even go on there for testing so make sure that your snapshots are up to date test that regularly and away you go so my preferred configuration it's not the only way to do it but this is the way i like to do it it helps keep things mobile so if i have to go to a remote site i can still do this work if i have to stay in some other remote situation it still works i can take it with me wherever i want i

can load it into my lab environment very easily i can build it out in my lab environment any of that information all that's good so my preferred configuration is i like a small amount of purpose build infrastructure if i'm mobile if i have to go to a client site i'll bring it on a laptop that laptop has vmware configured and installed ready to go my operating system that hosts everything is fully patched and up-to-date shields up windows firewall is up and running data that's on it is perfectly disposable whatever's on it i could afford to just leave on the ground and it wouldn't be a problem i treat that underlying host as something i could

potentially throw away if i ever needed to inside of those uh vm environments you want one windows based vm for windows based malware as you might imagine there are no shortage of tools that run on windows for malware analysis lots of them open source uh really excellent options out there for sure you want a linux based vm for some of that analysis and some of that traffic handling so i use remnux remnox does work very well it's free it's built on a live cd so you can just quickly reboot it and it comes back to a bear configuration there's lots of excellent services that run on it that you can use to impersonate c2 handling so that your malware has

somewhere to talk to you and then you can see some of that chain go back and forth so with that configuration your windows vm should only be able to communicate to your uh linux vm so the linux vm is the one that isn't uh accepting all of the traffic requests all of the the data and just sending it back to your infected host whatever it's supposed to look like uh so remnox has a tool on it called inet sim it's an internet simulator it simulates a bunch of different protocols uh ports services that you can now use so if you have a malware sample that runs over irc for instance you can set up an irc

listener for any of the commands that are going to be coming across from your infected machine into your c2 handling it doesn't pass back what the malware might be looking for unless you know what to look for but for an initial triage and investigation effort it it works really really well you also get nice bits of logging so you can see if you're using uh if your malware is using http or dns you can see those logs coming across so if it's actually requesting something different uh or if that endpoint is encrypted or obfuscated that would get unpacked when the sample runs you'd be able to see that on your listener so that has uh immense value putting that

vm into that listen mode you make it so that your windows vm can only communicate to your remnux vm not anywhere else uh so if you have old hardware hanging around it can be done pretty easily i mean vmware practically runs on anything nowadays virtualbox uh will also take up this this work nice and easily but i did find that the networking connections getting some of those network connections to talk back and forth as intended did seem to fall down a bit more with virtualbox than it did through vmware uh and if you're using hyper-v uh i welcome all 10 of you who might be doing that that's that's totally cool too whatever virtualization software you

choose to use is entirely fine just make sure that you have the ability to quickly restore your environment effectively segmented from the rest of your network running on hosts that you really don't care uh if they get infected or not that you can also quickly restore again you're handling live malware so you have to treat it as though it's it's live malware your windows configuration there's a few extra steps you have to make sure of to get everything talking uh you need to configure your ip and dns settings inside of your network adapter make it so that the next hop is just your remnux machine nothing else so your default gateway is your rem nox

host and your dns is your remnants host as well uh those two things will now serve up the requests back to you uh in in that case yes yeah too funny steve porter good to see you um the the that configuration that connection uh is super important because your machine needs to think it's connected to the internet a lot of malware looks for disconnected uh network interface cards or the inability to reach out to something and might actually shut itself down as a precautionary measure if you have your remnux machine up and running it is serving up requests you can actually go ahead and replicate some of this and get that working the the tools that you need to have uh

it everybody's going to be a little different this is just where it starts to come from uh practice and just feeling away your your own systems here at a minimum you need some kind of file header analysis tool you need to be able to look at the pe headers uh p explorer cff explorer p id there's lots of them out there you need a strings analysis tool for finding initial strings so one of the the first bits of evidence i'm going to show you is just simply strings what strings do we see in the clear text even if malware is packed and obfuscated you still get some clear text information if there is an encryption uh scenario

where it either has to reach out for a key exchange or reach out for some command to go ahead and encrypt you might actually get an address that it has to go through first uh super super helpful you'll need some hex editors just to go ahead and validate any of the hex information that might be in there uh guidra is definitely a free and useful tool and then uh you have tcp done holy cow i have five minutes oh my bad okay well we're gonna speed through this uh so this is what it looks like when your connections are up and running google actually didn't respond with this it just showed up in my dns logs here

so that's what that looks like any questions on that i'm happy to answer never ever connect your systems to the internet unless you really need to one of the things that you want to be careful of is if you do connect it to the internet you might go ahead and make things worse but there might be a specific point where you you really just want to get that file that comes back down from the attacker this is at your discretion and with an over abundance of caution if you're never sure that you should connect to the internet don't uh make sure that your java and python installs are up and running giger specifically runs on java other tools

might live on python other tools might have some other capabilities any of those dependencies need to be in and running and up to date before you go ahead and put your samples on trying to connect to the internet to download these binaries after the fact definitely is going to be a headache you don't need make sure you're snapshotting your systems before you infect them or save your backup somewhere and then test that restore put in all your installs snapshot revert back to snapshot confirm everything still works and then carry on you want a plan what i mentioned before is malware authors will try and confuse you they will intentionally create red herrings for you to get lost into

have a plan jot down those desired outcomes stick to it but also know when it's okay i have enough it's time to pull the rip cord and we're out of here so you definitely want to find some very specific indicators of compromise some ones that really make your life easier c2 endpoints especially if you're monitoring firewall traffic files dropped if you're paying attention to any servers or desktops and any kind of file infections that might happen if you do have the ability to find the registry entries and query them effectively throughout the rest of your organization that's a good way to do it otherwise uh that can definitely be a bit of a hill to climb so

judge accordingly based on the time you have and also look for very uh very specific processes or services that get created some malware will reach out get some commands create a new service or a new process that's different from the one that you've seen you want to be able to catch that so that definitely takes some time the more that you can dig in and reverse engineer the more specific your search has become the less time you spend here but you need to identify a stopping point as i mentioned before tons of malware is designed to be confusing and misleading the actors will take great pains to encrypt or obfuscate the the really important elements of it uh

the really infectious pieces of it so if you can't unpack those for static analysis uh then you just move on that's that's your best way to do it definitely confirm segmentation before you get digging into it again if you cause problems for your incident it looks bad for you so static analysis you're not actually analyzing the malware with execution it's just looking around in a static fashion you can find useful artifacts very early on usually some unique strings some initial library imports any file droppers if there's there and anything of anything else worth noting malformed user agent headers you're going to see this uh quite a bit because if you have a misspelled user agent header mozilla with 1l

instead of 2 that's a unique signature you can start to pivot on so definitely start to keep going with that it the static analysis takes time but it's worth it i promise so here's an example this is uh this is strings analysis this is one tool with one command line that i just tailed out to an endpoint and it actually found very quickly that it does some imports we have a file drop here and then we see that it goes out to url files to download and thanks to practical malware analysis for the really excellent malware uh this is where it went right when we open this up in a pe header uh toolset we can see very clearly we

have an executable in the resources header this uh if you haven't seen it before all dot exes have the this program cannot be run in dos mode finding this at the very top of your executable totally normal finding it in the middle of your executable highly suspect and in fact this lives inside of the resources section so it's actually a program within a program this is more than likely uh the malicious program itself static analysis the whole step each of these three steps maybe five minutes at the most it really was very quick so now we have enough information we can go to our firewall teams and say i want to know if anybody made

poster get requests to this address now we can also go to the server teams and say i want to know if this executable is anywhere in the environment anything after the fact i i just need to know if there's an updater.exe i'm also looking for this winup.exe go see if any of that lives anywhere by the way here's a potential file path you can start to look through those are three solid indicators that you can now fare it out to the rest of your teams while you continue to dig in with dynamic analysis in this case you are executing the payload on your systems you need to have network packet monitoring registry change monitoring process monitoring and log monitoring

you want to see as much of this happen as possible similarly if you run into a packed and obfuscated binary you may actually hit the limit of what you can unpack in that time dynamic analysis unpacks the sample in in memory allows it to run completely and allows you to observe that traffic so when we actually have all of our listeners up and running this is the remnux side so it made a request to that endpoint and we and it was talking just to my remnux box makes perfect sense this is the request so if we're looking at proxy logs we want to see get requests of anybody who went to this that means they probably got

this file here's that file in the path we saw the file get written to disk here's the other one we were looking for that w update mgr there it is in our syswow 64 and when we actually execute that it pops up this little screen right here so uh this these are all rock solid indicators that you can use to confirm if an endpoint has been infected all of your firewall logs if you are paying attention to those or any web proxy logs should show a connection that went out to this those devices are known infected treat them as infected quarantine them accordingly go ahead restore whatever you have to do after that point this now gives you some opportunities to

start digging in we see one method of persistence we see two methods of persistence and from there now we can start to dig in even further with registry monitoring if we have the time uh additional file integrity monitoring you can actually start to work this file up in the same method uh the same functions and process to see if this actually has any further indicators that you can use if this gets deployed at any time now you have more to work with the really nice thing is anybody who's pestering you for updates now has enough to keep them busy for a while which leads me to reporting findings note down your findings pay attention to

how they came up when you ran them so was it under certain conditions was it when you ran the file as administrator or as a general user make sure you're validating that and paying close attention to it as soon as you have findings escalate them to the right teams anything network based needs to go to network management server needs to go to server management etc do this fast don't don't wait don't sit on this as a report treat this as critical information that has to get there now because effective communication is vital you need to describe what you found and why this matters to somebody who's paying attention to it have that ask hey go look for this file

if you see it quarantine this host it's known infected and let's carry on don't sit on good information just because you're worried about putting your hand up so a few things to consider static and dynamic analysis absolutely mandatory no way around it if you can't figure out where to start or if your static analysis is coming up short move to dynamic analysis the way it works is you start big with your filters give me all of the events and you start to whittle them down until you get to where you're supposed to go so you might have to restore and reload filters just to see what that is that takes practice but it happens pretty quick

uh from start to finish for all those artifacts i think was about 20 minutes in my living room so really fast way faster than any endpoint av is going to find probably way more effective too but time is limited don't go so fast that you're going to miss something but you have to make sure that you you're moving because incidents take a long time to clean up usually many many many hours many nights sometimes weeks and months depending on the scope of the incident and what was affected so go fast but not so fast that you miss anything always be segmented always always always make sure that your environment is well isolated from the rest of the network

that you cannot cause any further issues okay so sorry to have to speed through the last half of that uh i guess i like to hear myself talk that's a first so i'm going to open it up to questions i know that there's later sessions if you have any and you want to go to those and you want to circle back with me i will be online for the majority of the day uh i do have the ir table top talk coming uh so we are playing the game that i did build uh that does replicate an incident in all of its uh interesting and hilarious glory you can come and test some of this out i think

we might have a malware outbreak uh later on this afternoon so good opportunity to test some of that uh love hearing from everybody i'm gonna leave this now thank you so much for coming wonderful to see you go b-sides have a good rest of your con everybody thank you

oh yeah i can uh i can get slides and stuff up uh i don't know if there's a like a talk repository or uh where the recordings are going to be but yeah wherever that winds up besides organizers if you just let me know and i'll make sure it gets on there

yeah thanks everybody it's good having you like i said i'll stick around if anybody has any any questions

it also looks like uh michael is giving his talk here in a few minutes on uh mental health in information security uh yeah plus one if you that that's a struggle that everybody has i don't care who you are uh go to that talk if you have any questions about it that's gonna be good actually i think i'm gonna attend that one as well that'll be really good yes i will make sure slides uh get somewhere i don't know where that looks like yet if anybody from b-sides has any insight on where that might be let me know if not just hit me up on linkedin or something i'll make sure you get them

yeah thanks everyone uh yes jeremy it's the ir tabletop workshop so uh i take the first few minutes i describe tabletops uh why they're important i give you a few minutes to go get a coffee and decide which team you want to play on and then we play for the rest of the [Music]

afternoon [Music] uh so understanding assembly is mandatory from our analysis good question jason uh no it's not necessarily mandatory to do it in this capacity if you want to just be good at it for the sake of incident response for shortening the amount of time you spend on an ir situation in your organization understanding assembly probably has a few diminishing returns that said if you are looking to do more uh along the lines of reverse engineering or more in-depth malware analysis uh you definitely do need to understand some semblance of assembly uh that said ghidra has the ability to take that assembly and derive it back to original source code pretty quickly so whether that actually negates the need

for assembly or not is probably open to interpretation i found that the code the the decompiling there that came from the assembly was pretty accurate within ghidra so that that suited me just fine but it's personal preference

yeah it works

yeah is there anything in calgary that you would recommend uh so shameless plug for the malware analysis course i used to teach at state uh they do have a malware analysis class inside the program uh where you do cover a considerable amount of ground in this space more from a reverse engineering person perspective rather than an ir perspective uh but that's definitely the way to do it other than that um nothing in calgary specifically uh but tyler hudak has a course on pluralsight and he actually taught me malware analysis when i was at derby count a few years ago so that is a really excellent option as well pluralsight most organizations have free or nearly free deals for

pluralsight so they have good courses on their malware analysis being one of them other than that uh the practical malware analysis book uh really really highly recommend it comes with tons of companion malware excellent labs tons of excellent learning it's about 60 bucks on amazon it's one of the alien autopsy that's a good book so if you're interested in digging in further jeremy uh pick that one up for sure which tools would you recommend for malware analysis giger ida pro and yeah good question uh tools are subjective what you choose to use as your tool obviously is up to you it's whatever you're comfortable with i personally prefer ghidra for a few reasons one it's free

so you don't need to come up with a really compelling business case to use it whereas with ida pro you need a very compelling business case to use it because it is quite expensive the interface i find in ghidra is just a little bit easier to navigate i found getting around it a lot easier i did like the auto decompilation kind of on the side of it so you have all of the disassembled codes showing up in the middle of your view and then just off to the right you have all of the code that comes as a result of it and so that that really made it nice to sort of trace instructions figure out

what functions were doing uh and work from there from a reverse engineering standpoint um when it comes to tools you want to have more than than just the one you don't want to rely on a single tool because tools are fallible uh you can have two answers for the exact same sample of malware from two different tools so more the majority of the time you spend is just validating that your results were as expected or as intended um so as far as yeah disassembling and decompilers go gidra for sure the free nature of it and the user friendliness is too much to ignore yeah you're welcome jeremy anytime

good stuff uh well if there are no other questions uh i think we can drop off the track here i am still online through b-sides you can just hit me up through your dms uh whatever else uh and i think i'll take a cruise through some of the other cool stuff that's going on here check out some talks really awesome lineup thanks again so much for uh all of the attendance the b-sides organizers again thank you so much i know how much work goes into this and i cannot thank you enough for all of the effort that went into it so uh great stuff enjoy the con everybody thanks so much for coming