Came for the Ransomware, Stayed for the APT - John Dwyer

all right what's up nerds okay thank you first of all for coming out and listen to me talk to you for the next 40 minutes or go i don't know if you guys saw justin's talk before this but justin you are the most casually hilarious person i've ever met my life and i don't know if your memes got enough credit so allow me to say it for everyone else that's thinking it so uh i'm going to try to jump right into this because i've been described as verbose and i don't want to take up the rest of your day so my name is john dwyer and i'm the head of research for ibm xfors if you're not familiar with xforce we

are the consulting arm of ibm security so we offer defensive offensive and threat intelligence services one of those defensive services is incident response which is the team that i came up through uh over the years and now my job is the head of research is kind of two two areas one of it's kind of like a devopsy role where i'm overseeing technol technology developments to make sure all those services are operating as efficiently as effectively as possible the other side of it is to kind of highlight all the interesting findings in things that we've had over all these different engagements and make sure we're sharing those with the community i try to do a good job of

releasing like kind of tactical things on my social would be like detection opportunities or interesting findings on malware just because we kind of want to be you know operating appropriately in the community try to burn as much of the ttps as possible my background in a word is chaotic i would say so i i started out my professional career as a laborer went through help desk actually i don't know dugus are you out there right now he might have my first boss ever who gave me a chance is actually in the audience today i ran i ran into him um but i've done a whole bunch of different things over and i've gone up through it

and kind of pivoted into security and then into research and uh at first it was kind of like am i going to be too much of a generalist and then over the years it's kind of turned into like a slumdog millionaire situation where things would pop up like when proxy logon and proxy shell happened and i was thinking back when i used to be an exchange admin like oh i get that i understand how kaz's work and i understand how they interface with iis so i've been all over the place i have up there that my superpower is okay with being dumb mostly for two reasons number one is that i never think i should take myself too

seriously and you know it's one of those if you can't laugh at yourself who can you laugh at situation number two i try to bring this up as much as i can is that i wouldn't call myself the most gifted person ever uh and when i was in third grade my third grade teacher told my mom that i was probably going to struggle in life because i had such a hard time learning so that's always kind of stuck with me and it always seemed like i had to work twice as hard to accomplish anything in my life and that persisted throughout my career uh great story those of you who know me it's like you're going to get a lot of tangential

stories as we go through this um the first time i got like a sis admin job i was like i'm making it like i'm doing the thing and i was really proud of myself guys i went to that first team meeting and it was like these dudes are talking a completely different language i had no idea what anyone's saying it so i'm like feverishly taking notes about stuff that i'm gonna have to look up after work because i don't understand and it was embarrassing and it was like disheartening to go through that process and that persisted as i went through every single evolution of my career and then i as i got older i started to realize that

if i'm continually putting myself in the position where i'm the dumbest person in the room that means i'm choosing adversity i'm choosing to grow rather than be complacent and i look back on that now with hindsight being very old at this point and i'm very grateful for that time but if you go out on twitter or linkedin like it just seems like everyone is crushing it like everyone's putting out amazing research and all these great tools and it's really hard if you're just starting out to not slide into those negative thoughts about yourself or about your career so i i like to bring this up anytime i have a group of people just in case there's someone out there

who's just starting out or or they have that imposter syndrome to say that you know you're not alone we all kind of feel that way and if you put the work in you can accomplish some great things with that said let me get let me tell you about the one of the coolest irs that i've ever worked like i said it came up through the ir team and like if there's any responders in the crowd everyone knows shiz goes down on a friday right so people call on the friday a client calls in like many other ir firms we offer a hotline they call in to declare an incident so they say in one of our one of our domains we've

been hit by a massive ransomware attack and they're pretty large retail organization nationwide three main areas of their business but they got shops all over the place and at this point i mean this was the golden age of ransomware it was like every three days we were getting one of these calls so it was like there's another one here's another one so they call and they said about 200 systems they're offline they call in they you know it's actually kind of cool the av vendor they're not responders but they did some analysis found the domain admin account and some an ip address they blocked it and they called in they're like hey we just want

to run past this is it cool and we're like no dog this ain't cool like there's a lot that we need to scope right so part of being a good responder if anyone's trying to get into the industry is you know sometimes you got to save the client from themselves right so we talk about you can't move forward right now this hasn't obviously been scoped as well as it needs to be we need to increase our visibility the only tooling that they had right now was av so we go through the process of the ir and like i said we've been doing these a lot so they said all of our files have a dot ryk

immediately we know that's rio grande where and at the time that was the hottest one so we got that all the time also at that time we understood that the emote trickbot empire ps exec ransomware we knew that that was a thing and we did that legitimately i think i ran the stats on it was every three days for seven months that we responded to more than what we considered as a major ransomware incident was more than 200 systems so to put that like this was happening a lot one of the first things pro tip when you do an ir always ask for a security telemetry always ask for it there's gold in those av logs

and while we're deploying our tech we're collecting that data that all takes time but you can usually find something that's going to point you in the right direction in those av logs this was yet another example in fact a couple weeks ago we put out a research paper analyzing why are ransomware attacks happening faster than they did before which was a cool topic it took a lot of time but one of the more interesting findings in that that i think got glossed over by the media was year over year there is more evidence in security telemetry now before ransomware was deployed than there was in 2019 the start of what we would call the golden age so that means

that people are a investing in detection tech and b the detection tech is actually working better than it used to and there's usually something in there that could prevent you from having a crisis so make sure you're collecting that make sure someone's looking at it first of all and then if you're if you're an ir person make sure you collect that the other point i want to point out here is you can see that the older av alerts responders are going to know what i'm talking about here so older av alerts show someone trying to dump lsas show cobalt strike show them interpreter now what is weird about our timeline here and and bear with me here i'm not going to

try to sell you anything but why i think every organization should be interfacing with an ir team in some regard an ir consulting team if you think about this ir firms it doesn't have to be again it doesn't have to be export it's great if it is but if it's not no big deal but ir firms are one of the only disciplines in cyber security where you can can continually observe an adversary be successful right there's a lot of smart people and the mdr mssp avs all these they're putting out fantastic work but it's in their best interest to start to stop the adversary before they're able to complete their objective with ir especially with ransomware the detection

point is typically the ransomware itself right so that means the adversary is able to go from initial access to impact and ir teams are able to reconstruct that story and they're in a unique position to say what is anomalous and what is not from an adversary operations point of view and we talk about strategically how we need to operate now and in security is moving out of indicator driven detection strategies and moving into how we know adversaries obtain their goals and objectives so this bottom thing here is incredibly weird because we know from doing all of these rare ransomware investigations that trick bot should be the first piece of evidence it should not be some other c2 framework

that is generally associated with humor human operated activity so like they say back home that's wicked weird we're continually bugging the client about this again being an ir person one of the best things you can do is protect the client from themselves but understandably they want to get operations up as soon as possible we know how to do the trickbot empire ps exec we know to do that investigation we find all the domain admin accounts that were compromised find all the lateral movement find where they stole data buttoned up three days we're still asking about this system because it's only one system that's hanging out there that has this av alert that is the only thing there's two av

alerts out of a sea of mess that we're bothering the client about but they want to go ahead and go with recovery so literally typing the report out saying with a big section that says dude you might be pop still we're not going to sign off as this being a remediated incident you really should look at this the it admin texts me he's like oh i got that system dude do you want an image do you want edr deployed or do you want us to run your tools on it i say yes do it all right and immediately what we find out is that the same system that had meterpreter and cobalt actually also had the earliest evidence

of trickbot it also shows that trickbot was introduced through a meterpreter session that's odd right we know how trickbot goes right it's spam caming or downloaded through email emote that's incredibly awkward and weird to see that data point the other thing that we see is that the user account that associated with those alerts is from a whole other domain it's from the hq dom well hq none of these domain names are real but you know what i mean so it's from the the root so the hq domain was the the forced root of this domain and it was 37 days before ransomware happened so we have to expand scope right so we call hq and we say

you know houston we have a problem kind of deal like we need to start ex uh expanding our visibility and they're like well we didn't get ransomware so like we don't have a problem like dawg you got a problem right because this is this is abnormal but you know they also didn't get ransomware which was weird at the same time so we're moving forward expanding scope trust the process collect the data look at the data what do we find out is that there's cobalt strike within that infrastructure 96 days before ransomware now trick bought to react on our research is the longest life span of all ransomware attacks we've seen them prese persist up to three months at

the max 96 days is a behavioral anomaly based on how we know these attacks happen so 96 days that they're in this environment they have cobalt strike then we find out that that account that was used to pivot to west is an enterprise admin weird right and then we ask even more that enterprise admin account the security and i t teams have no idea who's associated with assumption the adversary created it for us you know a few guys aren't windows people and and if you get enterprise admin you effectively own the entire forest right so you've owned the whole thing but you didn't deploy ransomware in hq right hq is the largest domain within the entire environment it's got the most

critical systems if you wanted to create leverage why didn't you deploy ransomware within the hq domain incredibly weird right none of this makes sense multi-domain ransomware attacks i don't know if you guys know this they they do happen they're not as not as frequent but they do happen but typically it is a you know one domain gets popped they'll get enterprise admin and then they'll blast it out that same day to everyone this was way different there's the uh that event log up the top as you see that's the cobalt strike beacon likely an smb beacon if you guys aren't collecting tangent again that's system event log 7045 if you guys aren't collecting that building detections around that

write it down put it in your system tomorrow start doing keyword searches on it if i if i was a detection person with no budget or if i was a responder and i can only do one thing it would be to collect that log it is that valuable in 2020 we put out a paper about detection opportunities for adversaries in 90 of all irs that we work globally we found evidence of the adversary in that log and that's built upon how things like interpreter metasploit cobalt strike any one of them covenant all those c2 frameworks how they do things like privileged escalation or lateral movement they all do things like by creating a new service right so

fundamentally it makes sense because it works right it's the same reason why people use scheduled tasks all the time because it works collect that event log build detections off of it we're collecting our data going through two weird observations happen next first is that we're only finding evidence of the adversary within the server infrastructure nothing in the workstations nothing in the dmzs the only thing that we find activity was in windows core and internal infrastructure server infrastructure so that's weird because we can't identify what did they would they come in through hq why can't we find them did they go to west go to hq and back that's stupid that doesn't make any sense and the other thing we do

is we get a yara hit for m.exe and it gets a yara hit for mimi cats we pull that back now on this is on a domain controller and on this domain controller was an av and it was a real av it's not like bob's av that you download off the internet it's like something that's going to many cats on disk every av is going to hit that right like that's baseline we pull that back and the only thing that hits is our yara signature we run it through our sandbox the only thing that hits us on our yard we run it through our all of our avs that we have nothing hits toss it over to the reverse engineers

and we find out that it is a completely rewrote version of mimikats in fact the whole logon passwords module within mimikatz is completely obfuscated and rewritten and it extracts a go binary and that's how it ex gets the creds from windows holy right like that's way beyond the capabilities of most ransomware adversaries picking up the phone i'm calling people and i'm like we got something going on here this doesn't make sense we're going to have to do round-the-clock monitoring on this one the other thing we pull back is that sec url chk file start looking at that it's what's on the left here we start digging through that data and we find out that it is an arp it's a

custom back door that takes a lot of code from power runner if you guys don't know power runner is it's a it's a tool that you can use to run powershell commands or scripts without running powershell.exe you pipe it through it does the dotnet calls directly translate them so you can just run regular powershell commandlets without invoking the binary so it's a bypass mechanism the other thing that we see is that it listens on our pcs we'll accept any commands execute them through a command terminal on the bottom there you see that function install what that does is that that's its persistence mechanism right so that is creating a new shared service so within windows there's two well there's four

different types of services but there's two main ones there's standalone and they're shared process services now it's not unheard of to see adversaries use shared process services as a persistence technique it's very unheard of to see it from a ransomware operator right because you have to custom craft a dll you got to make sure it executes but it's in a really good sneaky way because you have service hose binary loading your pro so it runs as a as a library that's loaded by service hosts and they also tuck it into net services as a service group which gives it unfettered network access so they have some dev capability which isn't something that we typically see

amongst ransomware operators it's usually rinse and repeat there's not a lot of custom tooling we start digging around on that golden that domain controller top right there event log four four or four six six two if you see that without that it's not a computer account name in the account name write that one down that's an evidence that that machine has been dc synced right through manycats problem is the associated user account is not from the hq domain it's from the east domain so now we have a multi-domain hop going on to deploy ransomware like now that never happens right that that this is the first ever i've never seen it again looking around at more of the

data we can see that the adversary had created a golden ticket and taken advantage of the extra sid attribute if you guys uh there's a fantastic article that harm joy wrote it's called the truspocalypse he can explain it much better than i so if you're interested do that all you need to know is within active directory there is a attribute called the extra sid and it was created is either to whenever you're migrating content to a new domain or you're upgrading from like the functional level of 2008 to 2008 r2 something either one of those i can't remember but what he discovered and then mimikatz implemented is that if you own the child domain and you have the krp tgt hash you can

append the enterprise admin account to your account and the parent domain will respect it effectively meaning that if you own a child domain you can own the forest that's a big deal right because now i think that fundamentally changes what you think about trust boundaries in terms of active directory and what you should be doing from a prevention or detection point of view is that any child domain can take a parent's privileges and so on and so forth right in this case it was the root domain worst case scenario so again we need to expand scope call up the east domain and they're like well we don't have a problem we don't have any alerts our we don't have av

like avs clean i don't know what you're talking about there's no new admin accounts created we have we have no idea what you mean but you know ceos on the phone at this point they're like do the thing deploy the tech collect the artifacts start analyzing the data we find 100 days ago 100 days before ransomware that same custom mimikatz variant is in the east domain so 100 days like that's a long time to wait on a ransomware attack right too long we start enumerating all the data and we find some really interesting things some of the coolest things that i have seen in my career the stuff that didn't make sense honestly until we you know

until we went all the way through it let me show you some of the stuff that we found the first thing we find again windows event log 7045 we see hits on a detection creating a scheduled test now weird thing about this that caught everyone's attention that is that this binary which we thought to be malware was calling a wave file that's weird not unheard of though you know a lot of the popular malware families will do things like i'm calling jase i'm calling javascript through c-script but i'm going to call it in a text file or picture file right but it's just an extension rename we pull that back i'm expecting to see this is some sort of portable executable

look at the headers no it's not there pull that wav file back into our sandbox and i'm like i wonder open it up immediate player and the thing plays music like real music and i'm like am i about to get the most elaborate rick roll of all time right now so this doesn't make any sense we pull back the task list exe shocker it's not the microsoft one we have to do a long a lot of analysis on this what nothing hits in sandbox something hits an av virustotal never heard of this thing before no one knows anything about it so what we discovered is that task list w dot exe is a custom loader that

will only load audio files within the audio file using really packed stegonography is a dll that gets loaded into memory not not encoded in the header in the audio data is a dll packed in steganography right hopefully that lands how cool that is right so it extracts itself launches a run dll32 process then executes a bunch of shell code injects into that goes out to an internet resource downloads yet another dll then it does some anti-analysis checks so it checks to make sure like all adb isn't running checks that it's not running in a virtual machine checks that it's not running with any other analysis tools or sandboxes and when that only when that happens then it will load a cobalt

strike beacon i haven't really talked about this through this because i honestly just forgot but this checked into its sixth set of c2 at this point so we have this is a brand new c2 infrastructure in east versus hq versus three sets of c2 in west which shows not only seeing some developer capability obviously with custom tooling but they also have the resources to maintain and migrate c2 across various hosting sites across the globe again we go through the process right collect some more shiz get another detection for uh file name original file name mismatch check that one too if you guys have that renamed binaries it's a great way to find evil java workstation.exe

was originally named x.exe connecting out to an ip internet routable ip the other thing that if i recall correctly the other thing that the analyst flagged was this in windows event log 50 5144 i believe that's in the windows firewall file event log and that will log when a binary changes or makes an exception to the windows firewall and in this case java ws.exe was creating an exception to allow on inbound network connections from anywhere so that got flagged and then we go and we take a look at it again nothing in the sandbox nothing in av just blank so we have to do some real analysis on it and what we find out is that java.ws.exe

is not the real one um but what it does is it extracts a dll which then dumps all this data into the windows media player registry key and all that data is a bunch of shell code which then extracts itself runs in memory creates a windows firewall rule for run dll to make sure it does everything creates a bunch of registry keys for ps execs so it doesn't get if they're going to run like you know ps exec no one gets a prompt or if they're running at a system it doesn't just hang there and then once it all does that then it will load them interpreter session with a brand new set of c2

seven sets of c2 at this point we're going on now i have i should mention at this point we're a week and a half into the ir there's no indication that the adversary is active all these back doors are just hanging out they check in nothing else right the only sign that someone has been there was a ransomware on the west coast headquarters nothing no activity east nothing no activity although given the level of the tradecraft we're monitoring this 24x7 several days go by and then we get a hit so we get a hit on cert util going out to the internet github in this case downloading a package this was huge for us because it also

showed us that an adversary was active they came back for some reason the other thing that it is we were able to yoink their github repo and download all their tooling so that we could proactively go out and find systems of interest that they might be pivoting to now hopefully you guys know what ngrok is if you don't know what ngrok is just think of it as tunneling software in this case they're tunneling rdp not fan not all that fancy we see that all the time ransomware operators alike now what was cool which i thought was cool and it showed that the adversary actually had some understanding of the environment not only the environment they had the operational security to

understand how this helped desk software obviously it changed the name if you go back and did a historical look to see what did this binary do the legit one i don't know why or the purpose of it i think it facilitated something for the help desk but it was regularly making connections over 33.89 for internal resources so if you're a sock person and you saw helpdesk.exe make a connection over 33.89 there's a likelihood you'd probably just let it go or it's already accepted or bypass like it's already whitelisted so i thought that was pretty cool that they knew what that software is what that software does and then try to hide out so since we have this

also eighth set of c2 at this point right now we're working off the n grok now that we have an active adversary we try to find out where they came from obviously right and then this happens right keyword hit support 452. old heads in the crowd know that that is bad news right back in the day there was an advanced russian adversary they went by cobalt group and they used to attack swift and atm networks all the time and they stole so much money and one thing that they always did was create an account called support four five i was actually shocked because this but they've been so quiet i was actually shocked that that detection was

still running they used to create a an account called support 5.2 and add it to the hidden users list this right here oh windows event logs 7045 again guys hopefully this is landing this registry key if you're not monitoring for that for like any command line interpreters that are modifying that register key you should it's really good there's very very false positive rate and the adversaries are doing that what this key does if you're not familiar is if you go onto a workstation right in windows and it usually will either show you like a list of people you can log in as it won't show you that support 452 isn't there if you go to see users you

won't see support 452. so it's a hidden user registry key you added to that you could basically have an account without a cursory view no one would know that it's there so what is going on here that's weird that we would see this like immediately for me from my background doing this a long time when i saw that i was like blown away i was i was like cobalt group are you kidding me what is going on right now i haven't seen them in years the other thing obviously that workstation name is sus but that ip address according to the client does not exist in their network that's that was a hard no talking to the

network engineers they're like we don't do 192.168 anywhere because those are often the default for consumer grade devices so we do everything on 172.16 so that can't happen clearly it happens right so the hunt is on it's like where where did this log on come from because it came from somewhere it had to come from somewhere like you can't make that up well technically you can but that's a whole other talk so the hunt is on what do we find out is this company in the east shared an office building with another company and within that office building there was one security office within that security office there was one badge printer and what someone did

was set up a bridge network that connected company a to company b so that the security office could share a badge printer and so there was a fat pipe going right into our clients network that they had no idea now obviously we asked about it either no one wanted to fess up to it which you know being a former i.t person i don't blame them or they left or they forgot or they didn't do it or who knows but we go back we analyze those systems we find homies been hanging out there for 38 uh over a year over a year hanging out if you're a ransomware person homie why are you burning all this great

tooling to carry out a small ass ransomware attack this is nuts it makes no sense we're calling everyone at this point because it it fascinating right so we call the client everyone's jazzed up because the whole team's on board it's like we're going to get access to this organization we're going to find out how mystery hacker got in there if this is cobalt group that i'm going to owe a lot of people a bottle of bourbon because i said there's no way it's cobalt group briefed the other organization they were very appreciative gave them our iocs gets you ready to spin up the engagement turns out that they have an agreement with another ir firm

sad face they're going to engage with that irfm which i get i mean retainers are retainers that ir firm wasn't going to share data so we ultimately have no idea right how they got who where it came from how they got into that other organization or any of the juicy details which is unfortunate it was kind of hit of a dead end we're talking with that client and what we find out is that the other company this these guys down here they are an hvac and control system company and they have government contracts across the nation to install hvac and control system in government buildings boom right wild tinfoil hats go on because we have

no data so we're just like speculating at this point like who is this and now it starts to make sense right so but it's still it's like why did ransomware at the end all the other parts make sense right i'm hiding out in east beachhead that i have my advanced tooling nothing's detecting it we're good i can always get back into government facilities through this pipe that only i know about why ransomware right i'm going to tell you guys days later we're doing every time we do custom malware analysis we create a bunch of yara rules and we're always monitoring stuff that's being uploaded to things like virustotal reversing labs and we get a hit

for the java and waveloaders as well as our custom mimikats so we get on the phone we're using all of our intel contacts we're like who knows about this who knows about this who knows about this we find out and you can google this i'm not big on saying other company names but like if you google wave malware you'll find the article that i'm talking about uh we find out this other uh firm who we have a really good relationship was just getting ready to release a report about turla executing the same payloads to carry out government espionage activities across the middle east and europe not only was it the same tooling we have a c2 overlap

with the stuff that we had in our ir looking at the time frames we see that it is the same days so same days that we have activity of legit espionage going on from turla we have an overlap of infrastructure and overlapping in tooling and we have an overlap in time with our ir what the right so who this turned into like a real whodunit right so we no one no one's gonna say turla is doing ransomware as a as a sponsor group like they're legit enough that we're not no one's saying that i'm not saying that they carry out legit espionage activities we see overlaps from cobalt troop and obviously wizard spider with the trick

bot so why do i bring this up guys well this is very relevant and we need to be prepared as a community for this is we can see that the lines between nation state and cyber criminals are continuing to blur like this is a thing and it is a legit thing we can see that and it's a two-way street right so in 2020 there were two chinese nationals that were indicted by the department of justice because they were carrying out espionage activities on behalf of the chinese government using chinese tooling for financial gain so like harvesting criminals from the dark web or all these forums say hey do this we'll give you some money alternatively

what we see is you can see this quote from christopher wray is that and the evidence is out there you can google this we have cyber criminals that are are espionage analysts people that are doing legit things maybe it i don't know something like turla that are carrying moonlighting is cyber criminals at night for their own personal gain using the tooling that was developed by those nation states and in fact what timing is almost like i planned this yesterday we put out a paper detailing how the trickbot group is systematically attacking ukraine now now if you guys have been tracking ukraine over the years or any of the trickbot malware you'll know that trickbot malware is specifically

designed to not execute on systems where the ukrainian language is detected that has changed so now we see that the trick bot group is carrying out attacks suddenly now that the russian government is attacking ukraine what a systematic shift in behaviors they said we've never seen this before ever we all like take take notice right i've had friends who i respect a ton very smart people and they've said ransomware is a problem for a soft target i think that's small thinking because this isn't the only case where we've seen custom tooling bypass your edrs bypass your your avs do all the things and people still get popped right so we need to take this away as do do

that stuff buy buy the edr buy the log central you know get that threat hunting do those things but we need to properly prepare for when those things fail right and attribution is going to get harder as these lines continue to blur but does it matter does attribution matter as much as it used to right if the if the attack's going to happen all i mean if your work for a company do you really care who it did who did it so we need to we should be starting to shift our mindset away from caring about too much in my opinion about who did it rather than how they did it right and that's when we go back we're talking

about value of data and why it's so important to do things in terms of analysis trusting the process in terms of saying collect analysts and security people all like they say collect all the things let me let me find the stuff right i'm not sure that's the right way to go about it and in fact we should be using security op security and threat hunting and all these things we're talking about we should be using that to drive investment we should be having these conversations with our executives and we should not collectively forget the opportunity that is presented to us from ransomware we may never have a chance to implement the stuff people have been talking about for 20 years

getting rid of admin rights right network segmentation you guys seen the zero trust stuff we've we've been talking about this for years but we've never had this opportunity to tell you right now there's not a ceo on earth that is not scared about ransomware so we have to understand the opportunity to do the things that i'm talking about you know build those detections really work with an ir team to fundamentally understand the goals and objectives of an adversary and with that i really think that we can make a change to how we do computing globally so i hope you guys liked it it was a cool ir and i can take any questions if you want

or if you just want to chat happy to happy to do it thanks [Applause]