The Continued Evolution Of Userland Linux Rootkits by Darren Martyn

all right um hi everyone um so i'm here to talk to you about linux rootkits i'm darren um i do security research uh and linux stuff um all right so i'm just going to get straight into it because of a shitload of stuff to cover and half an hour um so i'll open up with so what i'm talking about here is a ld preload based linux userland rootkits um so i guess the first thing i'll cover is what what is ld preload and ld preload is an environmental variable interpreted by dynamic linker and it allows you to put libraries in ahead of other libraries so for the people who are more familiar with windows it allows you to change the

search order of how libraries and code are loaded into software so you can put stuff in ahead of other stuff so it's really useful for debugging you can change execution paper at runtime you can replace functions you can substitute out functions you can hook and stuff and of course that makes it really useful for root kits um it can be set globally system-wide on linux with the cld.soda preload file which is it's just a file and it it parses it really loosely so like the dynamic linker will look at this file little anything that looks like a library will try load which is quite fun um on os x there's an equivalent called ld insert libraries on windows we've

happened at dlls but um if you mess about that in dlls and windows just nothing works um it's pretty much documented as heroes here be dragons if you if you go here you're on your own um pretty much any platform has some way to override the library search path which is great because it means you can write super portable root kits so we can replace functions at runtime we can mess about with what programs with what programs are doing so we can make programs live so write rootkits and this allows to hide stuff do stuff in the background blah blah blah um so i kind of come up the list of like because everyone when people think root

kids people like oh kernel root kits you know kernel modules and you know ring zero stuff it's like no no the problem with writing kernel root kits for linux is the pain in the ass because the linux kernel api is not considered stable they keep changing [ __ ] all the time so like you end up having this like big five big file of if deaths like to or like having to recompile your kernel room kit for like every single major release of linux which is a pain in the butt so instead you just write one userline thing which works everywhere um unlike the pros are like they're stable they're easy to write um it takes like

20 minutes to implement you hook the cons are like that they suck um they absolutely suck because there is one place on disc where they live so if you're a forensicator you just cat ld etsy ld.soda preload and you probably found the kit and they also mess up performance which i'm going to point out later so i'm going to talk you through first how to write one of these um for those following along who want to have a bit of fun with this so what you do is you just identify something you need to hook so if you want to hide a file you just s trace ls and you see what's it what function

calls ls looks at and does and then you go okay what what are these do i have to mess about with to hide stuff um you figure out how to hide the stuff which is usually just like do a string comparison if it's got like a magic string in it hide it and then you write the hook or you know draw the [ __ ] owl i believe it's termed enough and then you just repeat until you know the thing works so we're good i'm just going to go through a few examples so this is a very simple hook that cripples the randomness function on linux um it'll just return 42. so we've a simple program that just

calls rand and like gets a number but when the hook is installed it'll only ever return 42 and like this kind of you know it's kind of like the the hello world of like writing these hooks because like three lines but it shows you like you can like break the randomness in video games or like break the randomness for generating cryptographic keys by corrupting the random function um you can also do conditionals in your hooks so like if we go back like that looks like it's just return number you know it's not very sophisticated but we can add like we can have it load the actual random function and have like a getting so like

if a certain environmental variable is set actually return real randomness otherwise return broken rand and like this just an example of that where like if we set the packs equals lol or hexagons whatever and load it it won't do the thing it's like this conditional will come in handy later this just like the very contrived very simple example that i can use like show you how these hooks are written um that actually fits on a slide um you can add multiple conditions um you can do stuff like basically your hooks act as i o filters um so here we have a reader example it's like an actual practical example that's partially borrowed from the jinx 2 root

kit so all you're doing is you're creating reader and then you're creating the original reader and if string string your hidden file name if it might if the res response of reader contains your hidden file name return buggery nothing return like nothing however if it doesn't if the directory being ls doesn't have your hidden file then you can just return the file contents so the admin doesn't you know just have all empty folders everywhere and it's just an example of this working like you run ls and you've got like the hide me's you do the preload and run ls and the hidden files are hidden um and this is actually a great time to show the one the timing attacks i found

that allows you to detect these root kits because one of the interesting things is because you're doing all that string comparison crap in the middle of your root kit you absolutely murder performance so like every time reader is called performance like takes a hit makes you do all this diversion and like go off like to check string comparisons which is computationally actually quite expensive so because all that extra stuff we can actually demonstrate like the actual performance impact so with the hook installed you've got like it the real time is like three milliseconds whatever whatever that value is like 0.003 without the hook at 0.002 that's like a 50 percent in worsening of performance and that's just

like one pretty contrived simple example if you imagine that system wide you know every time reader is called or any other hooked function is called you're getting whacked with like a 50 plus performance hit somebody's gonna notice you know it's like okay you might be able to find all the hidden files and stuff but your computer is slow as [ __ ] you know your computer doesn't work anymore it's like the whole thing you know virus is slowing down my computer but actually real um so another thing you can do with these is you can like it's really trivial to implement local backdoors with preload kits and they all like all the root kits i've examined in the wild that use this

that use ld preload pretty much all implement the exact same local back door and it's just if an environmental variable is set and a set uid binary is being ran spawn a root shell um they all do it exactly the same way and it's kind of funny because like just this one functions just been copy and pasted across every single rootkit in the wild and it's this is this is exactly that function they're all just variations in this where it's like oh if an enviro is set and the current uid is zero like if you're running like pseudo or something drop a root shell instead and then what they do is they put a call

to this hook in every other hook so it makes sure it gets called so like you just shove it like in the middle the access hook where it's not going to do any harm or you could stick in a constructor or destructor but this sounds like done in practice and this is how that back door works you know you on the system's got this rook installed you know you set the magic environmental variable to whatever value and run a set uid binary next thing pops a root shell instead of like doing whatever it's supposed to do it's like pretty straightforward but every like the thing is all these root kits also implement like a wide variety

of remote back doors um this where it gets quite interesting so this is where it gets neat and i'm going to go back and forth a bit so most of them the jinx 2 family lineage of these root kits pretty much all use they all hook the accept function and the accept function is like what's called like when a socket connection is made to like when you when you connect to something like when you are listening on a socket and then you accept the socket blah blah they all basically just have like a magic port that like if a mat if a certain magic port is like set then it drops you a shell instead of

giving you like access to the web server a bunch of others like umbreon father vlani and a few other of the fields of the slightly lighter generation rootkits um hijack pam they add like an extra check to pam to implement like a magic password and ssh um that's pretty popular in some of the stuff in some of the in the wild ones and then another what another example when i saw hooks right where if you try log in with a magic username to ssh it triggers like a hook that spawns like a bind shell on a high port and it's like just kind of weird and then some of the earlier ones used like port knocking magic packets for their

more remote back doors like if you send a packet with a certain like magic value in it it would send back a reverse shell some of the more recent ones i've seen have done stuff like hot swapping etsy password and etsy shadow at runtime so like they will intercept like the call to like read e2c pass wd and give it like a replacement passwd file in memory which i thought was quite neat to like implement like an extra user and some of them do that with the authorized keys file um basically there is infinite potential to cope with like weird new ways to like implement remote backdoors entirely in user space by hooking various functions um you have to kind of

think outside the box a bit these are just ones that we've actually seen implemented by people and actually used so like i'll go through the example like how the accept hook works is pretty straightforward so this is from jinx 2 and all it is is you're replacing the accept function with your own accept function and you just get you know you do accept on the socket by getting your old accept function and then you call the drop tube shell with the socket file descriptor and the address and what happens in that function is actually really simple it just checks if the source port of the connection is between a high and a low number and

if the source ports in that magic range instead of like serving up apache 2 content or like an ssh login or whatever it serves you up a root shell um it's like really straightforward and it's like really kind of elegant and simple and also absolutely murders the performance of all your network demons like your web server's performance is now absolutely just no you know it's just slow it's it's messes things up pretty badly um like your web server powerful drop your database perf will just go through the floor um and this is like an example of like what to the user of the rootkit what it looks like so if you connect like if you connect to a victim like from

any random port you get the ssh banner right but if you set the magic source port and you're connecting specifically with the source port of 3137 the destination port of whatever service is open it'll instead prompt you the password for the back door shell and then you know you get your shell so that's like that's the most popular because jinx implemented and jinx implemented the accept hook um and then everyone just copy and pasted that into their own rootkits because that's how malware is written it's just control c control v apparently everyone's lazy so the other the other kind of main one we see is the backdoor and pam which is really neat so pam is the plugable authentication

module it's like how authentication is handled in most linux so what the rootkit name father does is quite neat it um so replace the pam authenticate function and it just checks like if there's a magic secret password has been entered it just does a quick comparison and like if secret password then oh you also get logged in as any user which is pretty neat um if the magic password is not is not provided it'll actually check for a real password so you don't disrupt normal users trying to log in now the one problem with the palm back doors is that they're really easy to scan for and i'll get to that later you can detect these you can detect them

remotely um if you know you're looking for but we'll get to that later because with with remote back doors like the accept back door um when they break they really break like when that when you're when your back door stops working it really stops working in a really funny way so i thought to myself i'll try find like broken root kit installs on the internet like live infected boxes where like something's gone catastrophically wrong and i was able to find load of them like i found a i found a copy of a rookie called lib mdma and it's actually it turns out it was one of the first ones ever written using the ld preload

technique and i found that just by like i found like an irc chat log of somebody complaining their system was banjaxed and like i started googling for like the file name they mentioned and then found it like on some random web server and like clearly the rootkit bit had broken because it had stopped hiding itself and it was now just serving itself up as a file so i was able to get a sample reverse it wrote a lengthy piece about it um but i decided to look for other root kits that have gone wrong so i found that the dynamic linker on linux spits out a very specific error message when it's trying to preload a library but the

library is broken like it's the wrong architecture or whatever or it's just missing so i put that string into showdown just just kind of thinking like this is not going to find anything this is dumb you know this is but no it turns out it did find tons of stuff so you get this error message object blah from etsy cannot be pre-loaded but like you get that message everywhere like every single network service on these teramaster nasa's was spitting out this error message and i was like okay so all what the story with these is these are network nasus and somebody had gone around somebody had compiled some rootkit and tried to install it on like

thousands of nasus but they compile it for the wrong architecture because they were an idiot um they compiled like for x 86 64 and try to install it in a bunch of arm systems and i'm like script kitties man you know couldn't even you know they don't even check so they just hosed all these things but like they left a copy of the rootkit just sitting around you know like on all of these and it was pretty funny because when i started googling i found like people have been complaining about like their nases were broken because the skids had like put this broken root kit on them so people were like posting on the terramaster forums going like hey i'm

getting this weird error and like the terramaster people was like oh uh just delete the file nothing is wrong you know completely ignoring the fact like completely like denying and like trying to steer their customers away from like you have been wrecked because our software sucks but you know i thought it was an interesting aside like that because these guys had like compiled their root kit wrong like they had very publicly messed up in a way where like you could actually see um so i started digging into this one a bit more i thought it was interesting you know because like the vendor is like oh just delete the file um it's fine everything's fine you know

it's just a software bug it's not like it's not like you've been owned you know so i also found like reference on like random chinese forums to like hey i'm getting this weird error here's how to fix it and like i was like okay so this is clearly like somebody has clearly tried to use this lib system that solar kit in the wild quite you know quite uh want to be the word generously i guess the you know they've spread the love around a bit so it turns out that it was the kinsing group and the specific root kit they were using was a modified version of one called bjork which is open source and it just didn't work on some hosts um

because they compile it for the wrong architecture because they're complete idiots and they were using it to like hide badly a crypto mining campaign and they were spraying around like log for j to drop this root kitten you know script kitties like really low grade absolute bottom of the class tier cyber crooks um zero out of ten you know they get absolutely no points no style points either you know like copy and pasting stuff and then somehow still screwing up so after this i started trying to think like for like for like if you come across one of these samples it's it can be a bit of a pain like you okay you've got like a shared

object file that's your root kit like how do you categorize it like as a for the for the blue team guys like what do you do like how do you how do you categorize it how do you bucket it and cluster these samples and i started trying to come up with like some kind of way and it didn't it's in excel the spoiler it ended up being an excel spreadsheet in the end because that's apparently how everything's done but i started coming up with like a worksheet for analysts and i'm still working on this and i'm probably going to upload the thing in the next couple weeks to github or something like a worksheet for like your

incident responder to go through when they encounter one of these root kits here are the steps to like identify it mitigate it and then work out how to detect on the rest of your network and it's quite straightforward like the first one be like like just open it up in ida or ghidra or radare or whatever your favorite reverse engineering framework is and just identify what functions it hooks because you can often like figure out which one it is just based on the unique subset of what functions it actually implements a hook for the second thing you can like you need to do in like your worksheet of identifying these things is does it has

it just copy and paste a code from other rootkits and the answer is probably yes because they all copy and paste code from jinx2 because it was like the first one that actually worked so like the diversity of code out there for these things is pretty small um then the next like you can then figure out like what remote access method does it implement like does it implement a pan back door does it implement an accept hook does it implement like port knocking and this is very useful for like remediation for scanning your network for it later on um does it implement self protection methods like when you think you've uninstalled it has it actually sneakily

reinstalled itself while you weren't looking so a few of them i think it's the umbreon one when you try to delete it it hides itself in memory lets you delete it and then by hooking the destructor of rm like when rm is like when after rm has rammed a file before rm's process terminates it then like reinstalls itself like it specifically hooks like the delete like the rm binary in a way that like allows it to reinstall itself after it's been deleted which i think is hilarious to like ah yes i have fixed it the problem has been remediated and it's like actually no i am still here um another thing to look at you can

fingerprint them pretty uniquely by like how do they obfuscate strings most of them don't but some of them use like really really terrible xoring of strings that just like you know you can figure out the key to pretty quickly um and then very very few of them a limited subset of them actually encrypts strings in a way that's meaningful and a pain in the ass to reverse engineer the other thing is like the thing that's important to figure out with what with a sample this is how does it decide what stuff to hide like is it magic string based is it magic group id based is it extended attributes based and like once you you know you figure out these

things and once you've gone through all those steps you've probably identified which exact piece of nasty you're dealing with um and of course it'll all end up in a giant excel spreadsheet um like the what functions is it hook one is is something that i still i've got a giant excel spreadsheet that i need to publish um for this but like i basically picked a bunch of example root kits that i'd source code for and dumped out every single hook they implemented into like on one side spreadsheet you've like a list of every single possible hook and then the names across the top and little tick box when you get your sample you go and you cross

reference it and you can pretty accurately identify with no further reverse engineering or hard work what root kit you're looking at just by cross referencing it against a spreadsheet the it was a pain in the ass to do this because like the laney root kit seems to implement like a couple of hundred different hooks because whoever was writing it was like clearly like oh well i'm gonna hook that function and that function that function oh more functions just gonna back door all of them so to like like documenting that took about a week um using like just by looking at code reuse um one thing that i think would be interesting to look into is writing like

flirt signatures or yara rules to automatically detect like code reuse from like snippets that have been stolen and repurposed um i actually found out that's how a company called inet sir i've no relationship with them but they have a weird product for malware stuff called code dna which turns out already implements this um so and they say it's super effective and it seems pretty effective at like reducing my workload um like if you know just instead having a reverse engineer the function just check have i already you know is this is this present anywhere else um identifying what backdoor methods it ha it implements is important because if like if it's got an accept hook using

ssl you're looking at a jinx variant every single time if it's doing a pan back door it's going to be magic password-based every single time um they're all they're all very similar they're all copy and pasted from the same code so this word gets neat because if you actually take apart the remote access bit you can do network scanning so like if it's the accept hook you just figure out the source port that the sample is looking for get mask and set dash g source port number and scan your network and then diff the output against a scan with randomized source ports [Music] and if you see something weird it's probably it's probably one of these um because

you'll get a differential response for pam back doors if you can like figure out the magic log in just sweep your network for it um same for port knocking ones just replay the knock sequence across your network and if you start getting connect backs you've got something is very very wrong um the self protection methods usually constructor destructor hooks um very few of them implement working self protection most of them just rely on like stealth the ones that do reinstall themselves though are very funny because they're real pain to remove you to like reboot from a live disc and you know actually delete the thing um string obfuscation most script kitties are too lazy to bother but like

some of the samples like zazzle actually implement xor stuff and some other stuff um automating the obfuscation these will reduce your workload there are ida python scripts and stuff for that um the american files to hide again blah blah [Music] already covered this um so yeah this is the bit that i want to this the bit that like where it's going so like we've seen where it is but where it's going is kind of neat so like most of them are general purpose like remote access and hiding but some of them we've started seeing like much cooler limited scope specialized ones lately um so i'm going to talk about a sample called lib curl so this it was trying to hide by calling

itself lib curl um so this root kit was dropped as part of a cryptominer campaign um it was covered by a crowd called sandfly security and they claimed that it was capable of evading a bunch of linux edr products when i took it apart it didn't have any magic that would like it didn't have any special sauce or anything so i guess just linux edr products suck um i guess that's the case um but this one was neat because its one job was to hide a crypto miner and all lib curl did was it hit the miners process and files but also hooked the functions like to check cpu usage so it would lie and

it'd be like oh the load average is really low and there's not much cpu usage going on while your cpu is turning into a space heater in the background system is logged in like oh no no load is really low you know everything's fine um but yeah this like somebody written this specifically to like hide their mining which i thought was really neat except for the fact that like if like it's a physical server you're near you're going to hear the fans going and you're going to be like something's up but they've been they'd like you know they they've put a lot of time and effort into like finding a way to coerce like h top and ps and stuff

into reporting the wrong like load averages which i thought was kind of neat because somebody had like you know they'd actually put effort in they put a bit of work into doing their crime um and the last thing that the last one that i want to point out this just kind of where things are going that's kind of interesting but it's completely irrelevant because nobody can afford gpus but uh a few years ago some guys figured out that some team jellyfish they worked out like they could use the ld preload thing and then load their code into the gpu like hiding except their whole trick was hiding executable code in the graphics card's memory by like

loading in some opencl stuff it's open source their work it's really really neat and like malware hiding in your graphics card is like kind of neat because nobody writes antivirus for graphics cards but also nobody can afford graphics cards because they cost like you have to remortgage your house for them that might become relevant in the future if you know we find a way to get graphics cards cheaper again but no point um but like i can see that like that kind of stuff like hiding in weird bits your system like being potentially where things go um and these are just references um and more references and there will be yet more references on github and stuff when i upload the

slides and all the supporting material um the github is just my github.com my name uh yes i guess any questions [Music] yep have you done any research on like say containerization does that help in any way so i didn't get to cover that here but i have seen some kits that are specific to containerized environments or environments using containers like i came across one that has a function that allows it to create a hidden lxc container and like hides an entire container from the admin which i thought was really neat i've seen others that try to abuse docker a bit um [Music] but like they kind of don't really work yet um so i guess like some

some stuff like i think we're gonna see more of this in container environments though um it'll be neat to see like how they impact the performance of your container environment because i imagine like because the performance impact they have they'll probably trash like your kubernetes set up by just making it slow

um for your showdown search um what was the search string what's what's the google door to find the broken real kids uh let's go back it was just the error message um ld.soda preload cannot be pre-loaded kind of open shared object file you just stick that in the showdown you'll find so many systems that could have been backdoored but the rootkit didn't work because somebody was incompetent

did you test for uh jesus did you test for a windows subsystem for linux uh not yet i haven't played with that very much yet to be honest um i only recently came into possession of a computer running windows [Music] i'm still not sure how it works it's kind of weird you know you go from like seven to ten and you're like oh they've they've moved everything guys i'm scared um but i actually do intend to test uh some of this some of these on because i do think they will work in wsl because wsl is just linux in a container now it's a vm now

the networking will break ever so slightly depressed sharing will break everything slightly they'll get you 99 of the way there and then drop it off so we're nearly in the year linux on the desktop yep

so graphic stuff works okay i'll actually have to like power on the windows machine then and actually maybe use it because then i can actually add a power points properly all right uh that is that um if anybody has any other questions you can grab me later um