Clean up on Aisle APT

it's a grocery cartile all right thanks everybody um obviously you're hopefully you're in here to hear my talk about cleanup on aisle apt um real quick who am i my name is mark parsons i am a threat and tall analyst i do a little bit of the development on the side more importantly i'm a husband and a father three wonderful kids i am

a huge solar power enthusiast um i'm actually looking forward to the talk after this where the guy's going to talk about some stuff with solar power inverters i'm also a burner owner i can't not have a presentation with my dog on it um i am sad that i had to follow rob lee that's hard to follow up on um for people that follow him or his organization his new um chief of uh threat intelligence sergio has been driving across the country and has been taking pictures of scada systems and tagging them on twitter with ics across america so i wanted to pay homage to that i'm a solar enthusiast enthusiast this is my solar panel array on my house

so the purpose of this talk um you may not get from the definition about cleanup and i'll apt but what i've been doing is i've been purchasing nation state nation-state-related domains after they expired and i've been sinking them but before i go into that i want to talk a little bit about what is a sinkhole does anyone here know what a sinkhole is yep so something like that so that happened here in baltimore recently um i think this is the actual street it happened on but what i'm actually talking about is from a network perspective so i'm talking about dns sinkhole um and i want to talk about it from an enterprise or a network defense

standpoint and um basically what it is it's a custom dns configuration that allows you to take over domains by takeover i mean you can tell your dns resolvers to say i am the authoritative zone for these zones out on the internet i don't want my systems to get routed to it so you basically you basically made the middle dns out on the internet and say none of these domains my clients can go to so the idea behind that is if you're using bind you can use rpz i think certain dns hardware components can do that or you can use custom zones you basically say hey i'll use dynddns.com or dyndins.net make that a zone zone you

say i own it any dns requests go over to this ip that i own and then that way you can mitigate things like dynamic dns if you see a report from a threat intel company you can say oh we don't want any traffic to these domains sync them point them at your own infrastructure that way you can also stand up listeners and you can say oh hey why is someone beaconing to my sinkhole a previous job we had this and we actually found one of our web servers was getting hacked because an external web server was touching our internal sinkhole it was because of an injection attack where the attackers were running scripts and they were trying to download

additional files off of that dns domain that we had sunk so we're like oh hey that should never happen and in this case the sinkhole actually worked and alerted us to something nefarious was going on so why is sinkhole so i kind of highlighted that from a defense standpoint i think it's a great defensive strategy overall i think it's pretty much low overhead especially for smaller organizations as you get bigger it can you may have some scale issues but i think it overall it's very easy to maintain at the end of my slides i actually have a link to a sans reading room white paper so if you just google sans reading room dns sinkhole

it's a fantastic paper that kind of walks you through all the the concepts and how to actually instrument one um from a research perspective it's a little bit different so a lot of threat intel companies are ir vendors what they will do is they will take over domains they'll either get legal lawyers involved and say hey this domain's malicious here's how we know it's malicious let's get the registrar to basically let us take it over in some cases they do that for one for threat until but they also do it for in some cases for jumping up incident response business i don't have lawyers i don't have that capability so i wanted to do it for

research and actually see what would i find so i wanted to do a sinkhole because again i think they're easy to maintain minimal cost you can host the infrastructure pretty cheaply in the cloud spin up a couple of vms in digitalocean even amazon or google's cloud you can spend less than 15 bucks and have a decent infrastructure set up um if you have uh development skills or even just basic log analysis and shell scripting skills a lot of the reporting can be automated as well to make some of this triaging a lot easier so i started this little over a year ago i want didn't know exactly what i would expect to find when i first started thinking about

doing this again i wanted to see what would happen if i bought different nation state groups that i follow what would happen if i bought their expired domains would i actually see any victim traffic um so i sat down and kind of came up with some hypotheses if you will i figured i'd see lots of dns queries i figured there's uh different systems out there that are probably set to resolve domains um i figured i'd see threat until vendors i'm going to lump in things like websense and other proxy vendors in there as well i figured i'd see limited victim traffic i honestly didn't think i'd see much um and i also thought i'd see just the

random crap of the internet people scanning for wordpress blogs people scanning for port 23 all the crap people just looking for just random stuff so my initial findings after about the first month i bought a few domains and i had no victim traffic i had lots of dns queries and a couple of them were interesting one of the domains that i bought there was only two places that queried for a sub-domain so foo.badguy.com in this case foo was an acronym for a company and one of the two two places that was querying for that sub domain was that exact company um looking doing some open source intelligence that company had had an intrusion six months prior

they had stated they believed it was a chinese threat actor and this was the chinese threat domain so chances are this was the actual c2 um i couldn't i did some reco i did some basic network analysis on the ip that was coming from it looks like it was an op resolver on their end possibly for a sim i don't have any more of that other than it was just interesting to see um well i didn't see any victim traffic i definitely see dns queries so my thinking is it was some sort of network defenders that had said something in their log management or sim query this domain on a regular interval if it changes alert us or alert us in

the sim um so i was actually kind of a sad panda at this point i was really hoping for some victim traffic and then i kept deciding well let's this is only three domains let's keep going so i'll pop some more and oh my god uh the amount of beacon traffic that i started having open night was um almost incredible um so when i say beacon traffic this is malware that was on the system running and sending check-ins back to my sinkhole so when i built all this infrastructure i had i have a dns server i have a couple of other boxes that i point the domains two and then i have the log aggregator on

the back end um immediately within uh probably within the hour of me turning on the domains i started seeing beacons from multiple places around the world so after that that kind of got my my uh my mouth wet like oh yeah let's keep going let's keep going so been doing this for 15 months now and this is where i am today i've purchased 68 suspected chinese domains again these are suspected chinese operators or chinese apt um 27 russian and three indian yeah i'm one of the just there's a couple indian groups i followed i just wanted to see what would happen so let's start with the in let's start cleaning up a little bit let's start with the suspected some

highlights from the indian domains one thing that was interesting for the three that i bought they all were in use from anywhere from one to three years meaning the the adversary was actually actively using these for this time period two of them had expired three to four months before i bought them who here buys expired domains anybody so one of the issues with buying expired domains or even if you own your own domain if you let it expire you actually have 45 days after it's expired to actually buy it back but every registrar is different that 45 days more of a sliding window in some cases i've seen it go all the way up to 60 days which is very

frustrating when you want to buy something and you want to get ahead of any other researchers potentially so that's just one thing to keep in mind one of the other interesting things was one of these domains had been expired for 14 months it had been sitting out there for anyone to take over and i was i was actually just shocked it was out there that long but what's been nice about this all three domains i'm getting consistent beacons meaning i'm getting malware speaking back to me on regular intervals across all three domains different malware different types of data what's interesting geopolitically the majority of these connect these beacons are all from pakistan which if you think about it

india and pakistan they don't really get along so it kind of makes sense um what's also kind of funny and sad at the same time as most of this traffic is only base64 encoded so easy to decode easy to look at what's coming um and in one case um it looks one of the pieces of malware looks to be like an auto data collector or auto data miner that looks for new documents powerpoints excels text files and when it finds them on a weekly interval sends them in clear text not encoded just as a post with the name of the document and the actual document so from me as a network defender like why is this happening this should

be caught they should be seeing this but then me as the sinkhole operator goes oh crap i need to turn off pcap analysis because i don't want some random dudes pii sitting in all my boxes so that's that's been some interesting highlights with the suspected indian domains so in this case i've been 100 three domains 100 all of them have had beacons up next comes the suspected russian ones um unfortunately for me majority of the domains that i've purchased have not had any victim or any beacon traffic but i have seen a lot of things like websense and other vendors hitting the same urls over and over and over even though i reply hey this file's not

here um i've come to the point i've almost wanted to create blind views for when it comes from different researchers and point it back at their sites so redirect the traffic back on them um just to see if one if it will actually show up in any intelligence reports or any passive databases to kind of measure where they're getting data from but i haven't been that ticked off yet um only ten percent of the domain so i've bought 27 domains we've got beacons from three of the domains that i've purchased have had beacons um all of them well majority of them are in europe um one's in belgium two are in the ukraine and there's actually a beacon um from

moscow and it's consistent um unfortunately i haven't been able to gather enough data to figure out um who it is within moscow but it is interesting to see a suspected russian domain and a moscow implant or an implant that's um targeting people that live within within within russia up next is the chinese domains so some highlights here um by far this has been the largest group that i've been purchased against 67 so it's i have a little bit of bias here as well only about 25 of the purchase demands have had beacons so um right now i'm like okay is it really worth the depending on what provider i use the five to twelve dollars

every domain to keep buying russia or chinese domains when i'm only having a 25 success rate i think it is um i'm also only focusing on uh roughly three to four different chinese uh groups but what is interesting to me is we when you look at the data there's traffic from every continent except two antarctica and south america and the chances are if i start looking for chinese groups that may be targeting south america i'll probably start getting beacons the one that really stuck out with me was africa china has a lot of interest in africa especially from like a resource potential but in this case all the beacons are from one country i started looking at why why is china

has interest in this one country well one of the primary diplomatic partners of this country in africa is taiwan and they have said they are not going to break their relationships with taiwan and we all know that taiwan's number one china's number seven so please google that you'll thank me later um so just interesting to see especially because the group that i that's doing this they target typically south south pacific and taiwan so it makes sense that they would also target this particular country in africa what's also been fun about this is especially from the chinese domains traffic comes in over http and some of it is it's straight up http with posts or check-ins with

within with encoded data um others it's https again i've had to set up ssl certs look at the traffic like um so that's also sol cert then basically proxy the clear text traffic to something else so that i can capture it we are getting dns beacons so in this case it's been plug x using dns and a couple of the ones we've had custom protocols so unfortunately we haven't actually figured out what the beacons are what they look like or what the actual data is but we definitely see there's something custom and something we're still looking on it but overall there's 50 different victims if not more um that i've lumping it together by asns

where i can across multiple intrusion groups um so from a data perspective i'm like oh this is kind of huge this is kind of fun but it's also starting to get to be a little bit of a nightmare for more than one person to actually manage and to try to actually reach out to people to say hey you have a problem so this is a snapshot from a 30-day time period back in october november to show all the incoming hdb requests to my sinkhole infrastructure just for the chinese domains um peak was almost 60 000 requests within like an hour so some of the malware beacons a ton some does not that little dip in the middle is because

i was not doing proper log management in my disk field and for those that may be threat analyst or just like to look at what the beacon structure looks like these are the top http request uris for the chinese domains in that same time window some of this will have changed because of domains i've bought after that does anyone recognize any of those okay um the fifth one down photos.asp that's typically with the de rusby implant um multiple intrusion groups use that um there's been multiple write-ups on it so just interesting to again see more acquiring domains from multiple groups and seeing that across multiple groups as well so sitting there looking at feel sometimes i feel like i'm juggling all

these fun little barrels in this cleanup aisle but i wanted to quickly go over some lessons learned so again waiting for domains to expire especially once you've really been tracking for a while and you know they're going to expire and you see they're sitting on the um waiting expiration time window and the registrar is taking their sweet time this is literally how i feel i every day i check them and like oh come on please i just want to buy this domain so i can move on with my day um some some basics so if anyone else is interested in doing this one thing i would ask that you do is check who is history unfortunately i

have bought a couple of domains mainly in the russian intrusion groups that other researchers bought prior so they were already cleaned up so there was no victim traffic so just do your due diligence before you buy something i was doing this late one night and did not do it i'm like oh yeah let's buy this and i'm like well that's why i didn't get anything also have good log management as you can see in one graph i did not for a while so making rotation compression will be critical especially depending on even if you do this from a criminal it's like a crimeware standpoint i've done this for some primary stuff too um where overnight i went from no beacons

to two 200 different clients around the world beaconing almost 100 000 requests an hour and my infrastructure was screaming at that point i'm just from from a logging perspective again because i was using cheap vms in the cloud so having good log management and being able to ingest those logs so that you can not have to store them all locally is important i would also say network ids are a necessity in this case because like when i talk about the custom protocols or just even some of the beacons being able to classify the the traffic as it's coming in and tagging it for for reporting or or anything like that to say oh hey here's

basically if you want to write a report here's all the ips that have hit for this particular beacon or we're seeing something new having the ideas is great for that one thing to keep in mind though if you're using custom rule sets or things like emerging threats or even snort rules chances are the directions are backwards because you're looking typically visuals are looking for beacons leaving the network not beacons coming into a network so if you're like i've got a plug x rule i know i have plug x but you're not getting plug x alerts change the beacon around chances are it'll start firing um i'd also uh recommend a repeatable process for adding domains

to the infrastructure especially if you try to add in additional researchers or friends um because it helps if you're not the only one doing it and it helps if you're trying to do this late at night you may not be a little weary so that you don't make mistakes so the not so basics notifications um so for anything that's been us-based i've actually reported at law enforcement i've made some contacts through multiple law enforcement agencies that i say hey i've got this data they do the victim notifications for me i don't have to worry about that but anything else i'm like i'll take a stab at it of all the isps i've reached out to only

one responded and i think the only reason i responded to that is one of the main higher ups within that isp started following me on twitter because of another presentation i did and i dm'd them and said hey i'm getting ready to contact you guys because i've been seeing some traffic because again put it in this this email box i'll tell them you're you're going to email us and within three days i had a notification they actually told me who the victim was and said yeah it's actually um where they were and that how they were going to fix it which was kind of interesting um going on the custom protocol parsing i am not a reverse engineer

i don't even want to imagine to be one um so in some cases we have copies of the malware that's how we decided to buy the domains but you have to go in some cases we've needed a reverse engineer to figure out what the protocols are so that we can either build custom detections um with snort signatures or possibly even just build being able to build a simple back end that can decode the data as it's coming in um so if you have someone that's good with re i'd say hey maybe throw them a couple samples and see what they can do um money so 98 domains roughly 10 bucks a domain that's almost a thousand dollars just in

nation state domains over the past 15 months thankfully my wife is really understanding um and then just the costs of do i want to renew these probably not depending on if i'm still getting victim traffic or if i am getting victim traffic i'll probably keep them um but as this started to gain momentum and we started getting more traffic we i was like i need to segment this a little bit more and not have everything on one server so then i have to add multiple infrastructure pieces and that and more capability from a logging aspect so that starts to add up some of the costs from a hosting standpoint um and then multiple sinkholes might be needed

um the reason i say that is in a couple of cases uh domains i bought there were subdomains that were queried but the beacons were a custom protocol so i couldn't put a basic apache listener which would look for the host header and the port and do a log that way it was something completely random and trying to figure out which sub domains we're doing which beacons is tough when you're all pointed to the same one or two boxes so in some cases i had to stand up five to eight different systems and point domains at them for small amounts of time to see uh which systems beacon to which domains um so depending depending on that depending

on your infrastructure it could be it could start to add up and start to be costly again suggested reading if anyone's interested in doing this from a network defense standpoint but i think a lot of the same ideas could be done from the research aspect as well and with that i'll open it up for any questions

um some of them meaning parked on like a non-routable ip or park dead pointed at 8.8 or wherever um majority of them were parked i haven't gone back to look and do us a comparison um but when i spot checked some they had all been parked probably a month or two beforehand which is interesting considering they were still beacons because it makes you was trying to figure out why would they park it if they're still active beacons maybe they didn't need to clean it up maybe they were retasked with something else who knows just interesting yeah where do you draw the line between uh gathering some information about these beginnings and actually interacting with an implant

like if you're doing custom protocols so my thought process on it is no custom interaction at all because of different laws and different like if i let's just say there's a uh comp a beacon from germany and the data ends up on my system i really don't want to have any interactions i don't be able to send kill commands anything like that i want to be able to at least open up the data and see what see what commands it's trying to gather or issue um or send back to me but i do not want to be able to issue any commands i don't want to have that blooming over my head

so so the malware was configured for auto exfiltration so what it's doing is it says oh hey there's new word doc a new powerpoint i'm going to exfil at this time on this day and i was started looking through all the day i'm like holy crap in this case i i had a lot more data to the point where i had the person's email address had his linkedin profile all sorts of fun stuff but i feel sad i haven't reached out to that guy yet because i really don't know how to go about saying hey i have all your data that's not something i really want to do especially when they may not have a good

relation with us right now

um yes there was one um particular implant where um one particular victim where everything was ssl encrypted um and once i stood up that it wasn't validating the ssl cert once i stood up a generic one it didn't didn't have to match the domain at all it would just send the traffic create the tls handshake then i'd pop pop open the data yep last question uh no i haven't um again i just thought it was a side research project i didn't think it would get to the scope honestly if i continue doing i may do that or do some sort of non-profit and do it that way that way if people want logs i can figure out a way to share it with

them cool thanks