Weaponizing Splunk: Using Blue Teams for Evil

all right guys so i'm ryan i'm gonna you talking about weaponizing splunk um wow these colors are gonna suck with you sorry um anyway so i'm the director of security engineering with tpg security uh we're consulting firm based out of boston doing offensive security um i got a few certs up there so hopefully that shows that i know a little bit about what i'm talking about so this talk uh last year my team and i were on a pen test we got internal access to the network uh we're doing a little bit of recon enumeration we stumbled across a splunk box uh upon navigating to the website it automatically logged me in with admin credentials it was like that's awesome

thanks guys um but how so then we took that we thought about it how can we actually leverage this into something that i can you know attack this box or attack the rest of the network so we came up with a custom application that we installed gave us a shell on the box and then we thought well how can i get even further well in a few slides here i'll talk about the deployment servers and universal supporters and all this stuff but you can basically push out applications to all these universal forwarders that we have inside a customer network so on top of installing an application we also deployed out an application to every box in the customer network

so we took you know a two week long pen test in about 30 seconds so that's pretty cool anyway so what am i going to be here for um i hate this standard mic this sucks i like to walk around and the lights got even worse now now no one can read anything other way with the lights no okay so we're going to start out we're going to talk about briefly what splunk is hopefully everybody at least has some idea what's going on with splunk um if you're here i'll talk about some of the misconfigurations that we've seen um usually just the basic stuff nobody changes their freaking password things like that um and then we'll get

into actual the meat of the talk weaponizing splunk there's three attack surfaces that we'll be going into attacking the server itself attacking the rest of the organization and then how we can actually leverage splunk in as almost like a c2 so actually if i install splunk in my attacking machine and collecting the data that way um and then i'll wrap up the talk with kind of talking about how you can mitigate this in your own environment and since um actually if anybody has any questions while i'm doing this go ahead and raise your hand and we can address them as they come up it'll just be easier than waiting till the end so what is splunk as i said hopefully

you guys have some sort of idea because i'm not really going to get into all the details of it but it is a log aggregation tool all these devices that we have in our network windows machines linux boxes net firewalls everything that we have splunk's gonna take all the logs and alerts that come out of those devices and put them into one location as i said earlier the universal forwarders these allow you to collect those logs off all these disparate devices if it doesn't send out syslog messages you can install universal forwarder on it and then pull in all those logs so things like windows logs that are normally what you see but any other custom web applications if

they don't syslog out you can install the universal forwarder tell it where the log is and it'll collect them what's really nice about these is they do allow command execution on the box so if you need to run something like on a linux box i want to run netstat and collect the net the connections that are going on my on that machine and send those to splunk so that i can review those over time um it will allow that all these universal forwarders they're actually sent out through what's called a deployment server um the deployment server allows you just basically a collection of all these machines and you can separate them out by uh machine name or uh by the

operating system that's how you can categorize them so the first screen you would see if you go to the deployment server you would see all your list of clients that have the universal folder installed you can click over and you can actually split them up into what they call categories or classes so for instance if i had a windows application i only want to install that on windows machines not linux machines so i split them out and say hey here's all my windows boxes based on operating system only install those apps and that's what the apps the last section here is apps so those are the basics of splunk as i said there's a ton more to look

into and you actually could figure out with it um if anybody has any questions there go ahead and ask them otherwise we'll start getting into the regular stuff

yes all right as i said the basic misconfigurations that we see i'd rather the dark let's just go dark um nine times out of ten i usually run into an organization that didn't change the default password um it's admin it changed me obviously now as of 6.5 splunk does force a reset on that admin password upon installation but you know who upgrades all their software all the time so we still do run into a lot of older machines where the passwords haven't been changed um the older versions as i said that on our at the beginning the installation that we ran into i think it was a 4.5 installation and this was just last year

so that obviously shows people are not upgrading very well those automatically log you in if you didn't have an enterprise license um i think that was really the main thing um and then a lot of times i don't see ssl turned on it's one check box and a save button really to turn that on so you can run man of the middle attacks if you you know if you were seeing that and finally the universal forwarders and the splunk box itself is usually running as a higher privileged user than it really needs um i see more the actual splunk server software running as a an underprivileged user which is good but those universal forwarders nobody

really wants to configure the permissions on every single logs that the universal forwarder can read them so nine times out of ten i run into those boxes those are all running as root or system which is awesome for us so actually weaponizing splunk so upon upon first getting into a server that's what we're going to talk about what when i when i get into the attacking server part what can we do when we when we get on that box so first of all we want to we want to increase our intelligence so let's look at the logs that are already in there we can identify operating systems i can identify the domain controllers the exchange servers everything else in the

environment then if splunk is running as a privileged user account i have read access to every file on the operating system let's pull an etsy shadow let's pull in you know etsy password whatever file you can come up with let's pull it in let's look at it and if you do it the right way that doesn't get logged so that's awesome um if we have administrator access we can install applications which can be any application which can run any code that we want so if you have a python shell or if you have anything you can run it and then finally we're going to talk about extracting the data which is you know all these configured files are

on there let's let's look at the data that we can pull out of splunk that changed attacking the organization we're going to get into actually how we can utilize splunk to laterally move throughout the network and compromise everybody and then the final part attacking the data we're actually going to use it for c2 so that's going to be cool [Music] so reviewing the logs all the logs since this is collecting all the logs as i said custom web applications windows logs linux logs everything's in here let's start looking for things go to the search application type in passwords see what comes back if it's a custom web application a lot of times i've seen developers pull in

the passwords why the heck not no one else needs to see this except for me why would anybody be looking for passwords it's pretty cool um we can get we can get a good sense of the hour's operation if we started looking at maybe let's say the windows event code for logging in i now i now have a list of users that say okay this shop runs nine to five so i see all my logins at nine i see all my logouts at five and i can actually probably pick and pick out where lunch is too because a lot of people log in and out there too um or maybe i can see if it's a 24 hour

shop but now i have a list of users these guys run the morning shift these guys are an afternoon these guys are a night shift so it kind of gives you a lot of intelligence about it about a company so let's actually go to the demo here oh come on why are you not working there it goes [Music] wow that looks like crap in the light hold on let's see here let's just step through it so the first stop here if you click in the middle there you're going to add data

so if the splunk if the splunk application itself is running as root i'm going to look at monitoring a folder or a file so let's click on monitor can we bring the lights down just a little bit so we can read some of it otherwise it's not going to make any sense so on the left over there we're going to look at files and directories wow that sucks so i'm going to look at etsy password right here and then i'm going to hit browse and it's going to actually show it to me crap well if we could see that it would be the etsy password file so i'd now be able to enumerate user names

there um so as long as you don't pass that screen right there and hit keep hitting next to finish it this data didn't get indexed it doesn't get logged and i i now can pull this down and look at it locally on my machine if i need to pull up hashes i can start cracking that but no administrator would ever know that i was looking at any of these files so no alerts are going off yeah that might get a little better hopefully all right so that's kind of reviewing the logs everybody's cool with that we're good all right so let's start installing malicious applications so i built an app called splunk shells it's it does a bind shell or reverse

shell it's just interpreter and it's just a perfect concept basically to show off the capabilities hopefully this one looks a little better since it's not white so let's start with metasploit let's start up a handler then we'll start looking at the

application

so we're setting up a handler right here to catch the shell

so our handler set up now let's go actually to splunk so splunk applications are basically just a zip file of folders and scripts that are running browse we'll grab the zip file and then we can install it so what this does is it basically adds an application to splunk and gives you a command line that you can run through the search and reporting application so once we install it we don't have to do a restart on the box so no again that doesn't alert anybody we'll just change one permission set that i don't think i can do in config files at least i haven't figured out which one it is so you do have to set this and this is

only in 6.5 so hit save and then we'll go to search and reporting and actually execute the reverse shell

so from the command line i can specify do i want a reverse shell do i want a bind shell who do i need to talk to and what what port are we going to communicate over so we'll do a reverse show i want to do a interpreter shell since i set up that that handler already there's our ip and there's our port so what that what the show does in the background is it's actually going to fork itself off for the splunk process so that it doesn't use up all the resources while i'm still communicating and attacking there and then we have our connection there so that's awesome right everybody wants to install it i have it on github um

that's a crappy link basically weaponizingsplunk.com has it all um and then i also submitted it to splunk for the splunk base it was really cool i got approved it took like three days um they were really cool about it it was awesome so instead of going to browse for file you you could just install it right from splunk base except for about 57 minutes later i got rejected so i don't know who got in trouble over that i've asked but they won't tell me but i was it was fun for a little while um i if you go i'll have them all at the end but if it's github.weaponizingsplunk.com and that'll have about everything it

actually will have all the slide decks and the videos and stuff so you can actually see them in proper lighting so extracting data all these applications that that are installed on on splunk you know active directory rapid7 has an app everybody has an app and you have to install them they have to configure them different ways um so some of these apps for instance the active directory application needs a domain level username and password so what you would see when you're configuring that would be a screenshot like that i'd have to install it you know give it a give it a domain give it a user and give it a password now splunk is really good about it they

do store these passwords encrypted and they are there's assault and everything that is unique to each splunk deployment as you install it so if you go look at the file that's what you would actually see in there but if you have 14 lines of python code you can actually get clear text credentials damn can we turn the lights down a little bit more dang that sucks yeah um so about 14 lines of python code and again it's all on github you can run that you'll run it with your admin credentials that we obviously already have and it loops through every application that's installed and extracts any information that it has in there so if you could see that we would have

our username and our domain level password in cleartext and that works via the splunk api the splunk api actually has access to all these configuration files the salt and everything else that it needs to decrypt those credentials so that when it does actually need to use those it can't communicate with the domain controller luckily it's really nice and it'll actually give it to me as well so now that we've already attacked the server we have a shell on the server we've extracted some passwords let's actually move laterally in the environment and start working on other computers so within the application the splunk shells application i've built two technology add-ons and again these are just perfect concepts

they're using basic interpreter code i'm not trying to hide from anybody if you want to use them i would highly suggest substituting your own custom payloads in there and it's literally just swapping one python file for whatever else you guys have so the windows add-on uses a batch file and just runs powershell code um i'm actually using the trusted sec unico unicorn uh program to generate that and just general generates obvious gated payload for a reverse my temperature shell and the linux one is just a basic python one and again i'm not trying to hide anything it's all just a proof of concept so let's see what that looks like

all right so we're setting up another handler let's just skip through this a little bit

uh inside each of these technology add-ons there's there's a couple things that you have to change one way the other all right we'll just go with the dark let's just just keep going up and down it'll be a party we'll have fun so there's two things you got to change inside these payloads since they are just static payloads you're gonna have to change out your ip so give it your attacker ip give it your attack report and you do have to substitute in the powershell code because i didn't it's it's it's not set up for you know it's set up for me to use or somebody else um so you swap out your powershell code here

we'll swap off the linux payload i'm just changing our attacker ips let's actually get got our handler set up here let's actually go to deployment all right so here's it here we are in the deployment server so we have those applications they show up here it's literally just clicking edit telling them what servers i want to install these these things on all of them obviously and then it does all the heavy lifting of sending those out and actually running the code what's really cool here is there's a configuration file for how many times you want this to run i'm obviously just running it once but i could say let's run this every 30 minutes or let's run this every two

hours so if somebody does detect me and kill me i now have a way back in it's going to call home here in a couple of couple minutes or an hour whatever you set so there we got our first show back and then here in a second we should get our second one now it will take a little bit of time depending on the splunk server configuration how how often the agents check in on everything else that's that's why it might take a little bit longer but since we're an admin we can change all that too and so there is our seconds so now we have two shells in the environment but i mean literally if

there was 100 machines 200 machines however many machines until metasploit crashes really you could just keep getting shelves back i mean i've had 60 or 70 callback in my matter of five minutes so so that's really the cool stuff now i actually want to talk about the command control so we're going to be early for lunch would be good um so i built another application i just call it weaponizing splunk and basically what what i'm starting to do and this is still a work in progress is when we when we do a pen test we you know we have to run nmap we have to run nick though we have to run all these different tools and every

single tool generates some sort of output so i'll do a pen test and i have this really elaborate directory structure so i can organize all this information so when i come back to write the report that's going to take me another week and a half because i have to go through all this stuff and all these all these all this data what if we just shoved it all into splunk so right here i have a wi-fi pineapple app um is what we're looking at immediately in real time i could run i could run my wi-fi pineapple run my attack on there but i have a universal forwarder installed so all those logs are coming right into splunk so if i have splunk up

and i have my pineapple running in real time i know all the clients all the ssids that i'm seeing i have a nice little pretty chart if i wanted to send that to the client as well all that stuff's already given to me in real time there's no parsing logs there's no going back and you know it actually speeds up that dwell time from running an attack and actually compromising moving laterally into the environment and another one that i have is responder so responder is the man in the middle tool that actually will capture hashes through throughout the environment so in real time again i would have you know i know how many credentials i have

another user names i know the domains they're for i know the hash type and then down at the bottom i would actually be able to just extract in a nice little text file all the hashes really quickly so i can send it over to my cracking rig and again it just decreases that dwell time for you so mitigating actions obviously update update your software enable ssl change your passwords in 6.5 it's really nice because they actually have two factor authentication now with duo so you can do that all you have to do is change your password enable two factor and that literally stops everything in here that i'm talking about unless obviously there's password reuse or

some other way that i got that password um don't run splunk as root not even your universal forwarders i know it's a pain to configure them but take the time it makes my job harder segregate your splunk server on your network don't obviously allow all the connections to it from everywhere i i yet have seen haven't seen anybody do that i did have a guy last weekend in nashville that was talking about he has it set up the proper way but he's the first person ever it'd be awesome let's actually see it but uh segregated on the network don't allow the connections um and don't run it as root i mean that's really it and then

there's a really nice guide it's about 30 pages long but go through that and actually configure splunk correctly any questions i mean that's pretty much it there is the information as i said github.weaponizing or weaponizedplunk i think i have both domains so they're both go to the same place all the slides all the videos what's up

i do yep all you have to do is the same way we installed them all you got to do is delete them and smoke actually do all the cleanup for you

yep yeah probably in the in this payload i mean as i said this is all basic payloads these should all be detected i would hope but you know yeah these are all basic payloads i highly suggest if you're using this in a real real engagement i would swap these out for you know custom things that you you guys run with your team

yep

exactly i i keep forgetting to add a slide about that but yeah you could do cleanup as well um if there was anything that lo that was logged obviously these shells that are executing on the box would all generate windows logs if those are all getting pulled into splunk yeah you could just do a search for it pipe delete and now all my my traces are gone as well now it doesn't remove it from the box itself just from splunk so if somebody actually did forensics on the box they would still see that stuff anything else all right well that's all i got i've i've built some reports for some of our customers that we do splunk with but

i don't usually see that happening for people um so we do have some reports for um detective