Ryan Hays - Weaponizing Splunk: Using Blue Team Tools for Evil

how was that good all right and don't leave yet we actually have a giveaway from rapid7 so if you put give your business card over to the rapid7 booth we're gonna do a drawing right now all right guys I'm Tyler uh with rapid7 and uh we very rarely turn down pen tests um so if you need a pen test let us know [Laughter] but uh sometimes it's usually like a legal thing where we just can't work it out and something like that happens so I know you left that for me to answer but yes um and of course uh we have met a split as part of Rapid seven I think that was covered a bit and and Expo is our

vulnerability management platform so there's a lot that we're doing in the incident detection response space as well but we'll uh Adam you want to grab one

looks like Michael Guido Guido and Guido Consulting we'll pull it back up we'll find him and uh we'll grab one more just in case um this is a 100 gift card the JC uh hold way the backup is Dallas Hannah so that's who won I don't think any of them here so we'll have one two thank you all right we got a few minutes uh before the next talk does anybody need a poster I'm trying to get rid of posters yeah all right one two up there I can't see upstairs three all right yeah thanks is that Kenny thanks Kenny there you go

were you serious Kenny I will walk one up to you I'll come up there and get it all right uh give us a few minutes and we'll set up for the next talk thanks are you leaving now actually yeah which is that the order yeah this was um so he was number one okay all right and uh if you're in uh pres Pub and your name was drawn find me all right and I've got uh I've got the gift card I'll be able to give guys

I'm gonna go over the radio

we'll see you sometime all right yeah thank you very much yeah anytime our wheel we got oh yeah you got like eight ten minutes for good good just the microphone

gotcha a couple bags yeah um I do not um if you put your okay it looks half readable

I'll be right back

foreign

foreign

was just looking for you oh Ryan how you doing nice to meet you is he still he's straight back hi you need me he just asked where you were when he walked back now do you need me okay oh I didn't know if you were still looking for him are you guys you guys go with walk-on music and everything

you know the Splunk I know the Splunk kind of deal with Splunk on a daily basis have a deployment of Splunk two deployments this one all right how many people are up there anybody upstairs you get blinded all right this one's for uh up top because I can't see you uh does anybody want a mug

I have to uh ask up top because we can't really see you from down here so I'll come up there I'll be up there in a minute

how are you man anybody leave a badge in nama does anybody want a second badge yep there you go oh sorry there we go oh mine's over here

probably

don't slice of town I used to live in Jackson so yeah yeah quite familiar with the area a lot of Technology hubs in Tennessee Nashville's got is growing it's growing yeah Knoxville's got it still open it just and then like once you get out of those two centers

thank you

so are you based up no I was so the tip about 10 11 years and then last year I moved over here I love working um you know I travel sometimes it's business sometimes

U.S space or international team we're International with like one two customers but mostly please

ready to go play one more music for Aaron you ready good all right all right so next up on the agenda uh we've got uh Ryan who's going to talk about how to weaponize Splunk [Music] [Applause] [Music] all right I did not pick the music but that works all right as you said my name is Ryan uh so I'm the director of security engineering at tbg security we're based out of Boston uh we specialize in offensive security engagements um prior Navy vet and basically professional hacker now and all my contact information will be at the end as well in case anybody wants to find me so where'd this talk come from so last year myself my team myself

were on an engagement for a company I guess I'll say um and we got internal access to the to the network and we started doing recon and enumeration and we found a Splunk box and as soon as we navigated to the Splunk box we saw that it automatically logged Us in with admin rights thanks guys this is awesome but what can I do with that really um so I can look at data I can find you know logs and stuff that's going in there and they had a lot of information so we're able to enumerate the company and figure out what was going on what boxes did what what's the DC what's the exchange server

what you know what what's what um and then we decided hey let's build an app so we built an app and the app basically gave us a reverse shell on the box but now I want some more access so let's build another app so then we built another app that we deployed out to all the boxes inside the inside the network that had Universal forwarders installed which in this engagement was almost every box in the network so I went from a pen test that usually takes two weeks to about an hour maybe two and we had domain access domain admin access and basically everything we wanted so it's pretty cool um and you know this is really not just

a Splunk talk well it is for this talk but uh the concepts that we're talking that I'm talking about here really could be applied to any any software that we're talking about that has this level of access you know McAfee EPO has this kind of access it allows code execution on boxes this this can be applied to a lot of a lot of different things not just Splunk anyways so what I'm going to be here for I'm going to give a brief introduction about what Splunk is hopefully everybody here knows what Splunk is if you don't I'm sorry but it'll just be brief um then I'm going to talk about some basic common misconfigurations that I

always see and that everyone always sees and usually it's going to be the same things you see against all the other software not just Splunk um and then we'll actually get into the meat so weaponizing Splunk there's gonna be three attack surfaces that I'm gonna talk about we're gonna be talking about attacking the server itself how we can actually attack the organization move laterally and then the final the final subject is not really talking about Splunk inside a customer environment but more Splunk that you're going to use yourself that you would have on your attacking machine or in some sort of attacker fashion and you can correlate and aggregate all that data together and then finally after I tell you how to

break everything I'm going to talk about some basic ways how to fix those things oh and at any time if anybody has any questions I can't see a lot of stuff with this bright light but raise your hand or start asking a question I don't mind being stopped at any point make sure that I get you know all the topics covered so what is Splunk it's a log aggregation tool um all these devices that you have on your network your windows boxes your Linux boxes your appliances everything they all generate logs and right now as a security instant response guy you have to go look at all those logs and then I have to look at all those logs and I

have to correlate the logs and say hey what happened at this time during this incident yada yada this that and the other Splunk allows you to bring all that together and kind of search it in one single location which is really nice but it also allows you to execute code so that you can expand it with as I said Splunk applications these applications can be written in Python Perl bash Powershell just about any scripting language that you want normally these applications are usually used to enhance your data or parse your data that you have coming in but as I said they can they can be used to execute code a couple other Concepts that we need to

talk about are the universal forwarders so all these other boxes that you have in your network your Windows host your Linux hosts those all need to have either syslog or you have to have a universal forwarder installed on them so you can ingest the logs that are coming in and send them over to your Splunk box so you can actually search them um these these Universal forwarders do allow code execution as well which really is the you know the center of this talk the universal forwarders are managed by what Splunk calls a deployment server all the the deployment server does is basically Aggregate and it allows you to configure and say hey send this app to these these computers

and these computers can be determined by operating system or hostname um yep that's about it for that and here's kind of what the deployment server looks like when you first log in and click on the deployment server you'll be presented with all the clients they're separated into what's what's called server classes so this is where you would separate things out by all my windows boxes all my Linux boxes here's all my exchange servers my domain controllers you know going forward anything that you need and as I said these are all determined by operating system or the hostname and here's where I decide what applications I want to install to which server classes everybody cool there

cool so the misconfigurations that we see obviously default password is the biggest thing that I run into admin change me as the default password for Splunk as of Splunk 6.5 it does force that password to be changed upon install which is really nice but you know nine times out of ten you do see password reuse so whoever installed your Splunk server is probably using the same password that they're using to log into their same Windows box um the older versions of Splunk as as I said during our engagement last year and I think that was 4.5 so it was quite quite a few versions ago but the older versions did Auto log you in as admin

which is really cool so if you don't click if you don't install SSL which is a one or two click button one is to check the box and one just hit save you should enable SSL otherwise I can run men of the middle attacks and you know obviously then your password really doesn't do you any good and then finally running Splunk is as a higher level privileged um and when I talk about Splunk I'm not just talking about the application itself but also those Universal forwarders that I was just talking about so a lot of times I'll see Splunk installed correctly installed as a different non-privileged user but all those Universal forwarders because they have to touch so many logs

and so many different logs for different applications nine times out of ten those either running as root or running a system because no one wants to configure the permissions correctly so that that Universal forward can read all those different logs which is really nice for us

so weaponizing Splunk as I said there's three attack surfaces first of all attacking the server what can we do when we actually log into the smoke server itself we can review the logs we can enumerate information we also if Splunk is installed incorrectly and running as a higher level user we have read access on every file on that operating system finally we can talk about the malicious apps that we can install it actually allows us code execution on the box when we want to actually talk about attacking the organization we can laterally move to all these boxes that have Universal forwarders now there's usually two different configurations that you run into when you're on a pen

test people will either have Universal forwarders installed on every box in the organization or they're only going to have them installed on boxes they want to see logs from which nine times out of ten is the boxes that I also want to attack because those are the high high level boxes that everyone wants to see the logs from in case somebody attacks you so it's kind of nice that deployment server is going to come into handy for us in that section and then as I said finally we're going to talk about attacking the data and that's you know if I install Splunk on my Cali box and I run nmap and then I start ingesting nmap

and I start ingesting responder and I start ingesting all these other logs that I have it gives me up-to-date information in real time and kind of decreases that dwell time from running a scan reviewing the logs and attacking a box so now I go from a 20 minute thing to a two minute thing so it's kind of nice so when I first log into a box let's start looking at the logs Splunk pulls in everything custom web applications the logs that I have coming in I mean every everything's here what can I start looking at here's some things that I would start doing so let's go to the search and reporting app let's type in username

I now have a full list of updated usernames that are actively being used during my pen test type in password if so if it's a development environment I see this all the time in development environments the guys are just going to pull in every log that they have which sometimes includes passwords ah um if you start looking at things like um let's look at window event codes for logins and logouts I can now start enumerating does this company run a 24-hour shop or are they nine to five what hours do they work so now I can I can kind of tweak my engagement based on those operating hours and then as I said as I said earlier you

can identify which which systems are doing what thing in the environment where are my domain controllers what are the host names what are the IPS where you know where are all these boxes at the applications that are being used and then oh just understanding what's going on so if if the system administrator has alerts set up or reports set up to look for certain things I know now I now know either disable those alerts or just don't do those things so I mean you know if you have admin rights in this box there's little that they can do to stop you here's just basic common things that I look for you know Etsy Shadow Etsy

password but let's enumerate the box that we're on so let me show you this this is a basic walkthrough so as soon as you log in hit add data let's go to monitor we'll figure out what we're going to look for so first off we're gonna look at Etsy shadow and we're going to hit next as soon as we hit next we're now looking at it at the Etsy Shadow so I can grab these hashes throw them on password cracker and now I have an account hopefully and as long as I don't keep hitting next none of this information is ever logged ever indexed and no one ever knows that I was ever here so this is really nice for if Splunk is

best configured and running as a higher level user so malicious applications as I said spunk can be expanded with the applications you can run python code you can run you know whatever operating system is actually running on the box what's nice with Splunk is it is packaged with the python interpreter so no matter what operating system this thing is installed on I can always at least run python so I you know when I built my malicious app I built it with python um it does do bind it does do reverse it does do basic shells it does do interpreter shelves and as I said this is all perfect concept it's highly detectable I'm not trying to hide it

it's basically as a proof concept for you guys to play with or to use on a pen test engagement where I really don't care if I'm caught or not well play there we go so initially we're going to set up a Handler well we're going to start metasloit and then set up a Handler but yeah

there any questions right now all right

so I said everything that I have packaged in the app is all python based so we're gonna use a python Handler for that or a payload

all right so now we have a Handler set up we're going to go back into Splunk if I could type there we go let's go back into Splunk now let's actually install our application so we're either going to log in or if we have somebody that is running an old version we're just going to navigate to the page and auto log in

we're going to install the app from a file because we have it downloaded already and all the source goes on GitHub and the links will be at the end so we're going to upload that

it doesn't require a reboot which is really nice now in 6.5 it does require this one permission change and I can't figure out for the life of me of what configuration file this comes from which would prevent this but change this one permission and now what this does is actually install some commands that I can run from the search and reporting app which I'll do here and I can specify what type of show I want reverse bind meterpreter basic and then where I need to communicate to if I am doing a reverse so we'll do a reverse interpreter show we'll give it our IP and the port that we want to communicate over and we're

going to hit enter the nice thing with the with the code that I've packaged with it is it does fork itself off Vice using all the resources from Splunk which would you know alert some administrator if they're watching so it does fork itself off and we do get our our session back here in interpreter so that's really nice oh crap I hit the wrong button sorry

everybody wants to download it everyone wants to install it well right now you can grab it from GitHub as I said and you know you used to be able to grab it right from Splunk base I submitted it I got approved it was really cool it was just a basic interpreter show it's awesome they approved it I thought it was really awesome and about 57 minutes later I got another email that said that I was disapproved so you do have to download it from GitHub and install that way um sadly that sucks but it would have made a lot easier instead of installing from file you can install right from Splunk base but hey whatever

all right so we have a shell in the Box we can exploit the box that way what else can we do with Splunk so the entire Splunk application is built on or has an API built into it um the whole front end that you actually navigate and use with with the uh web UI is all based on the Splunk API um all of the applications that are installed including our own has configuration files so if for instance one of these configuration files such as a Windows application would ask you for a username and password so you have to enter this username and password these are all stored in files found on the file system now Splunk does do this correctly they

do take your password they do encrypt it they do salt it the salt is unique for every Splunk deployment which is really nice and if you go in there and you look this is what you would see in a configuration file on a basic Splunk installation can't really read it can't really get anything out of it but if I use the API and about 14 lines of python I do get a clear text password as you see down here so this this application actually will Loop through every app installed on the Splunk Implement implementation and pull out every configuration that it has for it so for instance if there is a Windows application I would get the

username and a password for a domain user account now that doesn't always mean that I get a domain admin but at least I have you know some sort of domain account and I can escalate privileges from there

no this is actually clear text password right here no just a super secure password that I was using at the time I mean I was using numbers and letters no one's ever going to crack that all right so so we've we've attacked the Box we've gotten clear text credentials we've installed an app we have code execution we've we've pretty much owned the Splunk box itself let's figure out where we can go from here so as I said before Splunk has Universal forwarders and they have the deployment server that pushes all those apps out so again I built two more apps one for Windows one for Linux um and Splunk calls these technology add-ons and it's just a Splunk term it

really doesn't mean anything you could push out whatever you wanted so the Windows app is just a bat file that runs Powershell um you do have to generate your own Powershell code and put it into the application to install it so hopefully you do have command execution on the box and you'll see that in the video in a second the Linux one you you will have to make configuration change for the IP address um but basically it's the same shell that you're running for the initial app that we installed and all this is packaged as I said in in the initial application

so again we're going to start up Metasploit oh I'm sorry we're going to start with generating the Powershell code so I use the trusted Tech unicorn project I don't know if anybody's heard of it but basically it generates a Powershell payload for Metasploit I'm using it in this instance for reverse Metasploit shell

so that's what it kind of would look like you could go to any Windows box you wanted run this from a command line it'll get your reverse shell back we're just going to stick it inside of our application so this this shell that we're looking at is on the actual Splunk server we're going to update the application that we have and make sure that they can be deployed out

so let's copy out the windows and the Linux app and let's move them to the deployment section of Splunk

and now let's edit the Windows app to to inject that Powershell code that we just generated

all right so our Windows app is set up we're going to edit our Linux app and just change the IP addresses to make sure that they match what our attacking box is

all right so we have our Windows app set up we have our Linux app set up now we just got to push them out to all of our clients well I guess we need handlers first but then they need to get pushed out to the clients

now as I said I'm using basic Metasploit code here any of this any of these payloads can be swapped out with custom stuff that you guys are using for your pen test or your red teams if you want to be a lot sneakier this is just basically perfect concept stuff so I could show off the capabilities of it so we got our Windows Handler set up let's do a Linux Handler here

all right now we can go back to Splunk come on there we go so we'll log in with those credentials or Auto log in hopefully one of the other we'll look at our Splunk server and all we're going to do is we're going to edit these and assign them to the actual classes that we have set up so I already have a class set up for Linux for all our Linux payloads and then we'll set up the same one for Windows and what this basically says is any Windows box on the network deploy out this application and execute the fun part with these configurations is I could say hey run this payload every two hours

so now if somebody catches me kills my payload well in two more hours I'm going to get another shell back or you know you could run it even more or even less whatever time you really want you can all you can configure that with the application that's deployed out so we got our first one back and the second one takes a second come on

go back over there it's coming there it is and there's a second payload so now I mean in this instance this is a demo server this is just two boxes I had out there but if you had 60 boxes to you know in the network or 100 all these shells will be just coming back and bring it in so now you can go from here run Mimi cats you know dump passwords whatever you needed to do from there I mean you now have access to every box on the network all right so we've attacked the server we've attacked the rest of the organization what else can we do with Splunk so as I said before install

Splunk on your Cali Box start looking at ways you can ingest all those logs that you have nmap runs it has XML output Splunk reads XML pull in all those logs every other tool you run has some sort of output or some some sort of regex that you can build to read in those logs for instance right here I was running a Wi-Fi pineapple so in real time I can deploy a Wi-Fi pineapple out I can get all the ssids all the clients that I've already attached to my network I can I have pretty charts that I can insert into my report to make all my customers happy um and this kind of decreases that well

time as I said from looking at the web interface and clicking around and refreshing all that stuff comes back in real time right here this is just one one thing that we've built and we have another one for responder which is a man in the middle tool that um hopefully you guys have used because it's awesome um in real time again I I would have all the responder logs coming back so I know how many credentials I captured how many users I had to blur some of these out because this was actually a real engagement I was on um how many usernames I have what usernames I have what types of credentials I'm looking at and then down

at the bottom here would actually be the usernames and password basically so that you could dump out and you know hand over to hashcap

some other thing actually some of the things that I wanted I want to actually expand on with this because this is all still in development is I want to start adding some of the contacts menus so you can right click on some some dashboards actually get you more data or you can actually execute action so I want to start adding contacts menus to this application that would allow me from here to just okay send those credentials right to our password cracking box or start executing XYZ attack or a d off the attack for the Wi-Fi pineapple all from this one interface so again it would decrease that dwell time all right so mitigating actions obviously updating Patcher software

enable SSL change your passwords don't run it as a root I mean it's all the same stuff to do here every single time Splunk has a really cool documentation it's about 30 or 40 pages long really good at documentation and I don't see a lot of people follow it every now and again actually I gave this talk a couple weeks ago at Nashville I had a really smart guy there apparently he's followed this documentation to a t but I don't run into that very often so you know look at the documentation follow that it I mean it's right in front of you and so that's really all I got that's my contact information as I said I work with tbg we are hiring

if anybody wants to talk to me about that afterwards I'll be at the bar with an open tab we can talk about it there or if you just want to talk about Splunk thanks guys

[Applause] oh what's up

so the question was has this technique been caught by the client um in the engagements I've run no but this isn't a new technique people have talked about it before they just haven't talked about it in all the same contacts that I have so I have I have run into people that have run this before but yeah no no one's really caught it on the engagements I've been on and you know one thing to think about is again if we are admin on the Splunk box I could literally just erase the logs that are being generated for anything that I'm running in the environment anything else I can't see upstairs so if anybody's asking just go ahead and ask

yes all right cool thanks guys foreign