Blue Team Fundamentals

[Music]

hi everybody so I didn't start out in information security I actually started out as a lowly system administrator but fortunately was at a company that once I started to express some interest in security they gave me some room to grow there and as soon as I realized what was was going on I was like I need to read everything that's going on so I started looking at information security news and quite frankly I was overwhelmed like because we're looking at people who are figuring out ways to exfil data from an air-gapped computer wrapped in a Faraday cage and somebody knows the precise JavaScript parser used in a Windows Defender to trigger a bug just by dropping on a

file and I was sitting it was like do I need to become the expert of all of the things in order to make actual security work and what I would have found as I've been in here longer is that while those shiny types of news stories might be what gets attention that what gets the biggest bang for the buck is security fundamentals it's it's it's doing the day-to-day things that are going through so for those of you who are relatively new to information security great this is an awesome talk for you what I hope you get out of this is realizing that you don't have to know all of the things in order to be good at information

security for those of you who've been in this space for a while nothing that I'm gonna talk about here is going to be mind-blowing all right but I want to explain going through this by documenting actual attacks that have happened against me and my company I think the Red Team side of the equation does a really great job of sharing proof of concept codes or documenting vulnerabilities but that blue side of the equation of actual defender experience I don't see as much of and so I wanted to do more sharing with the community so this is not selecting the best post quantum TLS cipher suite kind of talk I'm so glad that there are people in our information security

community that do that kind of thing and I'm much more glad that they make nmap scripts and various scanners online with simple little letter grades so that mere mortals like myself can be making informed decision but this is gonna be a talk that's about fundamental so I'm Benjamin I do security at Lending Club and what we're gonna do is we're just gonna walk some actual attacks that I've needed to defend against we're gonna start with the broad in general and then we're gonna narrow down into some things that are specific and targeted all right so throughout this here's my three fundamental questions here's the three fundamental things that I want you to be asking yourself when you see new attacks

first is can I see it - can I throw away what is obviously garbage okay and then three can I make whatever is less more expensive for an attacker all right can I see it can I throw away garbage can I make it more expensive alright that's our framework for all of these attacks so let's start out with some door knockers I call them door knockers some other people call them spray-and-pray the basic idea here's imagine if you would the worst home burglar known to man they can't pick any locks they can't disable any alarm systems all they do is go to every single house in the neighborhood knock on the door to see if

somebody's home and jiggle the handle to see if it's locked and if the door is locked they'll just go on to the next house but if nobody's home and the door is open Oh we'll go inside and steal your TV same idea here there is absolutely zero customization involved in these attacks it's hitting absolutely everyone with a public address on the internet just to see what's happened all right so this one came from Hong Kong and before I show the actual attack I want to talk a little bit about hello world anybody with anybody kind of programming experience the first time you might be learning a new language you might start out with something that

looks like this doesn't do much good just shows hello world on the screen nothing super fancy like there's no multi-billion dollar startups disrupting the hello world industry but it just is a little simple thing that says okay I know what I'm trying to do actually works and in my experience one of the things that I see is an attacker hello world is trying to get or cat out at sea password and I think there's a couple of reasons why Etsy password is a is a happy choice first it's in the same location in most major Linux distributions and second of all for the system to actually be working it has to be world readable so it doesn't matter

if the vulnerability they're trying to exploit gets them root on the box or some low-level privileged account they should still be able to get Etsy password also if they do get back they get a bunch of useful information that's helpful for identifying and fingerprinting what's going on it's gonna be a loser list and you can figure out things like oh is an Apache web server installed if the Apache web server is installed there's gonna be an Apache user in here and is that user called dub dub dub - data is it called Apache - is it called HTTP that lets you know how is this in a boon to system maybe this is a Debian based

system it's just a really simple like nice first step to see if it works alright so let's take a look at the attacks so this is the start of our attack here it's a really simple directory traversal so what you see there that dot dot back can you see the red in there somewhat okay it's hard to read from this angle but glad you guys can see it so dot dot slash just means hey go back a directory and so the idea of true directory traversal is hey if the person's serving up a website hasn't locked things down maybe he'll back back back back back back back all the way to the root of the

filesystem and then see you know arbitrary config files underneath this totally doesn't work by the way you can fire this against Lending Club nothing's gonna happen so of course when it didn't work the first time unfortunately they didn't just stop there they moved into iteration or fuzzing which is hey the first thing didn't work let me try a bunch of other things changing just one little tiny tweak and see what happens so if this doesn't work let's try it as a query parameter which also doesn't work let's try it against a PHP file which totally doesn't exist at Lending Club no PHP in Lending Club also doesn't work let's try it as a post instead of

again this also doesn't work let's URL encode the slashes which doesn't work let's add extra slashes which doesn't work let's send the poop emoji which doesn't work let's try this without any dots at all which doesn't work so and I don't even know what this is so also doesn't work so at this point a couple hours in we had blocked over 34,000 requests from this particular IP address at which point we said dude you're just dead to us and we blacklisted them and they kept hitting us about a hundred thousand more times afterwards maxing out at 93 requests per second which is actually pretty nice so remember I said I had three questions again visible can you see this if you

get an attack like this where someone is fuzzing and trying lots of things over and over again are you able to see that that's happening do you have alerts going off of the velocity if your whaff is normally blocking a hundred things minute and now it's a hundred thousand maybe you should know garbage alright if one IP address over the course of a couple hours has five signature hits ten okay maybe maybe that's fine maybe that's I don't know 34,000 that's garbage get rid of it and can you make things more expensive one of the things that we do at Lending Club is that when we blacklist an IP rather than just setting that connection resets the

entire time the behavior is actually identical to what you see if you get a wife block which makes it just a little bit more confusing as to whether or not the things that they're trying are going against our blacklist or if they're actually iterate against the web it's a tiny thing makes it a tiny bit more expensive it's good to go all right next one and this is my favorite door knocker from like the past six months absolute favorite came from Croatia and in order to talk about this I need to talk about CVE 2017 5638 to which some of you might be thinking wait a sec Benjamin didn't you tell me this was a fundamentals

course nobody told me I was memorizing CVE numbers you don't have to you might actually know this by the more common name of hey that Apache struts buggy thing that was used to hack Equifax all right so let's take a look at what how this works so Apache stress had an issue with the multi-part upload parser and it didn't actually matter if you were using Apache struts to actually upload files and the issue kind of came around this bad air handling of the content type or similar headers and it was is a sweet bug just straight-up remote code execution all right so let's walk through how this works so normally when you're going to upload a file the client

says hey I'm gonna send you a file server says great he says okay what type of content type and this client can say how it's an image JPEG and the server says great I know what to do with pictures or hey maybe it's an mp4 and then the server says hey I know what to do with that but if the clients sending the something that they just totally don't understand the server's gonna say I have no idea what Pizza is I can't handle this error do not pass go do not collect $200 the Apache stress attacks worked fairly similarly it starts out the same I'm gonna send you a file great and then they send a content type header with a

bunch of other stuff in it and includes a little this line here that command equals in this case I've said cat Etsy password and then the server says error I have no idea what type of file command cat Etsy password is and then it would run command cat Etsy password alright so let's take a look at an example Apache stress thing was fired ants that this came at us from Sweden now I have a certain intimacy with his vulnerability a certain understanding of its nuance that only comes when you get paged at 3:00 in the morning and are firing hastily put together curl statements at production going wait are we vulnerable to this so if what's going on here

doesn't immediately jump out to you like it might to me let me go ahead and bold what I think the important things are here so first off we have that command stanza that command I have config ifconfig is a regular Linux kind of networking gives you Becky an IP address some general Network information and then you got the user agents here which points back to a github repo which tells me that this is actually kind of boring this is a script kitty this is someone doing git clone and run because quite frankly Apache struts exploit attempts are boring these are shiny graphs from purva basically saying hey once this became public we saw it everywhere all

the time for me if I'm blocking 50 Apache struts X points as Hempton today that's Tuesday that's boring so if you're gonna be my most favorite door-knocking attack in the past six months you gotta go a little bit above and beyond so let's take a look at it this one is what happened came from creation again if you don't see what's going on here let me bold but I think the most important part is it's right here here's the command this guy's trying to run where they're gonna echo into this crontab something that says hey every 20 minutes or so I want you to W get down this logo jpg file and pipe it straight into the shell

which of course parked my curiosity of what the heck is this logo dot jpg file so I fire up a sacrificial VM go ahead and download it and surprise surprise it's not actually an image so this is a hastily put together zero formatting or code think bad code hygiene by the way I'll cowl together of a bash script and being a former system administrator having to take over other people's terribly maintain bash script I have a formula for this which is you'll swauk through this actually add some indentation in code stanzas and some commentary to see if I could chunk this out so that's what I did when I saw this so alright let's take a look at what I

think is going on here so the first step is removing what I think is a previous version of the config file remember the attack is trying to drop something in crontab that's gonna go out and reach for updates every 20 minutes so if this guy found a better way to do this then they changed their logo to jpg file and clear out old config start the new stuff then it's gonna go and list a whole bunch of running processes and look for a lot of different keywords except for this to her Joseph thing which we'll see come up later and anything that might be related to that we're just gonna kill - 9 okay then it's gonna see whether or

not the process that this guy's trying to start isn't already running because if it is then no need to do this again but if it isn't alright then we're gonna go forward so we're gonna have this turgor sift comp file might be a previous version of there we're just gonna remove it get rid of it we're gonna download this K worker comp file into this VAR temp turgor serf and use both Carroll and W get just in case one of them is an installed and then we're gonna make room for this VAR temp at D space or they're gonna swap this by an area that's going to be coming on later on and if I look later down I see

they're removing a VAR temp sshd my interpretation of this is that a previous iteration of this they drop the binary that they want to put on the infected system in VAR temp sshd but sshd taking a whole lot of server resources is kind of suspicious at D is really similar to the app daemon which does cron jobs style things and it could theoretically be a job that's resource intensive so it's a little bit flying more under the radar all right but wait a sec k worker comp what the heck is that thing so of course I go download that and that is a JSON with a little bit of basic configuration long username a third IP address

terrible password some of you might have been seeing some of these types of attack might know what's going on but I still didn't at that point so I fired up a web browser and went to what's that third IP address and it says pool dot my necks and marcom something something something okay what's my next two marcom ah it's a Mon Aero mining pool now I have an idea what these guys are doing so let's take a quick side Channel let's talk about Manero so for those of you who may not be familiar with it Manero is a crypto currency that's very similar to Bitcoin you mine it by using computational power but there's a couple

of things about monaro that make it particularly interesting for attackers first off while bitcoin has a 100% public blockchain where you can see every transaction going back to the beginning of time Mon arrow obfuscates it they claim that it's 100% secure I wouldn't bet my life on it but they at least take steps in that general direction also with Bitcoin if you're not mining it off GPUs or Asics and application-specific integrated chip you're basically wasting your time but Mon arrow can still be meaningfully mind through a general purpose CPU and this leads to a couple of interesting things so you may have heard of coin hive coin hive is a company that started up and

what they say therefore is to give people an alternative to advertisements so that instead of doing advertisements in bed this little bit of JavaScript on your page and anyone who visits you will mine Mon arrow for you in the background what I see it much more commonly used as is a one-stop quick way to monetize compromised websites you may have heard about texthelp's browsealoud they texthelp makes this kind of little JavaScript snippet that you can drop into a website and it will help read out the text of a page it's really helpful for making sure that your websites accessible to people with visual disabilities so it's used in a lot of government websites and someone had

compromised text helps servers that were serving up that JavaScript and they didn't change anything with the functionality whatsoever it would still read out all the text just as you needed but then they add in like four lines that said oh and by the way mine monaro for me and that was at like five thousand US and UK government sites I have a link once you get the slides if you're curious about having more that so anyway we now know what this guy's doing and the rest of this kind of goes a little bit faster now so this guy's gonna take a look and see whether or not there's the AES encryption set inside my CPU download of

different version of the binary depending on whether or not that instruction sets there again throwing it into VAR temp a TD and they're not greedy they're only going to take a little bit more than 50% of my CPU cores and fire it off and go make me some money so in all what's going on here from from head to finish it's gonna kill any process that it think might possibly be related to another Mineiro miner remember this script is delivered through literally one of the easiest remote code execution vulnerabilities seem to man so this person is able to do it they think oh wow other people might be doing the same thing they're gonna download a config file to pay to a

specific wallet check the encryption abilities of the CPU download the right binary and bam away you go okay so same three questions is it visible key throw away garbage and can you make it more expensive so is it visible do you have a wife is it turned on is it getting regular signature updates most major waffe vendors had an apache struts signature available fairly shortly after it became public but whether or not that signature made it all the way down into people's infrastructure is the difference between being able to see something like this or not being able to see something like this and okay so I have here slight increases in costs do a packet attackers

if you got ozone mine I have a pet peeve of mine I'm gonna go on a little rant here unfortunately you guys aren't any choice in the matter because I made the PowerPoint slide deck let's talk about HTTP host headers all right so HTTP host headers everybody doesn't this was the attack that we just looked at they put an HTTP host header that said host Lending Club com Googlebot does it post Lending Club com curl does it host Lending Club com why do they all do this well because it's part of the RFC spec the HTTP 1.1 technical implementation says everybody's got to send a host header if you don't send a host header you're

supposed to get back a 400 error bad request no cookie for you and it's so important that they repeat it later in the same RFC it says hey everybody it's super important you should have a host header and why because guess what it used to be that back in the old days one host when IP address just had one host so you could easily infer what was going on but now we have IP addresses with lots and lots and lots of hosts so everybody please put an HTTP host header we figure this out in June of 1999 all right before we figured out whether or not y2k was gonna get rid of all of the

nice cat pictures on the Internet we had the HTTP host header okay so why am i passionate about this why am I so basics okay the number one most active filter in my production traffic in my environment is off the HTTP host header if someone is coming at me and they either don't have the HTTP host header set or they have it set to an IP address I don't know what they're doing but it's garbage and I'm gonna get rid of it and I block millions of requests in a week millions of requests let's take a small sampling of this garbage but take a look at this host quad 0 not even an IP address even if it was not one that I

control oh look at this someone was searching at Bing for Amazon and somehow added at at non-existent PHP myadmin setup script garbage hello look at this cools dot PHP and we're just gonna add in as a query parameter something to download and run on a machine super simple command injection of course once again I'm curious so I set fire up a sacrificial VM and I get a Chinese 404 error page darn however security is a community and my IP attackers IP address shows up in iOS e list the Trend Micro blog post surprise it's another Mon arrow minor so what they found is that the hackers have mined with a look at this okay cools PHP mr. and mrs. cool SAP PHP

with this stupidly simple command injection made at least 74 thousand dollars at least 74 thousand dollars as a March of this year alright anyway HTTP house headers throw away garbage and the last point here hey cryptocurrencies are changing the game alright if you go to RSA you're gonna find a whole bunch of vendors that are gonna say oh look we've a I a the blockchain to innovate and disrupt all of the things I don't know anything about that but I can't tell you what the blockchain has disrupted and innovated is the market for attackers changing compromised compute into actual money it used to be lets you compromise the server you'd have to go steal creds or

turn it to a DDoS botnet or do something to us otherwise extort maybe ransomware cryptocurrency you just needed to terribly put together 50 line bash grip alright so that's nice those are all general who's actually targeting stuff at us so here's some stuff with some basic customization again nothing wild here spearfishing somebody looks up what our MX record is says Oh guess what they're doing office 365 we're gonna give them an office 365 login page take a look at this inside it shows that the credentials then get popped off to another site and then they reload the actual Microsoft Office 365 legitimate login page so that the user thinks that they've just fat-fingered a similar attack against our employees

once you figure it out we've hit office 365 it's dead simple to figure out that you just enter in any at Lending Club com email into the office 365 sign-in page you're gonna get directed to octa sam'l surprise not that hard and we saw a dictionary based attack that looks like they use LinkedIn scraping to figure out what the specific valid Lending Club com emails were and then we're just kind of using a dictionary attack and this it lets live very identifiable in the logs low and slow so they don't actually lock anything out but same number of failures across all of them and in the cross same time paper and from Germany which we

don't have any employees there all right same questions can you see it can you get rid of garbage and can you make anything that's left more expensive hey do you monitor failed employee logins do you see when there's a spike do you monitor failed MFA changes can you see those types of things uh and so to clarify is the glut queue here I think fishing is gonna be something that's going to be part of everyone's experience until the day we all retire and hand off our problems to somebody else and I think there's some sort of there's an undertone perhaps in some parts of the security community that says oh users are dumb they'll click on

anything and I think the real lesson is how can you make sure that it's safe for your user to click on anything and so part of that is making it more expensive for attackers how can you make it so that it's more expensive for your attackers if let's assume that they're gonna get a valid set of crits that there's someone somewhere is gonna click on a phishing link how do you make that more expensive at mfa anywhere you know add restrictions that take a look at whether or not the times and locations an IP address actually make these another simple one do you guys have a blacklist of passwords that people are not allowed to set their password to you

saw that they were using a dictionary style attack at Lending Club if you get to your password rotation and you try to enter in the password Lending Club one no we're not gonna let you do that you don't get to do go warriors you don't get to do season plus year take what you you already know what the word lists are there's publicly available word list for dictionary based attacks that your attackers are using take them drop them in a blacklist make it so that at least they can't do the very simplest way to guess a password okay so those are attacks against my employees let's talk about in tax against my customers and at

the end of day oh no formatting great all of our users have been pwned all right every single one and if you had a Dropbox account in 2012 or a LinkedIn account in 2012 if you guys haven't heard about have I been pwned calm which I hope most of you in this room have go there type in your email address it will tell you every single public breach that your email address was involved with won't tell you your password but just know that whatever password you use for that is already out there and that leads to this rise of something called credential stuffing which I'm gonna take the username and password that I know that you use in 2013 tumblr

and see whether or not you use that same set of credentials at your Bank of America account because guess what all of our users reuse passwords all of them all of the time and so here's what credential stuffing looks like for us multiple countries multiple IP address whole bunches are different user agents but I hit the last point is the key one again can you see it credential stuffing is a game that's played in the millions millions of failed logins all right so here's how that looked like shiny little slug visualization who's in Russia doing failed logins we don't have any customers in Russia Vietnam what are you doing with 800,000 failed logins all right and we've tried a bunch of

different things some of them have been less effective than others to trays to raise the cost for attackers so one of these what that we tried was hey the breaches are out there we can get a list of the actual usernames and passwords let's compare this against our user database and there's just reset the password for anybody who might have been using that same set of credentials which kind of worked there was a depressing number of overlap between us and perhaps some of the other public breaches but we're gonna get that information a lot slower than the bad guys are and users kept resetting their passwords back to the compromised creds so no it wasn't so

great blocking user agents kinda worked a little bit like some of these botnets were just standing out like a sore thumb hey who's still running Chrome 23 and why do they have a hundred thousand failed logins maybe they should be dead to us but that was still a little bit of a doing a whack-a-mole things that we found that we're a bit more effective we're rolling out step-up authentication so this is not true two-factor with like a time-based one-time password but it's hey you've come to our site you log in if it's a device we haven't seen before we're gonna send you an email you got to click on the link and go through there

so and unlike it's perhaps a true pure two-factor we can force 100% rollout rather than saying please download the Google Authenticator app and learn what a QR code is and so on and so forth and then they already again visibility can you see this I says a credential stuffing is a game played with the numbers in the millions and so if you see a massive massive uptake and failed logins you know something's going on can you see it all right so that's what I got if you want to yell at me and grab slides contact info is there there's my Twitter there's some email a so b-sides is doing this thing with peer lists so that there's never enough time

for this QA coming afterwards if there's things that you're curious about I want to tell me how I'm totally wrong about everything I just set up here go to that bitly link you'll go to the peerless thing for this particular talk and you can yell at me all you want there and that's all I got [Applause]