Dr Pentester or How I Learned To Stop Worrying and Love the Blue Team

thanks also thank you all right right I see her sister wide world client is the need or iOS

research of years

so right before I started working on this presentation I was talking to a friend of mine and he said something to defective I wish I had been doing this all along and it it kind of dawned on me that life in a way is a series of moments that you can connect where you say wow I really wish I'd been doing it this way all along so if you guys have recently looked at a vulnerability report or a penetration tester report I want you to kind of think about what you did with it and what was there and what was missing and this is kind of the heart of of what this this journey is

for me so in the industry we use a couple of different terms phoner ability assessment penetration tests red team engagement they're all used and abused by the marketing department center in our organizations and so sometimes you'll ask for a pen test but get a vulnerability assessment sometimes lasts for red teaming engagement and you'll get a vulnerability assessment and various different combinations of those three unfortunately what happens is you'll end up with a 300-page vulnerability report whether it was done with exploits whether it was done with nessus and you'll have to deal with that somehow i regularly generate 200 and 300 page vulnerability reports and not for lack of care for risky issues but merely because of the size of t environments

that i work in i regularly work in organizations with a hundred thousand two hundred thousand nodes and they asked me to come in and do an internal network penetration test and it just so happens that they have enough stuff to fill reams of paper and these vulnerability reports generally are not the audience of the security operations team I'm sure you're all quite familiar with what a blue team is this being a very blue kind of conference the audience of these reports is actually the developers the IT team when the blue team ingests vulnerability reports they put it into the vulnerability management software and a triage it so what I really wanted to do was increase the

utility of these reports for the the blue team or the security operations team in the organizations that I was working for so my very first client at ioactive was a retailer an online retailer they did lots and lots of business and we came in and we did the standard penetration test got in there got domain admin dumped all the creds and got all something 120 million of their credit cards this was part of a PCI assessment but they were one of the companies who said we want a real penetration test so the entire environment is in scope go so this one great the first time went into the boardroom had all the executives listening to me presents all these

critical vulnerabilities which led to the complete and utter destruction of their environment as the defenders hate to listen to because you know Here I am coming in and breaking everything you guys have worked really hard to protect and you know that's obviously an awkward conversation to have but the problem was that i came back next year and the same thing happened things had changed things had been patched a lot of things in the report had been fixed but i got all the credit cards again and i got all of the domain passwords again a third year same thing fourth year same thing and as the years had gone on with the same client I kept telling myself wow these people are

idiots what are they doing they have five people on their security team and they can't get one thing right is to keep me out of their domain controller and I chewed on that for a while and eventually it occurred to me maybe it wasn't just that client maybe I bear some responsibility for their lack of ability to mature based on the work that we were doing for them so along those lines we'd been talking about defense in depth for like ten years like what even way before I got into security you know over eight nine years ago people were said defense in depth is the way to do it because there's always going to be a

vulnerability and there's it only takes one right somehow we forgot that defense in depth was a thing and nobody actually did it and now we we call this era of security the post vulnerability era which is a synonym for defense in depth because we have to assume that there's going to be a vulnerability assume it's going to be exploited and what we need to be able to do is detect compromised regardless of the vulnerabilities in the environment if we can't detect compromised we're stuck we can't respond to anything now when you order a penetration test the guy who comes in to do it thinks about this this is what's going to happen they come in they break

it they leave a mess and this is I'm going to speak generally about penetration testers but this is the general attitude of penetration testers there for interesting work they want to get in they want to break things and they want to solve their their problem which is breaking your environment they don't they're not interested in solving your problems which is securing your environment and unfortunately for most penetration testers their ego gets in the way their methodologies opaque they won't tell you exactly how they do their job and there's a backlash against penetration testers i went to the dfi our summit in nola recently and sitting among all of these blue team members and defense defensive security people I they had

nothing but scorn for penetration testers as I listened to them speak so one of the first things that I had to come to terms with was different clients and different organizations have different levels of maturity and that would have to dictate how I went about doing my penetration testing in terms of red team so just to be clear when you do a red team engagement there's going to be penetration testing and that penetration testing needs to have other components in order to fit within the model of the red team and if there is no blue team there is no red team so you cannot have a red team engagement in an organization that does not have a blue

team to sit down and work with them just so those those terms are clear so if you're in a if you're the type of organization let's say okay so I was recent I see it was obviously I want to say it's 2013 I went into a very large organization i sat in their sock and this was the first time I spent an extended period of time in a sock they had well over 50 people in this sock and they were processing billions of events per day and now that's really not that unheard of and it's not that uncommon but I sat down to start my penetration tests and my pork turned off and i turned around the guy behind me he said

a man my network jacks not working he's like oh yeah we we probably flagged you with the IPS and turned off your port and I'm like whoa yeah that's pretty cool why hasn't any of the other clients that I've worked with done that and so I started talking to people with a sock and asking them what they do what their workflow was and they they were a fully mature organization nothing about what they did was you could consider deficient security wise and it was very impressed and I've modeled everything that I've done after their operations so you have that you have company that processes a couple billion events per day all the way down to what would call a half a

security guy which is the IT guy who also sets the passwords on the VPN or adds users or gets a daily email with events attached that he's supposed to review for compliance purposes and this difference in maturity the spectrum that means that when I sit down to do a penetration test or a red red team engagement in this case I have to be able to tune my methodology to their capability of detection if I'm out there doing apt stuff with these guys with these companies that only have a half a guy doing security it's it's wasted on them if they can't do basic detection if they can't do basic incident response if they can't detect I car on one of their

end points there's there's no point in going full bore so at this point I started gauging my my clients engaging my my projects with the maturity of the client in mind instead of just going at my old standbys over and over again so prevention as I talked about earlier vulnerability management is about prevention and prevention only if vulnerability management is largely an automated process now so getting to that level maturity that's the first step if you can ingest if you're doing regular scanning of like Nessus or qualis reports there you go you're on your way if you stop at that point you fail to address real life threats because as I as I alluded to earlier there's always

going to be a vulnerability especially vulnerabilities that nobody else knows about and it only takes one of those for your organization to be compromised and you have to be able to detect the compromise speaking of detecting the compromise detection so I was in an organization whose financial financial organization they did payment processing they asked us to come in after years of doing for us doing audits for them basically checking boxes and making sure that you know they said they were they made sure they answered correctly to the questions that we were asking them we've done this for several years previously they asses come in and do an actual penetration test and I had been thinking about that

large sock for many many months and this was the day that I decided okay so I'm going to do something a little bit differently after performing every action after running every tool I opened up an Excel spreadsheet and added a row that says this is the date and time this is the thing I did and this is where I did it and at the end of that engagement I handed that Excel spreadsheet to my point of contact and I said go to your scene show me exactly where you saw all of these events happen and he came back to me 24 hours later and he said uh yeah we didn't detect this one which was my

Nessus scan of the internal network the chap has been unplugged we don't know how long and we don't know who but we haven't been detecting internal network events for at least three weeks and he didn't know because they're logging ended at three weeks so that box could have been been plugged for a year and he would have never known and this was the aha moment where I said wow when I go in and I do these penetration test I throw up all of these these indicators these things that that that nothing that should not be happening on a network during a normal workday and if I can communicate that information to the client in some way they would they're

going to be far better off because I can tell them how to be better at detection and there is no response without detection even if you have an environment that's that's perfectly free of public vulnerabilities and let's say somebody found the the NSA cache of awesome Oh days and found your Cisco device which happens to be vulnerable and you know maybe you were asleep that day and somebody popped it you wouldn't know because the your your ability to detect is deficient if you didn't have logging on that router to detect the unusual login behavior from from that particular attacker there's no way you're going to respond to it and that's part of the discussion during these red

team engagements is ok so I did this you need to do this to detect it and this is the only way to get to the response phase one of the other things that I needed to learn about being on a red team is that it's very hard to do security engineering and operations you have the constraints of time you have an overwhelming number of events to analyze and categorize and a good portion of your job is trying to automate these processes on a continuous basis ingesting IOC's making sure they get to the right place is initiating responses and analyzing doing threat hunting analyzing you know incidents and things like that it's and you're constantly fighting your management for money and

time and resources and for me to not think about the business and the impact that my testing has and my recommendations have on the business if I just say wow you know if you just enabled SSL client certificates on everything in the organization that would solve all of your problems obviously managing SSL client certificates across an entire organization is a daunting task another thing I really needed to grasp was that in order to truly give value to these organizations especially the more mature ones is that I had to go out I had to look at the attackers and see exactly what they were doing the they need the red team needs to do evil as evil does

and as a penetration tester I learned to hack in what you might call safe places I mean you know back in the AOL proggy days of course you know I bent a law or two but the the majority of my learning experience has been in environments where I've been mitted to work and that means the the amount of noise that I made and the care that I had to take in order to prevent from being detected it was very it was very low and what I have to learn from attackers and especially the most advanced advanced of attackers is to be quiet and to be stealthy and to not leave tracks red team engagements absolutely positively need to be

goal-oriented because attackers are goal-oriented I I recall the the Microsoft blog about attackers thinking in graphs and defenders thinking in lists a graph leads to a goal a graph leads to a single point and organization they want to get to that golden egg and if your red team isn't planning and executing in a goal-oriented manner with the business objectives in mind then you're not being effective as a red team so in the same vein as emulating attackers the tools that I use the existing tools that I use are okay for using in a red team and engagement and that spectrum of sophistication allows you a little bit of wiggle room but the most important thing that I had to do

was learn how to tune the tools effectively so that they weren't leaving bread crumbs all over the place the very first thing that snort detects is an nmap scan and if you're out there doing recon on the internal network with an nmap like almost hole penetration testers do you have already given up you've already given up essentially you've you thrown out the flag the end map is coming from this host and this is where the blue team is going to come find you so you can't just dash dash your way dashti your way to stealth in the network you need to do things like scan a single port once every five seconds on each host that you're looking

at and do so in a random order across different routes on the network if you do sue if you do so sequentially on any given host you look at the snort rules and you're gonna get busted and that's part of what I do is I I look at these detection engines and I say how can I use these tools in order to avoid these detection rules that the blue teams are using oh and log deletion by the way I I never before had deleted logs before I decided to become more sneaky most penetration testers wouldn't bother with that either social engineering it was really important and it's really important for any red team to have at

least one person who is good at social engineering because we're seeing that probably around fifty percent of successful breaches originate from phishing attacks and if you don't know how to craft a convincing email template to get someone to open an attachment or visit a link then you're missing a huge part of the infiltration phase of an actual real live attack and I learned social engineering well before I got interested in red teaming just because i really like lying people lying to people it's it's a lot of fun I'm really I'm exceptionally good as a liar I just I hate I hate lying for the wrong reasons so doing social engineering gives me an outlet for that skill like lock-picking

like you buy some test locks to fiddle around with you know you know go to your place of business and like start trying to open the data center door you know you go home and try to open your acrylic locks and team so it's really important that a red team be an actual team with diverse skill sets if you don't have people who are catching shells and if you don't have people who are skilled at building payloads and doing you know you know moving laterally on the network and combining all those skills and getting different perspectives on how to execute attacks then you're also missing the other side the other side of the coin attackers work in teams they had they

have diverse skill sets you need to have a mirror of that in your own red team and at ioactive that's what we strive to do as well I never want to go in an organization just on my own and try to do a red team assessment because then it's just me trying to figure out this complex ecosystem of threats so collaboration is really important I suggest that blue team's can be composed of people from multiple disciplines within the organization I mean the blue team should ideally have three or four people at least on it who you know people who look at the glass people who you know recommend configurations for for improvement yeah yeah Giada there

should be a lot of people involved in this process the entire red team the entire blue team and if and everyone needs to show up to the table so when I do red team engagements I schedule regular checkpoints and everyone needs to be at that checkpoint so everyone can be on the same page and we can communicate information back and forth i can say we did this this week did you detect it and they say no we didn't detect it okay now let's have a discussion how we can go about detecting that type of attack now I don't need to give up I don't need to give up the Golden Goose immediately I don't need to

tell them what I'm going to do in the future that that kind of thing can stay secret but there needs to be open and honest communication between the red team and blue team because the red team is learning from the blue team how the defense's work and the blue team is learning from the red team how the attackers work documentation oh my lord if you ask a penetration tester to log everything they do they will look at you like you killed their dog they have no patience for 44 process and organization and methodology and least of all writing stuff down and this is the only way that I've found to turn on logging and all my

tools but in a centralized place and then summarize it for the client in order to effectively communicate the stuff that I did so they can go find the stuff that I did and that's what's most important that they be able to detect what I do not actually what I do external teams I'm extremely biased because I'm a consultant but i'll tell you a quick story i was talking to some of the red team leader on a very big client he had a very large red team and he told me you know what our blue team is great at catching our red team and that's about it so red team's can get stale if they're not continually

refreshed with new brains and new intelligence they it can be very easy for your blue team to detect them and they were also using a standard rat package so the blue team is able to find them based on the IOC generated by their rat they had to whitelist their rap in order to help minimize the impact of using the same tool over and over again for every red team engagement in that organization so I feel that bringing in a third party every once in a while to keep things fresh and exciting and teach your team new tricks is a great idea for me Intel gathering was really important because I have to study the attackers on

a continuous basis and I do this i follow researchers on twitter primarily to do this there's lots of interesting malware and defensive things just being chatted and communicated on in these circles on twitter i read publications from different organizations and vendors and even government agencies like you know i read the stuff that comes out of the NSA it's like wow that's a really good idea i'll try that next time and I spend a lot of my free time studying and following and mimicking and I do so in my lab I have a virtual lab and I can test all my tools and techniques in there I have it set up well on esxi and I have the V switch set to hub mode and

then I have security onion installed and I use security onion to look at what events my attacks are generating and this is how I can tune my attacks to minimize the the detection signatures of my work and that helps me to continuously improve the blue team by telling them how to detect me and ethics so I went to the dfi our summit and this was my first exposure to large amounts of blue team members and I saw a presentation by a very large company who had created a tool that's designed to be installed on every single end point on the network and that tool allowed them to query on disk and eat in memory any

grep string that you could come up with and that meant that that person or that team that had control that to tool was the NSA of their corporation and you you you generally I think most you guys are blue side you as the blue team are the NSA of your corporations and me is the red team I'm the attacker I'm the China of the corporation we're both going to encounter stuff out there on the network that people think or find personal to them like we say yeah it's a work s it's a corporate asset they shouldn't be doing personal stuff but we need to be very mindful the fact that these are real people doing

real things who have lives they need to talk on Twitter they look at youtube occasionally and they download stuff to their workstations so if you find stuff be very cautious about how you deal with that thank you very much I appreciate you guys listening any questions let's see I did okay without detection there can be no somebody said it who said response yeah right there