Whatever Happened Last Time, It Wasn’t A Penetration Test by Joe Sarkisian

all right awesome thanks for coming guys um came here from boston massachusetts my company paid three grand from here coming around trip to give the stock so what what to them um my name is joe starkision uh i'm from den secure which is our brand new newly branded like offshoot we have like we're a financial company wolfen company we do like auditing and stuff like that so i like to say we're a security company that's essentially bolted to the side of an audit firm so it's cool that we got our new brand done secure um i've been there for about four years i'm the lead penetration tester there got a team of about five people

um again this is our i don't think anybody from work watching this is a boring slide it's just about my parent company wolfen company a lot of people that work there if you need audit whatever come to us ideally come for a pen test uh about us again social engineering advanced security assessments so just real quick to touch on that we don't say anything's a red team because most people don't need a red team they just think they do because it sounds cool so we called it advanced security assessment it's objective and goal based right so that's that we do threat emulation but let's talk about why we're here how many people in this room are pen testers

you got few hands okay how many have had to buy a pen test on behalf of their company okay a few hands awesome all right so we got like an even mix there okay so it's gonna speak to some of you hopefully so i've been there for four years in this entire time we've had this problem of we have a new client they're not sure what a pen test entails they've had one in the back like i don't know a year or two right like maybe they're regulating they have to have it every year i don't know whatever it may be and then they contract with us and they're like okay what's a pen test

essentially and it's like didn't you have somebody do this for you like a year ago like that they did not like put you through the paces and tell you how this works whatever right and then so we're it's almost like we're starting from scratch and we have to like hand hold people through this process and it's like we're fine doing that but we need to get better as a as an industry as a as a security industry where that starts to be understood when we come in the door right it's very important so it's still poorly understood right by practitioners by clients uh by people who aren't even in the industry obviously it's just poorly understood i

tell people what i do and they like walk away because they think i'm a criminal right like that's not what we do um as a serious consequence if you get it wrong right like false sense of security for one um there are a lot of vendors in the u.s i don't know about uk eu i see not like people like yeah people suck at that okay so there's people doing this that don't actually perform them like the classic von scan sold as a penetration test right those things are not synonymous um and we're here because it annoys me and i want to tell you about it uh this tends to lead a lot of this

tends to lead to the following like responses that we get either during a scope call or an exit meeting after we've like given them a report to go through and it's every single time but we let you in so we're secure right because you had to let you in i'll talk more about that or that's not a realistic scenario like okay uh our mssp would have stopped you this is my favorite one why didn't they um does everybody know what mssp is i don't know if that's like translates are cool um this report does not actually adequately reflect our environment okay but we're tracking that issue okay risk acceptance awesome our report was clean last year

okay why didn't the previous vendor find this this has no answer i can't answer this question but i get asked it a lot so here's a scenario company aids regulators require pena a pentest annually at fdic in america you have to have pentest every year right uh the bigwigs don't get security at company a so there's very little budget to have this done uh they look for a cheap vendor right there's plenty of those there are surprisingly tons to choose from at their price point if you google penetration testing services when i do it at home the top google result will be like 99 penetration test and i'm like okay and then like a lot of the rest of

the first page is something similar like that i'm willing to bet you're not getting a pen chest for 99 okay uh they pick the one close to the cheapest fair i mean look at their situation right who wants to guess how this turns out dumpster fire okay so they finished the test and here the report looks like this it's either one of two things there's no or few findings right or a massive spreadsheet of vulnerabilities that was not tested for or for validation right if you get handed one of these as someone who's buying a pen test you didn't get a pen test you get a vulnerability scan and they're telling you it was a

penetration test right and we're gonna go over like things to look for if you're trying to get a pen if you need to buy a pen test but i'll just say it right now like the first question you should ask if somebody sold you a penetration test and hands you one of these is what did you exploit from this list show me and they won't be able to now they have either a false sense of security because they have very few if any findings right everything's ship shaped good to go uh or new job openings after you slam the it department with a massive report that they now have to sift through and they're not going to get through either

that audit year or probably not the one after that okay so it builds up if you're using the wrong vendor over time so why did this happen lack of buy-in from the top right the sea level is like oh god security is a pain in the ass like this is too expensive whatever it may be like we see this as a cost center uh lack of budget because the c levels don't understand why this needs a budget which leads to a crappy vendor as we talked about which leads to poor work false positives false negatives which are even worse leading to a false sense of security massive i already say that or massive yeah okay my massive pile of

monetary work leading to a breach right because you think you're secure you don't have budget nobody gets security at your organization and this is what happens uh-oh sorry surely this will happen again next time right next step so another scenario company b's regulators also require an annual pen test the big ways understand security there so there is solid budget they do research first and they send rfps to people that they have researched that have seemingly good track records this takes time right but the rate but company b has people that understand that this is important and therefore the people who are going to make these decisions and pay for these services have the time necessary

to do it right there are surprisingly very few of these to choose from in america there's a company called secure ideas anybody know who that is security is kevin johnson owns it okay on his and i i love their company for this i love him for this on their own web page they sell pentest right they do a lot of web app um api stuff like that but they'd also do like social engineering assume breach whatever it may be and honestly it literally says if you don't go with us here are the seven other people you should buy a penetration test from and he names like but like trusted tech black hills like all the yeah i see

everybody knows those companies all the all the good ones right i wish he would put us as the eighth but [Music] um and they picked one that makes sense for them cool once he gets how this turns out and i forgot to put a cool meme here after this so you're just gonna have to guess that it went well so this penetration test finishes and the report looks like this validated useful information right so it wasn't just here's all your scan results and uh we'll see you later it was here are the things that matter so like how many people in here are responsible for like remediating findings okay two okay well all right i'm going

to pick on you guys i'm going to pick on you guys then has this happen to you where they just slap a spreadsheet in your face okay good does it happen to you in previous life at another company yeah yeah yeah so a lot of those scans if anybody's ever run a vulnerability scan you'll come back with like stuff that's like critical high whatever critical would be like oh my god you get ms-17010 right and that's critical i'll be honest but some of that stuff in there is like you might see like spectre and meltdown type things in there that like really like it's critical based on like cvss or whatever the metric is that they're

using but if i look at that report as a penetration like i don't give a crap about that right i care about the informational that says that like smb signing is not enforced okay or smbv1 is in use or lmnr right stuff like that now if somebody is handsy with scan you're not going to like if your sop or an sla rather in an organization is we fix criticals within x amount of time we fix highs whatever medium whatever they're never getting informational which is where the gold is as a pen tester or as a bad guy right so just think about that if you ever end up in the situation where somebody hands you

that dock um anyway i'll get back to this so uh it proves risk right so we're intensely proving risk we're showing screenshots we're saying i did that like i don't believe you did that here's a screenshot showing i did that right i did that that happened in your organization that was part of the penetration test here's why it matters uh it helps the company make good budgeting decisions in the future right you can say look at in the report like okay sure you have a knack and you have a firewall and you have this like those things work well here's the thing that doesn't focus on that if you need budget we can help you get budget because we

told you you have a problem there right you have a hole in your in your security so we can help with that um it informs stakeholders of day-to-day staff about weird security issues they won't notice so that's kind of what i was getting back to like the vol scan with like the informationals where like i'm gonna pay the most attention as a pen tester or a bad guy um you can kind of re-re-risk those things to people and say look i know that you have to patch and that's part of your sla but you should probably pay attention to these other things right fence and depth um and there is no option b right because it went well like things things

went well there's no other problem here right now company b has a solid point of reference um their team is bought into the security process right like if you have a good vendor they're not just gonna like even if they do a good pen test they're not just like slam a report on your desk and walk out the door and never talk to you until next year right what they should be doing if you're buying this type of work um is in a way hand-holding you through remediation and i don't mean they're like also being paid to come back and fix everything that you found because then there's a conflict of interest there potentially um but what they are doing is saying

like here's the recommendation right like here is what you can do like not quick fix but like the general idea of what you need to fix and if you have more questions like we'll provide you more documentation um we'll we'll tell you like yeah i know you want to fix it this way but there's a reason that that's not going to truly remediate the problem like you need to do actually x y and z as well um so there's you know you should be able to pick up the phone two weeks later and say hey i have a question i know we already had the exit meeting but can you answer this yeah i can right like i'm we're not going to

bill you because you have a question for 10 minutes of my time any company not just us um and there's less likelihood of a nasty reach like yeah you're going to probably sooner or later if you're somebody of interest you're going to have a security issue you're going to potentially get breached whatever like there's no magic bullet there but there's just less likelihood of it being bad right so i am blowing through this talk it's also the first time i've ever done this talk so that's why i'm blowing through it uh we've kind of talked about this a little bit um do not shop for the lowest cost provider okay like again i part of the reason i'm

giving this talk is because as part of our pen test we're sometimes given a prior year report for other like whatever vendor they went with last time that was cheaper and oftentimes it'll like it'll be just like 20 pages of just a wall of like text and it's like about our company about the people here about our culture and it's like there's no executive summary where like somebody who's gonna make decisions just look at it and be like in two minutes tell me what's going on here and then i can leave and go back to my next meeting right it's just fluff and then you get past the fluff and then there's no narrative right so there's no section that says we

did this and then we did this leading to this and that's a finding right and then we did that and blah blah blah and that likes we kind of spidered out here and we got access to this data and whatever it may have been right there's none of that there might be sorry one more time how many people here have ever received a penetration test report five six people okay how often do you get a finding that says like uh tls version like one is not is not enforced or enabled and they have that as like a high risk vulnerability has anybody ever seen that you guys have seen that yeah randall to me that says they couldn't find

anything else and they're like make it a high because we got nothing else for you um that's kind of what you get with the lowest college providers right that's like we'll scan we'll do the and then like we'll poke it like whatever we find right like whatever the scanner tells us like we'll poke at if we can't do anything with it oh well look into the vendors you're considering right so like take the time to research um again when you google for providers you're gonna find like the first page is gonna be taken up by people who paid for like the most google ads or like however that works and they have like all of

their like super low pricing and all of their like elite hacker speak can you go to their website and it's like bits and bytes all over the place and you know all of the typical jargon and lingo look into it i would say i mean is anybody else here from the us nice one person okay i don't know i mean i know that sygenta obviously they do their stuff around the globe a lot of black hills trusted sect do as well but they're also very expensive right um ask for references right so like if you do the rfp process and you're like i need a pen test uh and you pick say three places those places should be able

to provide you references right so like who have you spoken to you did a good job for who's willing to allow them to be who's willing to be reached out to in order to like talk about how that process went for them and kind of what they got out of it you know complaints or the good stuff the bad stuff whatever um if they don't want to give your references that should be a red flag unless of course all they do is work for like the dod or something like that and like i could tell you but then i'd have to kill you pay attention to communication this is big so let's say you've you've done the rfp thing right and then

you're gonna have an rfp call right so you're gonna talk to these people for the first time like on the phone you have your stakeholders there right if they don't have questions for you about the environment you know like what are we looking at here like you tell them i need an internal penetration test and they say okay like that's not the end of that conversation right there's rules of engagement that have to take place there's like okay what does that environment look like i there have been times where i have not been on one of those calls as a lead pen tester not my current job and the the i'm not on that call the work

gets sold i'm now doing the work and i say i call the client or we're now we're going to have like our pre like engagement scope call just to like make sure we're on the same page like what are we actually doing here and i realized that nobody from the manager the partner whatever took any notes when they had the rfp call and then sold the work and i'm like i don't know what i'm doing for this client because we didn't have that chain right so that needs to happen like there needs to be that chain of communication so things questions be like okay what do we need like how flat is your network right what do you have in the

environment uh there are times where i've gone in thinking i'm doing an internal penis like we don't have any servers like everything's in the cloud and i'm like [ __ ] okay i changed things uh did they tell you about their process right so like if they don't say okay well based on what you're telling us and what your network looks like here's like what we think makes the most sense for you right if you say we have this we have active directory right we have whatever x amount of servers we have workstations this is out of scope this is in scope like whatever right okay but what about social engineering and it's like we we asked for an internal pen test

like why are we trying to sell social engineering now right i mean it's like if we know what you want why are we trying to then like build you out of more money for something you don't want or maybe don't even need um it's part of the sales process sometimes but like if you're like way over here when you're trying to like scope these things and sell people things like it feels kind of dicey i guess you could say but let's say it's the internal pen test what do they do for that internal pen test like do they just like throw mask in at your entire network and just hold at like 10 gigabytes a second or something like

that like rate and just hope for the best right you need to know what they're going to do what kind of tools are they going to use are they going to uh you know exploit things on business critical systems without telling you first right like if you fi if we find an exploit we think it's valuable or vulnerable we need to like tell the client before we do it because it could be a problem uh do you know who's performing the work so who here knows like what i mean this is like an american tax thing like anybody know what a 1099 is contract so basically it's i found out recently talking to other people who've been in

the industry far longer than i am than i have that uh groups like ours right like so there's a lot of cpa firms right and all of them are like we do audit we do tech whatever and they're all trying to get into security because it's like really like you make a lot of money doing it right if you're the firm so what they'll do is they'll put on their website we do penetration testing right call us if you need a penetration test whatever how many people do you think work for that cpa firm that actually is a penetration tester right i found out that a lot of them are 1099s which means they are subcontracted out

from like god knows where to perform this work they're not vetted probably right they're just like oh i do bug bounties and i guess like i'll send my resume to like all of these places and hope somebody bites and once and will let me do work for them on a contract basis so if you don't if you ask like okay who's performing the work are they contractors like how does this work do you have an in-house team whatever it may be if they're dicey on this question you should also like a flag should go up in your mind right this is a problem um so yeah he was saying basically that when i was talking to him he's like you

know you guys are one of the only like teams that actually have like a cpu from actually like an actual security team which i thought was kind of cool um yeah so going back to that initial slide that i all the reasons like why we have these arguments with clients right well not arguments but conversations let's say so but we let you in one is always fun um if that what happened last time before we like we or some other reputable company came in was uh you know essentially they didn't get in they didn't get anything right they just tried to do an internal pen test but they couldn't get on the actual network right like they plugged in but they

couldn't get access to the domain they couldn't really get anything right and then they get a clean report so we we or whoever comes in this time when we say okay i'm gonna do the same thing but if we let's say it's a week-long engagement right where you have to do this pen test if within like a day and a half we can't get access to the domain we're just gonna ask you for it right because what are we proving by just beating away at like i don't know like your phone vlan hoping that we're gonna get like access to the corporate network or something right like that proves nothing so if you ask for a user account

right basic user account now i'm actually on the domain maybe i didn't find anything before i got that domain access but i can guarantee you i'm going to find things after i get that domain user access there's way more vulnerabilities within ad than there are in like actual infrastructure in my opinion when you're doing corporate internal testing like this so how many people heard the term assumed breach awesome yeah good a lot of people for some reason in america still don't really understand that concept and we have to explain it but that's a good thing at least we get to explain it so that's what i'm talking about it's like assuming somebody got in why are we

wasting time here where we can show you what will happen if somebody does that's not a realistic scenario so if the last person scanned and called it a pen test you're probably right it's not a realistic scenario um so when i say realistic scenario like we have this one client where it's a big bank and every year we come back and have the same problems same stuff every time i can practically copy paste the report from last year and hand it to them we don't but i probably could and every year the guy goes but we have a knack that's not realistic you wouldn't have gotten in how many people know what a knack is

okay if what is the again going back to scigenta and their threat modeling idea or their threat like what is your actual threat um whatever scenario like igor from russia is not going to get on a plane fly here to your office building in burlington massachusetts you're not a big bank by the way fly here social engineer their way through the door get all the way upstairs to the it department walk into your server closet somehow and plug a device in your knack's gonna help you there but like how realistic is that scenario right that's the one security control they had except for crowdstrike on endpoints right so think of all the holes that if there's only a knack in

crowdstrike that that are not being filled right so what's more realistic somebody sends a fish and c2 gets established or credentials get stolen right where mfa is bypassed on the perimeter and they weave their way through right that's more realistic so to tell me you have a knack like that's not a realistic scenario to be honest with you because that's not until one layer of like the onion right so we have those conversations a lot because those things weren't explained from last time our mssp would have stopped you well did you tell them we were coming probably shouldn't have right and if not why didn't they stop us there's a whole other talk that i'm

working on right now that is do you even know what your mssps are doing for you we have so many scenarios where we go to do a pen test and we're like look this is not a threat modeling exercise this is not you know threat emulation right it's a pen test but i do want to know if you get alerted to some of the things we're doing because i want to give you credit for it in the report right so here's what happens if they have a knack they get alerted when we plug in great if they have crowdstrike they get alerted when we try to like dump memory or something like that right on an

endpoint they get nothing else typically right so now i have to go back i'm like okay well those are all the alerts and whatever hap whenever this happens it's always show me your alerts sure i get like a 10 megabyte pdf of like every single security event that happened for like that entire week and they expect me to dig through it that should be a finding right there because you don't know how to dig through this and figure out what we're doing oftentimes it's hey was this you we got this alert like what's the ip address oh like you know what my ip address is can you correlate right so the point i'm trying to get at is

there's hardly ever good data that they get back from these mssps and i have to ask them like okay well do you know what's in the contract that you signed we didn't sign it we've only been here for three years and this is a five-year contract and the guy that signed it left so we don't even know what he bought have you read it no okay well i would have a phone call and then find out where their sensors are on your network what they're supposed to be collecting like what they're ingesting to their sim right what they're supposed to be doing for you because these are low hanging fruit items that you should be picking

up on right actually that same bag with the knack the nat guy same argument with him every year because he never picks up on anything we do right uh this report doesn't automatically reflect our environment well did the last person actually ask you about your environment right did they check in throughout testing the same page you on their findings along the way so one of the worst things you can do as a pen tester is go in beat the crap out of somebody's network not talk to them the whole time and then have an exit meeting call where you show them the report and go look at all these criticals you guys have a massive problem here and

they're like what like why weren't you talking to us during this process so we could be like this is not even this is a false positive and here's why we think so we have other controls that are that take that kind of take precedence over this so like maybe you didn't hit that control but we have it in a production environment like those conversations need to be had how are we on time ten minutes all right cool i've been i stalled stalled well

sure yeah yeah uh but we're tracking that issue again status meetings should be had to avoid this right if they've risk accepted something and i'm telling them it's a critical finding they're going to be like our report was clean last year probably because you're company a uh why didn't the previous funder find this did they have a scope call with you did they tell you about their process and honestly i can't answer this question right we get this question i can't answer this question [Music] so finally questions to ask the vendor do you perform vulnerability scanning during a penetration test and if so why right they should be able to answer this question do you attempt exploitation of

vulnerabilities if you find them this kind of pairs with the first question right you're kind of getting to a what is your process and are you selling me a load of crap as vulnerabilities can as a pen test right how much do you focus on active directory and identity because that's like the new perimeter right this is identity right people are always like people when we're doing external testing they'll be like here's my three ips i'm like okay can i can i include microsoft 365 and azure and all that stuff in this and they're like oh yeah like it's always a second thought right they're still thinking of it in that traditional like heart like

i'm gonna beat a hole through your firewall type of thing which is rare can we see a sample report ideally they have one they can show you so you know where you're kind of going to get at the end of the process again are they employees are they contracted out uh what kind of ongoing training testers undergo are they allowed to just kind of sit there and like not learn anything new right that's not good uh what's your fishing process like i'm gonna harp on this for a hot second how many people have here how many people here have had a third party come in and do fish testing slash training for them were you satisfied with that process

there's a middle finger in the back that's how happy he wants to

they always focus on like oh x amount of people clicked and x other people entered their credentials bad bad user right like that's not the way to think about it way to think about it is let's say that person clicked and they realized they did a bad thing how fast before you got a phone call or an email that said i messed up i'm sorry right time to click or time to report after click are entering creds if they don't realize it is something wrong then there's user training involved in that but like if they know is there a good culture where they can hit a button pick up the phone ideally it's hit a button or forward off to a

phishing at whoever email address right so that somebody can start triaging that issue and find out what's going on because if the faster you can do that the faster you can isolate a potential huge problem so if you're paying for fishing ask them that question all right that's it questions for a few minutes i think sure thank you

[Music]

uh

[Music]

um

[Music] did you say time stamps yeah right

just um this has been managed to do this to explore this stuff like really always the other stuff

right so seeing the stuff

um

yeah that's a great point so actually to that point i would love and i'm starting to get to the point where i'm like you know what like there are times where we've been put on the phone and un unexpectedly during a status call and the mssb is there and now i'm forced to be like the guy in the middle that's like i did it and then the client's like why didn't the msp catch it who's on the phone and i'm like it's just this awkward thing where i've been put on that phone call and i want to be like look okay you caught this you didn't catch that can we dig into why right can we talk to them and speed like

do you have a rules to detect i don't know bloodhound running or you know whatever it may be um so yeah i mean it's one of those things where it's if the client wants you to it's if the mssp wants to it's if there's enough hours unfortunately we go on ours to do that type of thing right and it's cost money unfortunately um to do that but i agree with you 100 um i think that to your point about you know if we can help that one customer because we work back and forth now we have a detection for xy or z right we wrote a rule whatever it may be um we've had mssps near me that are kind of

local and they're like yeah but if we have to change that we have to change it for everybody and i'm like that is the worst thing i have ever heard someone's like yeah that's the point anyone else backs

is

[Music]

did anything so i've noticed that a lot of the places we work like they'll have crowdstrike a lot of like cementec endpoint protection or like whatever or like you know even some av right it's like not really hiristic it's just like oh we have a signature for that cool uh and basically the the alerts interestingly come from the client dashboard like internally they're not coming from like the mssp and then down like phone call to client and the client's like hey was this you right it's usually like oh we got this alert we're not really sure what happened we think it's you right so it's like again that goes back to okay well are you paying for the mssp

to ingest everything that comes out of your edr system right and then go back like through them back to like you're paying them to watch your network because it's really hard to watch your network and you don't have the people to do it so you if they don't have what they need to get the job done for you you should probably get it through them like it's kind of that back and forth conversation um but yeah i mean it's it happens a lot i think yeah

[Music]

so are you saying is it is it important to hire in to perform that testing internally as opposed to paying yeah so i think it's very hard for people to get that done because like if you if they're struggling with a security budget to even get like i don't know pay for better mssp services or alerting or whatever right like standing up an internal testing team like that is a whole different conversation and i feel like that's a lot more mature organizations that are able to do that um oftentimes our clients are like that steve that's the security guy over there steve does security and then you talk to stevie's like i do literally everything and then

they've made me do security right they don't have the budget necessarily yet in order to have that happen for them or like two or three people to that continuous testing you know but ideally everybody gets there and then when we come in we can work with those people and you know yeah if we got if we got hired to like it might be a situation where they just want uh you know we're not going to tell those guys you're coming right or it might be we want to work we want you to work with them so you can you can show them your process right you can kind of back and forth and maybe pick something

up [Music]

it's honestly just a sticker it's i'm a pen tester like i don't consider myself a red teamer i mean like technically on the spectrum yeah probably closer to that side but i don't consider myself like yeah that's all i do but yeah more of the red up for you

helping but some controls or injection rules that are the most likely ones to intervene over many cases are there detections that we would just like bottom floor recommend for everybody is that so the interesting thing about that is like i'm we go in i'm like okay i ran from an unmanaged host on your environment like let's say we have kali linux up or something like that on your environment it's not domain joined it's not a trusted host not part of your asset inventory i just slam the crap out of your domain controller and all of your workstations with running bloodhound right to find out where everything is in this network to me that's like a

ground level detection right that should be in place across all customers because it happens in the wild it does happen right um but you don't really see that so it's like are they just so i've heard sometimes people have to pay extra for that when they're paying for these services which blows me away so bloodhound i mean enumeration in general uh password spraying it's a domain controller right like there's a big one uh abnormal logins from like certain accounts right just weird smb traffic right like kerber roasting right stuff like that there's good detections for that nowadays so that stuff like the typical things that you find a lot of threat actors doing like those are the ground level things i

think uh where is he fc could probably give you more like this this is what you should be doing i think that's all the time i have correct cool thank you