Watching the Watchers: Exploiting Vulnerable Monitoring Solutions

good morning welcome back to b-sides this is breaking ground uh before we kick off i have a few quick announcements uh first of all i'd like to thank our sponsors especially lastpass palo alto networks and our gold sponsors amazon intel blue cat it's their support that makes this possible uh because these talks are being live streamed the camera's right there please turn off your cell phones because we are in person please keep your masks on so with that please welcome matt and rock

hey everyone uh good morning uh if you saw the keynote this morning uh hopefully what you have over the next hour uh probably not gonna go that long are complementary to a lot of the things that mr hoff had so in my experience every great talk starts off when you insult a large portion of the audience if not all of the audience and uh let's do that so uh there are some of you in the crowd or those watching online uh that either do not have a threat model that you're defending against in your you know home environment your work environment and there are some of you that absolutely have an antiquated threat model and what do i mean by that

that you're still looking at the insider threats the nation-state actors the criminal actors uh that wish to deploy crypto miners in your network uh yes those are all viable threats that you need to be concerned about but there's a more nefarious one that we believe you should pay more attention to [Music] and this is a threat that has access to all of your emails it has access to all of your workstations your servers your routers your switches your netflow all of your bring your own devices and more importantly this is one that you probably paid several million dollars to put in the network welcoming it into your environment so what am i talking about i am talking about the

security appliances and solutions that are there that you rely on for your security in the event that there is a zero day in the event that there is a compromise of one of these trusted solutions these have perfect access in your environment and over the next several slides we're going to talk about ways that you can even with a zero day you can't stop that zero day but what can you do to mitigate the impact to in your organization when you have an appliance with damn near perfect access to your end price so uh with that uh we're gonna go through everyone's favorite talk compliance uh everyone loves that uh what are some implications of having

these work from home uh scenarios uh we'll go through a very common uh exploit vector into your environment we'll have a guy checking for things just looking for experiment yep and uh we'll also talk about uh you know the newfangled uh aiml that people are moving towards in enterprise environments [Music] so on to compliance so these are uh you know if we're lucky once a year people within their environments will take a look at them because it's audit time these are extremely prescriptive they're going to tell you what to do and where to have it but it's going to leave you up to what's in leave up to you is to figure out how to do it securely how do you do it in a

manner that is actually not inducing more risk in your environment and another thing that you have to worry about is you know what are you actually attempting to secure with these compliance standards and like i said earlier it's really the old threat model the insider threats maybe some of the compliance standards actually don't actually consider that a lot of it is you know security at the perimeter uh but now there's things like the cloud and it's the compliance standards don't really incorporate that as well and i have not seen a compliance standard to date that actually accounts for when a mandated security solution is the point of vulnerability what do you do at that point

the good news is that we can actually rationalize this threat and we can identify that yes we must do these things to stay compliant based off federal state local standards however we can understand you know the implications within our environment and what is not covered and so you can take these you know very abstract very broad uh requirements and build something that's more tailored a complementary checklist that allows you to go in and actually define the individual things you need to be looking for based off of your architecture based off the security applications based off the software that you're running in your environment and then based off that hopefully what you've done is create something that's usable something that

you look at more than once a year who knows maybe monthly right and it allows you to uh you know stay ahead of emerging threats because it's internally developed and it's useful so uh some things have happened in the world since 2020. i have seen some compliance standards that have not been updated since 2014 but we are still required to adhere to those 2014 standards uh despite you know things being different at work and so with that matt's gonna talk a little bit about some of those changes [Music] all right good morning everybody so rock presented to you some some of the old modeling and what we're doing as organized as professionals security professionals but how does that feed into our

organizational mission right we can isolate ourselves and say my job is to just look at these dashboards look at the systems that tell me day-to-day what is actually happening like with my alerts and with that structure but going forward the challenge as we go through these this presentation is to marry that to your organization's mission to then inform them of new models to follow new threats to advise them on and a way to structure it out so that it's not a burden to you and to go back to the very i gotta go back and rebuild this whole 2014 model not necessarily no right it's laborious to do those things but if we understand the context of the

mission that our organization is doing in our mission we can then provide more agile responsive measures to then manage these threats and model them right so there are we're going to give two examples and for one is the work from home model right so we've over the last years with covid had integrated this work from home systems into our our networks now when we did that we took a lot of managed and assumed risk for the end user we're like okay we're going to purchase this resource that we're not going to own but we're going to rely on it to provide them access into our networks right that structure and that management are we are we 100 aware of what the

implications of that are right as security practitioners and for me as i was going through this the answer was no right people started managing things differently the net flow was was odd people were touching different things that we weren't aware of because of the structure that we had put in place so our understanding had changed we went back and tried to look at our baseline and well we had to do one but it wasn't as robust no it meant the prescriptive measures of my yearly requirements to then answer those questions be like okay this is what it does this is generally what people are doing here's what's happening well there are massive deviations when we started

integrating this work from home model right we had i.t and we had hr accessing the networks externally to internal so now we have to make comparative analysis of understanding what the implications of that are right and start building a model for threats relative to that so then security changed our our context changed and that's a difficult problem to overcome because reframing and re-un and understanding holistically what is now critical versus what was critical is different right it was easy on-prem to say hr you're just going to be doing these things with these things i t you'll do this with that i can manage that i can monitor that but it changed well now we have to support hr accessing

multiple different things different people with different software with different devices tying into this vpn concentrator than accessing i i can't necessarily manage those things right now because i didn't take that into account as i was building this work from home structure so as i try to understand what threat modeling is and understanding the data making sure it's one common information model making sure i can understand it and then advise people on it and then also in part is security practices on top of that was very difficult then the impact was massive right it was very hard to then understand okay if they do gain access to whatever our vpn concentrator is when are they going to let us know

and who does it impact and how big is that impact going to be to what they touch on my infrastructure right that's a massive lift to say your organization might might have 100 people in hr right and they're they're touching several different things i didn't expect them to do but have i taken that into account to understand the data so what can we do how can we respond at the end of that long road and how can we understand it am i going to say destroy it and rebuild it now that you have the understanding no because it's not feasible it's too much to say hey as security practitioners we now understand hindsight being 2020 we need to tear it

down and build it back up well i don't know if you're gonna keep anybody who's gonna be like i just worked on this thing for two years it was fine two years ago why am i going to rebuild it well let's just apply some principles that we can then manage and monitor and integrate to our day-to-day to have a better holistic understanding and one of those principles is just enough administration in powershell right as a security practitioner and having people external come into my environment i want to understand and define what they're functionally capable of doing so i can monitor it and if something is outside of those bounds i can then respond to it in a more agile

manner now with just enough administration we can define groups that over time have become bloated right but we can give them the responsibilities of using specific modules or specific commandments to do whatever they need to do right and if they come to us and say that thing i was doing last week isn't working what did you do what did you change well maybe i didn't have an understanding and inventory of what it is you were trying to accomplish so now they're in a roundabout way advising me as to how their interactions on that network are going to impact me right not just as the the manager of it personnel but as an advisor to those

that purchase products right integrate products integrate solutions right if we're just building on building on building with with a product and we're taking the human thought and the decision making away from from your your junior analysts or your junior i.t specialists well you're not staying ahead of the curve realistically you're just maintaining the status quo that prescribed measures that are necessary because that's what keeps your job going but it's not progressive it's not keeping you ahead of the adversary in this type of work from home environment right signing scripts is a simple solution as well you can manage and track them right you now understand as let's say an adversary gets into your network and wants to lock down the

entire enterprise by pushing out a script to then you know encrypt everything for ransom well if you're signing your scripts and you've denied the access of unsigned scripts you've now imposed the cost of time on an adversary right and now we're building a slog a quagmire into our networks but it's it's a designed one we understand it because we're designing it we're imparting these principles right we need to define who our trusted hosts are when we built the work from home model initially there was a lot of what we got to let them do what they got to do we don't understand how they're do like hr is telling us logically how they're doing it but how

are they doing it how is it doing it right well okay they need to touch a lot of stuff so let's just give them access to touch a lot of stuff but when it comes back to that baseline deviation and what they're actually doing have we gone back and refined it because if our adversaries are there they are they're absolutely doing that they're getting ahead of you as individuals right you as an organization use a team because you're taking prescribed measures but not defined measures right and then isolating remote systems understanding why steve has to come in via one jump box and that he told his whole team that he has to do that same

thing when we have 17 different right now isolation and understanding to then make more documented refined and defined decisions we can then impart all four of these onto our network and respond more effectively advise more effectively and build out models that aren't burdensome right and then we also don't have to tell our leaders hey this thing doesn't work because when we built it we missed too much so that we can then enable and empower our teams to then be like well this is a new way of doing it we can impart this on there this is a new function or feature that we need and those advisements make what we're trying to do more effective and we don't always have

to start over yeah so again you know back to the whole principle of this talk this is not a you know if you do this you'll be successful list this is a subset of what you could be doing but again it's about checks and balances if you are implementing these controls in your environment what it's going to do is provide some mitigation against a zero day against your vpn concentrator but now what happens when one of these have been compromised it's always iterative right so if you are allowing only signed scripts what happens when the adversary can now sign scripts and deploy throughout your network do you understand the baseline the additions the unauthorized editions the deviations

from all of these activities someone just added a new host a new trusted host overnight who did it why is there changed management in the environment that's robust enough to account for these anomalies so [Music] so then we're going to go to a more defined model right so when it came to exchange server abuse with proxy log on there's plenty of documentation of when it the the the the after effects of that event right we identified that our exchange servers are internet facing we know that showdown exists we know that there's an api that's going to be scanning for something whether it's an icon with an md5 hash that's tied to a version of an of a vulnerable system or server

right they're then going to take that and they'll use cve trends on twitter and be like okay does this apply can i use this this resource is there a poc that exists on github that i can then leverage and use on this on this environment right then they're gonna aim for privileged access and remote code execution right but their ultimate end state isn't just that privileged access it's they're leveraging the peers and the trust relationships that we all have to find because of how the enterprise has to work right we have to support these relationships to then manage monitor keep keep up to date understand the health and welfare of these systems but they're also interconnected

throughout my enterprise my admins are going there because there's no fault tolerance we have we have a load balance system with eight exchange servers a primary and an alternate and they're going to these boxes right how are they doing it why are they doing it this way are they using rdp right are they using secure rdp because they're trying to do a two-hop jump without with which is difficult with standard rdp or are they doing it in some other way right now our goal as defenders is not to disrupt those essential services but it is to inform and advise to then protect them and like i said provide that quagmire for an adversary to then impose the cost the the the cost

of time on them so we can get ahead of it we can understand it so when things start happening that are outside those bounds those left and rights that we have to find we can then answer them so i looked at a few things for this right and i know i just mentioned like they're using rdp well they need to stop like it's controversial but stop doing it right we can start using openssh with windows for windows right and we can tunnel through the network and use it for both domain joined and non-domain joined right so we updated our modeling we said okay if we're going to be doing these things and we're going to try to

stay ahead of the curve how can we do that if we're doing these prescribed measures that were there three years ago our adversary if he's if he or she is there they've been there for a while right they know they're aware so we change we modify and we adapt right now change frequently and i know what you're thinking i know what nist says about steve changing his password is not great practice but we're not telling steve to change his password we're looking at it from a organizational and a team perspective right so regularly updating local administrator and global service account passwords cool and then rolling regularly roll kerberos every 45 days and like these are simple

we know that the golden and silver tickets are going to exist we have integrated systems that are legacy that aren't necessarily very uh the question hasn't really been answered right for those systems because we haven't integrated them into our new features but we don't have the team that understands the legacy systems and their integration right so we need to understand that you know just like certain viruses and diseases they're going to come back right so for not paying attention to them in part making making simple solutions a part of our day-to-day in our process then the the argument can be made that it's going to be leveraged we need to go back it's just an example of going back

and understanding what was relevant what are we seeing and how can we defend against it when it becomes relevant again right it's iterative and we should be able to understand and document those things and build them up over time now the big picture seam data integration we're ingesting data into our seam solution all the time all the time but what i what i haven't seen in in my job uh was we weren't integrating the it team solutions or the the local defender solutions into these data sets we were saying oh they're doing whatever they're doing we're doing script block logging but are we looking at what it is are they signed and if not why aren't they how do we

define those things and integrate them into the seams so that we can then do checks and balances on ourselves so that those that do have access to that splunk enterprise solution when they get access to that admin do they not have access to splunk as well right how can we understand that how can we audit that we need to provide those solutions into what we do or i would say we are the problem we're advising but we're not checking ourselves towards that advisement that we're making right we're like oh hr is doing something weird but maybe sometimes we're doing something weird too and we need to be aware of that now as i said the leveraging peers for

trust relationships so hear me out right bloodhound is a fantastic tool at the end of all these things you understand what the adversary is trying to do you know even going back to the principles that work from home you defined it all you have admins running everywhere you have users accessing things you have sessions existing here and there you have all this stuff happening but how do you provide somebody that's so what right i can tell you all day there's microsoft has a bunch of scripts that'll remotely query my active directory pull down these relationships and map them out for me that's a lot it's a lot to digest and a lot to manage right

so i would argue using bloodhound is one of the best ways to graphically represent those trust relationships the interactions on your enterprise and there's a lot of functionality and modularity to cater to what you're trying to accomplish because you may not be responsible for the entirety of your enterprise but for the areas that you are responsible for to provide that so what to somebody to get their buy-in not just being like well we could just buy something right and cut you out and then you have to learn that new tool or resource no you should be the ones providing that so what to your leaders and managing that so what right presenting it so hey we have a person

over here that we didn't account for they had 13 sessions what we did was we managed that and now the adversary has to choose a different path not quite defined every time but at least we know where we are putting roadblocks and we're applying the quagmire right pass it over all right so again it's about intent so an attacker wants to use the trusted relationships not because of where it lands them but because of where they can go they want to deploy the crypto miner they want to deploy the ransomware throughout the entirety of the network so how do we proactively visualize where they could go and it's about again a couple slides back we had the model of

do you know where your critical assets and critical systems are if you don't bloodhound is one mechanism that can provide you a way to visualize that but also you know where are the noisy systems what are the systems that are talking you know with you know 10 000 open connections running ad admin why are you doing that like do we know this or was this some you know admin that was just trying to automate their job so they can drink coffee and play video games all day and you know from that you know to have the the data presented from bloodhound actually presents a heat map where you can quantifiably say where your critical systems critical connections are and

actually do like an exploit control flow graph of you know upon here they could enumerate here here and here based off of the existing trusted relationships so again the sludge this lock to actually incorporate barriers along that way to make it as slow as possible so in a smaller environment this becomes doable when you have these massive enterprises that span multiple states multiple countries and you're dealing in petabytes of data and dating how do you do this it becomes really enticing to start you know investing in aiml solutions that can you know automatically reason based off of some ground truth what is benign and what is malicious [Music] so you're not going to be safe just

because you're using next-gen solutions right so at its core these are just one more appliance that could potentially be a point of vulnerability an rce and your aiml solution presents the attacker an opportunity to change the ground truth to say what is benign what is and stuff leave right on through but let's assume your appliance is never going to be exploited that it's going to be perfect all along there are other ways an attacker can influence the classification of the data it's looking at and it's a field of study called adversarial machine learning so let's talk about false negatives first this is where an attacker wants to slide in undetected so what it can do is actually find a

benign file benign artifact and to start inducing changes on that file and just introduce it to the ml solution like hey this is fine and so slowly nudge it down to the actually full on weaponization of that file where it is the exploit it is the inject into the environment but because the aiml solution has seen it thousands and millions of times it has been trained to say that this is okay i've seen this a million times it's just bob from accounting let them on in and that's how you actually train those classifiers to let benign things in without it flagging now let's talk about the the noise the false positives this is where an

attacker wants you to lose confidence in your solutions that you have alert saturation where it's like oh this damn thing doesn't work i don't believe in it it's because they've done things like you know behaviors on startup are now being flagged as malicious default windows system files are now automatically popping up across the entire environment as a malicious the admin now loses confidence in the appliance that allows then the attacker to slide in through the noise that we now know that we're generating 10 000 alerts a day no one's looking at it the logs are rolling you know they only have a retention of 1000 logs chances are no one's going to see me come in

and these are the type of you know aiml could be a value added to your environment but you seem to understand the implications they're in all right so throughout this talk we've talked a number of things that you could do to assume that the most trusted security solutions in your environment may potentially be the point of vulnerability and because of the trusted relationships someone's going to abuse it to own your entire network we have a list of things that we can provide you uh both online and in a hidden hidden slide that are some references that you can you know look at what makes sense to your organization but again it's not about you know one solution fixing

everything it is iterative it is understanding the implications of your actions and assuming that you know no good deed goes unpunished that someone's going to exploit your signed script policy someone's going to exploit your ml solution always wear your red team hat even as a defender and assume something will go wrong so with that uh we can take uh any questions comments concerns uh we like said we do have uh some resources and matt has about another five pages uh that we'll add on uh in the twitter version that we upload later so please uh also this is uh matt's first infosec conference so uh if we give him a little uh

but yeah uh any questions uh y'all want to do we can do that and then uh if there are no public questions we can uh stand on the side uh it's in about half an hour before the next one kicks off so any questions please [Music] uh so is the next talk gonna be about absolute machine learning so uh about 2015 i wrote a paper on the implications they're in uh not uh it's called uh [Music] summoning demons uh so it's online uh where you know it's actually within uh your trusted solution and it talks about a number of threat models like uh passing floats and doubles causing uh enough uh loss of confidence that you actually

sway a classifier if you're not programming uh safely to actually account for uh additional decimal spots things like that uh but yeah so so many demons is a paper that you can check out on more of that topic and again there's about a million other papers in the field that can talk to you more about you know ways that you can abuse the oracle abuse the classifier to do bad things any other questions please so i guess um one of the things that i was wondering about during the thoughts was [Music] i guess i don't give any guidance as far as how you go about not rebuilding from scratch yeah like how do you go about improving what you have so

yeah so i'm let matt take this one we did about a two hour conversation last night about this and so to repeat the question uh so some of these things are going to break your environment or could potentially break your environment or like hey you know at some point when do you need to nuke the entire network and just start from scratch so matt in in what i've uh done throughout my career as i advise and understand these networks and see that the solutions that we would want to provide are never so burdensome to the the customer right the local defenders that they can't functionally do their job and you have to rebuild the environment

right the the solutions are aggressive taking measured steps between each one to but first identifying why you're there what you're doing what the real question is the work from home model is is a great example right i didn't have to nuke even though they wanted to like the question was do we just have to move everybody back in like do we have to bring them back in for the on-prem like it's like no like you're assuming a lot of risk sure but somebody is managing and assessing where that risk is where's that documentation and the answer is no there's no documentation okay so let's define what that is right you're trying to just enable people to get in right

and you're understanding as we go through modeling where those faults are then you can take measured steps to defend against like just enough administration it is gonna break some stuff initially it's gonna happen because you're you're defining if you don't put the right commandlets in there it's going to deny all commandlets explicitly so you use nothing but if you pick and discern what you're trying to accomplish and take a measured step through it involve your it teams involve your administrators and your local defenders those answers are easy to overcome that you don't have to rebuild it because sometimes what they're saying to you is hey we're just we're using these microsoft scripts to do x y and z but this is a functionality

we need with them not the whole thing here's the variables that we pass in we can define those variables right we can define the axis and the means and the approach that they have they're all using one jump box that's the only one they're aware of well that's a failure on our part to inform them right the only time it ever becomes a nuclear problem is when we've defined as we're going through it they had an incident response a year ago and they said that they cleared it because they used this measure to get in like like proxy log on they got in but they cleared it and it wasn't actually a true clear and then

they built more infrastructure on the backbone of it and it's like well they logically obliterated your trust relationships and are everywhere they're not really they're to be not trusted anymore right then you can talk about principles of rebuilding but it's usually not that aggressive and it's usually just providing the little measured steps one at a time it's not an overnight solution it is a stepped solution with decision points throughout that we have to i mean the previous talk said it great like we don't in enable people to provide those those decision points for us and the guidance because they're working the mission if you enroll them in it you never you generally i don't say never but you

generally don't have to nuke whatever solution they provided you can advise them and assume risk in certain areas and then they can enhance it over time and that's really what the the principles are for um and then managing your semen and allowing people to uh to take measured steps for them protect for their protection anybody else have any other questions great all right we'll be over here thanks