DevSecOps: When It's Not a Buzzword

foreign

[Music] thanks for attending my talk my Talk's going to be on uh devsec Ops when uh it's not a buzzword when you have the goods right so we're gonna go over a few things it's like the weirdest longest title ever so I apologize if you ever have to type up any kind of uh agenda for today so all right who am I my name is Brian Davies I've been in the software world for over 20 years I've been a developer for most of it actually manager director and sales I have done a lot of Investigation of insecure code I've done a lot of product management stuff in designing secure systems so I kind of have a little bit of

experience in kind of a little area of all the areas I guess and I currently am an sc and I work at Splunk all right so what's my talk about well a big thing I want to focus on when you start talking about devsec Ops or stuff a lot of people don't necessarily know what that means but I also want to talk about the idea that production is too late right and that's going to be a big part of my talk today we're going to talk about what can we do about it and I'm also going to share some resources with you guys supposed to talk forwards for you guys for developers operations teams security

teams and just sort of curiosity I know there's a lot of students that are going to the today's talks how many students are in here today so like a group okay okay good and the students are you mainly in security World operations cyber security okay okay so there'll be some stuff in here for you so all right and uh one thing to if we got the subs going for later it's going to be a dance party at the end I'll be a surprise but um if you guys have questions during please don't don't be afraid to interrupt me okay I'm gonna be going through a lot of stuff going kind of fast just please interrupt me

don't you don't have to say Until the End okay all right so why am I giving a talk well I guess I'm a sucker for a pun for punishment I did one last year um so I'm going to do it again this year it's actually quite enjoyable to do a lot of work to put them together but not as much as what James is dealing with so but the big reason is the containers and micro Services right they help us scale our infrastructure they help us scale our applications but they also massively scaled our problems right they introduced a lot of problems that never existed prior to using those systems okay so what is the problem I'm going to

go over first some of the stuff we know also got the little sleepy red panda you know this stuff is the boring stuff I don't have to worry about it too much right we still have security bugs going into production performance issues are still in production right we're still doing last minute changes to infrastructure who hasn't done a midnight patch right they suck I don't know if you've ever had to do those for the students it's it's a terrible thing it always it's always a panic you can never get a hold of people there's no food you're stuck in the office you can't get anything like it's just it's just a it's a nightmare so containers in theory

they're easy right it just it just works it's magic they're light they don't need much I.T resources they're set up right away they disappear to clean up it's just it's they're really really nice however that's not the reality container management is super hard right containers can crash and disappear and you don't even know there's no indication that that container is even running unless you're capturing logs which a lot of people aren't they just go up spin up and then they're gone so you have an error in the application you've lost all that information you don't know what's going on another side effect of containers is we started using microservices for architecture that's also really hard

it's complex it's like it's such a disaster to try to troubleshoot a microservice because these containers going up and down while you're trying to trace something through like it's really really hard and like another side note that a lot of people don't even realize containers are so lightweight because they share the OS kernel right so if you haven't patched your OS you essentially haven't patched any of those containers right everyone thinks they're just kind of magic they're off on their own oh they're fine they're going to be good but they do rely on the OS kernel so keep that in mind all right so we have this mess and we're going to add people to the

problem when does that always work right we're not adding more people to an issue first we're gonna have the development team so what do they do what can they like what are some of the things that we have to deal with the development team right are they actually testing for security problems uh I don't know if we have any developers in here but testing for security was like that wasn't a thing it's even now for a lot of shops that's not a thing right are you tracking the infrastructure changes that you're doing to uvt that's a weird thing about the operations and Dev groups they have this weird understanding that uat is for the developers everything else is the Ops

Team so the uit environment developers can change stuff in but they don't always track those changes and then all sudden it gets deployed we have deployed an area that doesn't line up with the uat test area right and development teams don't really monitor containers they don't have the tools too all the stuff around developers is all about the local debug or even some remote debug they don't have a lot of stuff for handling containers the security team so do they actually share what they find security teams like to be secretive they like to take all this stuff but it's like it's like their bonus is dependent on how many times they bust into the systems right you got to share that

information if you don't share that information it can't be fixed so that information has to be shared and has to be shared in the usable usable format if you just tell a developer hey we're broke into your system yep oh sorry I'm standing back I'm like if you if you don't tell the developers that you've broke into their system how can they fix that problem right so that's you just got to share that information right and the other thing that security teams sometimes run into and this is the operations team too but is uat and staging are they set up this thing right that if again it kind of goes back you could potentially be

testing in an environment and deploying it to something that's completely different and the operations group are they actually monitoring their UAP and staging as I mentioned I should go back for a second uat is everyone here know what uat means there's a bunch of students right you user acceptance testing is typically from a Dev world that's where they do everything that's their last stop before it goes to production okay so is the operations team looking after uat because again that's kind of your last stop to find performance problems right or sort of architectural problems okay and then again who actually owns changes to the uat environment right and is the dev team are they working with

the operations team to figure out what the design should be this is actually a pretty common problem devs like to just kind of do things on their own they developers like to work at different hours and operations team too like kind of do magic stuff in the middle of the night all of a sudden they get inspired so you can't really share that information with an Ops guide help you figure out the best path if you're working those hours so you end up with two worlds product world and a corporate world okay product world for the development teams careful that's basically where does the product live where does the product get ironed out right that's the uat staging and then

you have the security and operations team they were able to corporate world right they're worried about threats they're worried about performance they're worried about I have a client facing application it cannot go down I have to make sure the data is secure for it I have to make sure that we don't have Insider threats that are trying to steal our stuff and work for a competitor they carry both out of the corporate world they kind of leave the uat world completely alone and vice versa the developers don't care at all about the corporate world they don't care about the accounting system right they don't care about any of that stuff they care about their little uat world

so you end up with Silo teams using siled environments so the result is essentially a mess right again the security group only cares about corporate so they only test their security controls against the corporate if you have an application that's potentially being sold into certain companies you might have to follow a certain audit requirements like c-soft audits or PCI audits or whatever so that would be the one time the security team will come in and work with you to do audits other than that they lead you essentially alone the infrastructure performance is only tested again in the course right they don't only care about like do the clients if they're trying to purchase my whatever system I'm selling is my cart

going to survive I don't care if it breaks in production the ones are in the staging no one's actually buying that I only care if it breaks in production so you have these two groups looking after two and then they don't really share any of their findings which is another problem oddly enough that's a finding or a kind of a common theme I've been seeing in the other talks there's been a lot about kind of sharing knowledge in the besides this year okay so the real problem is production is too late right you can't I'll get into it but like it's just it's too late you got to solve these problems before you get into production

so there's stuff you know about this right fixing production is expensive if you are a company that's selling stuff e-commerce if your production sites down that's bad news right it's very stressful on the teams there's I know people that have quit their jobs because their production environment was too just broken all the time all they're doing is working at night working nights weekends it's a very stressful stressful environment if your production site is broken it's always middle of the night it's always hack fixes who here has actually done a proper middle of the night fix in production oh one got two was it after a hack fix and you kind of figured that I could patch this a little

bit better yeah it's always the hack fixes and if you can it works you're like I'll solve it in the morning right you kind of so you've kind of hacked together a fixed and then gone back to sleep for a few more hours have you actually fixed the problem or maybe even introduced another one right because you've just kind of got it limping again so that's also not necessarily the best way and you always need multiple teams right if you're doing these middle of the night things there's always one more guy you need to get a hold of one more girl you got a call like it's there's just there's not enough people and they're never around they know it's

broken they turn their phones off so they're not gonna answer again yeah production's too late right back Black Friday outage can result in tens of millions in Lost Revenue right I know one company they had a payment service that went down for an hour during Black Friday and they estimate that was well over 10 million dollars one hour so you know they were able to they they had the proper systems they're able to see it and reroute it they knew within five minutes what the problem was it was just the rerouting of the payment service that took a while but still they were like one hour 10 million dollars they figure data breach cannot be undone right

that's if it's gone it's gone it's out there you can't go and collect it all and critical infrastructure it can't recover and I don't mean like oh your network I'm talking about stuff like pipelines bridges that have like the you know like the bridges on the boats go underneath them uh hospitals that's what I'm talking about when having critical infrastructure if someone brings down a hospital with a ransomware attack it's not just the next day it's up and running fine right there's a lot of problems with stuff like that all right we're gonna go on a bit of a bit of a story picking on a little bit of a path here so who uses the weft

we had a couple hands okay web application firewall for those everyone know how many rules are turned off I see some snickering so those rules turned off right because it doesn't work otherwise that always just stops everything the proper athletes and zero traffic through that's kind of how they work so out of those rules that you have turned off how many are injection rules right a few couple heads nodding they're the harder ones to kind of work around from the software perspective typically that's where they end up kind of getting left on all right so we have a laugh we have some rules turned off and it's the ejection injection related so are you monitoring your web request logs your

application logs potentially you might even have some other firewall logs even though they're not going to see all that information if you're going to turn off a rule how are you kind of watching it right are you watching it you just disabled so you can do things like watching your web requests okay now the other thing this is really important a lot of people can overlook this all the laughs I've ever used anyways there might be some that don't do this you have three options with a rule you can block it you can disable it or you can log it so instead of disabling is what most people do let's turn it off I don't even

care about this rule I'm just going to turn it off but instead you should log it that means if the Waf still triggers it saying hey something's going on it at least puts a note in the log so your security teams can find this stuff later right too often you see people I'll just turn it off we don't need it like uh we don't care about PHP injection because our site uses.net let's just turn that rule completely off well you still may want to log it right so that's just one of those little things just kind of keep in mind so in this scenario we have some rules turned off right just disabled completely then the audit comes

and they find these they always do it was fine and complain about all the stuff that you don't have turned on the rules are turned back on they test it everything works okay well maybe that rule maybe we fixed it we didn't even know we had that one release when we switched the framework uh it's all good off a ghost production all right now it's a problem okay the checkout process starts to randomly fail doesn't fully fail just randomly and not very often again just random light failures so it's like well maybe it's that rule but at the same time when we enabled the rule that we push any other code and we do anything what's

going on and start looking into it you see orders are being abandoned not completed you take it to it a bit more you can start to see that well like what's what's going on what's there being essentially they're starting to tiement so it's starting to happen right so what is that well if you're always popular free-form text box right and you see this on every checkup shipping instructions special shipping instructions leave at neighbors apostrophe s oh apostrophe that always trips up just looks for apostrophe locator but anyway people use special characters in those boxes they'll maybe put like the ten dollars is Under The Rock and there's a dollar sign in there now right so whatever but those special

characters will trigger off and now that you've turned that rule back on this shipping box is starting to trip up the WAP but only in certain situations this is a really hard problem to find right and then what happens right you have things that are timing out what happens to your customer I've been sitting here for like 30 seconds and internet time this might as well be three days right they'll go somewhere else oh well Best Buy also sells it I'm going over there their site doesn't doesn't pause on immediately so now you're broke right because your site's no good it's stopping all these things and again it goes back production is too late you have to catch these

ahead of time right you can't just wait and hope they catch them in production so what can we do how can we fix this I can use a drink of water [Music] all right stuff we know we're already doing this we've already been we've seen talks in the stuff even today cicd devops unit tests infrastructure's code is a good one because that's starting to be a lot more prevalent which is good to see I don't necessarily agree that people are using it properly but it's more my opinion I'll get into that in a bit threat hunting hog monitoring that stuff is all happening right now which is good okay so what do we have in order to get better we first I'm

going to say let's take a step back and see what do we actually have maybe we don't need a bunch of stuff so what do I actually have the dev teams have a crap ton of tools profilers profilers more profilers analyzers like it's it's crazy debuggers every ID you can imagine Source control right they have a lot of things there's tools are really focused on kind of the local World though right their system maybe a Dev test or it's a very minimal container system or if it's a more of a monolith it's a smaller server you know well single server instead of a server farm right that's kind of where their tools really really shine

if you're Ops teams they hopefully have infrastructure as a code systems and I really hope their scripts are Source control they're probably not most time but that's a really good habit if you get stuff from Source control you can actually do iterative changes to your infrastructure as a code you can go back imagine you won't have notepad plus plus open 50 tabs across the top so it's every little change you've done while you're working through a problem right Ops teams have really good container performance monitoring systems or at least they have some available to them in-depth infrastructure like they have a lot of really good monitoring systems right and they have a budget typically not always but more budget than a Dev

team security teams all right they have they have more open source attack tools than there are like JavaScript logarithms right like there's just there's so many of those tools it's amazing how many there are they have really good monitoring systems detection response systems security groups are really protective of their little world though like they call it a sock right their security operations center no one's allowed in there you're not allowed to see the tools they have it'll tell you a report or maybe send you a screenshot but it's there's some pretty interesting stuff right um and they even have more budget like how who paid for the stock that they have right like so they have they

have some money those guys so if you're in Dev or operations become friends with your security teams they have some neat stuff right so now we know what we have what can we do we gotta share this stuff share the knowledge share the tools right I'm gonna go through this a little bit so Source control developers know this inside of note operations team of security teams do not where are our operations teams storing all their infrastructure skills where are security teams storing all their attack scripts right how many of you just have a folder on your desktop with a tax scripts that one oh it's a windows so I have a Windows folder right like

like why if if your machine crashes or whatever it's gone like Source control manages it makes it easy you hook that up with an IDE and it's even better Ides talk to Source control out of the box you can do historic versioning they call them diffs if you're a developer so you can diff version one against version 20 and see all the changes it's super powerful stuff Ides understand yaml I mean I don't even understand the animal so it's helpful that an ID does and oh like very recently I was trying to help a company troubleshoot a yaml setup file and I had to edit it in a like a Linux VM like a GUI list Linux VM and I had to use VI

and I opened this up in VI and I was like oh my good grief I don't even know I was so frustrated trying to even understand the yaml file when it's just all showing up it's just like one color orange and it's not really you know the indents and the wrapping don't really look proper I couldn't even remember how to exit and save so I had to go get like my quick Linux cheat book like it was just it was so frustrating just to look at it you look at this stuff in an IDE it's color coded it doesn't wrap the line so it doesn't you know it doesn't mess up the indenting when you're

looking at it of course those indents are important so when the line wraps you're immediately drawn oh here's where no that's not the problem it's a line round anyway so use IDs they work nice with Source control it just the frustration level goes way down way way down so developers teach your other teams about these tools right developers and somewhat the Ops Team also already use development or sorry deployment systems so like octopus deploys a really common one I use that one a lot I actually don't even really know much about the other ones because I use that one so much those systems are great they have release management they have audit Trails like they have approval process

so essentially you can say you can also package stuff so you can say this goes to uat and that package can be your entire infrastructure your entire uh code base as well like it can be everything that's one big thing and if it fails it'll roll it back it cleans itself up those systems are really powerful the operations group I mean they really need to benefit from making use of those and most of those systems will actually manage things like Helm and terraform right helmet terraform and yeah they're powerful but the front end management piece is the paint right like can you tell me if you looked across your infrastructure which one of those VMS was deployed with the latest patch

Windows versus three weeks ago patch no right well I got one good you built something didn't you that stuff's hard but that's like what these deployment systems are meant to do that's what they do they can show you this area has this version and as long as you version your little packages properly that's some of the stuff you can have so log ingestion systems this is going to be well I'm not Splunk that's a big part of what Splunk does log ingestion um and so an example again kind of going back to our story and I've had to do this where I've had to match up IIs Logs with a WAFF log with an application Alma log

and if you don't have log ingestion systems which I didn't where I was working at the time you very quickly find out timestamps do not line up between these things my IIs log was my local server my laugh log was my cloud time right which I think actually was UTF time and then my alma was my was actually the database server time so I had basically three timestamps across these things and I was trying to find out well I know I have this problem it was actually a laugh and yeah era where it was an injection problem on a application it took me just forever just to line up the logs right so that's why these

systems are really helpful but quite often operations teams have these so here I was as a developer fighting with this exact problem that had I talked to the operations people they probably had a tool that would just always oh yeah here you go you can look at this and here's the stuff so share these tools share the knowledge figure figure out how to kind of come together and and show she'll have like a show and tell or something inside all right container performance devs definitely need this they need to be able to see that when they release something it's getting better worse some of these deployment systems and build agents they can actually monitor performance and fail a build if

performance goes down so there's actually a lot of neat stuff in there your sim right the pen test results are you sharing them with your development team right are you telling them the good news or only the bad news are you keeping some to yourself for the next time you're on a pen test you can get in all right so how can we be better I hate acronyms so we need like a Common Language We're not gonna get away from acronyms don't get me wrong but for the students that are in the crowd I'm not going to go through all these you are going to see these a lot this is just the app I have three pages of this

so take a quick look you'll probably run into these you may want to learn them if you want to take a picture of the screen it's fine but there's more that's the application ones security ones some of these we even heard in a previous talk on miter attack they talk about that actually in the keynote Sims or sock there's a lot of stuff around security and then the operations ingest into indexing right site reliable engineering it's like a job title I'm sure everyone's now seen some of the neat ones about operations I like the term they have time to Glass I like that term that's basically the time it takes for an alert to hit their dashboard I don't

know if every company uses that one but that's one I've come across I I like that one all right so now we kind of have a bit of a common language we kind of know what everyone has we kind of know what everyone does so we need to dream and wouldn't this be amazing if your application your infrastructure is tested for both security and performance before it goes into production can you imagine before it goes into production that's just that's like a dream right and not only that but if you find a problem you can actually track it down I'm just going to two sides of this like that's the other thing okay one problem snuck

through can we actually find it right so how do we get there how do we get to that tree well we got probably gonna need a few more systems most companies will have these like I said already it's just they might be siled in a group right you probably have log ingesting and indexing right Splunk clouds an example elk that's the open source what is that I can't remember elastic I can't remember there's like three open source projects if you just Google elk you'll find um anyway uh your sim observability tools those blunt gobs or observability and open Telemetry open telemetries get in quite big it's all open source you can their data can be collected into many

different types of systems now it's very very powerful right your soar tools and your attack Frameworks and there's a lot of attack Frameworks I'm picking on miter attack because I've used it so the other ones I'm not using in here because I haven't used use them so I'm not listing them there's a lot of them um but I'll show you some of the minor stuff in a bit all right so now we have these tools we have everyone on board we're going to try stuff a little different okay so what do we do well we start the same okay I'm still writing the application okay that's good devops build deployment scripts now so that's a little different

so we've got the Ops guys helping we're building some deployment scripts but you're building them for both the application and the infrastructure that's important the infrastructure one's important right now does anyone here build infrastructure scripts for uat production well three hands okay we have a little ways to go with this then but the idea is you want to build them for both because essentially what is uat it's testing what better way to test your infrastructure scripts than in a testing environment makes sense right so let's use testing for testing okay so we got our strips now security steps in so the devs will work with the security teams to come up with the security code analyzers there is a ton

of these as well oh wasp has a great list of them if you look a wasp and search for code analyzers you'll see there's like multiples for every language and essentially it's just a static code analyzer it's kind of the bare minimum you can do to look at your security stuff right so everyone there's probably a lot of people already doing this right some build agents servers have this already baked in so you might not you might be doing this so I didn't realize it so you have your builds now you have a quick security thing that's been checked so now you deploy it to uat but you're going to deploy the uat infrastructure as well so basically the

old uat is not there anymore you blow it away it's gone you deploy a brand new uat infrastructure right it's all containers containers are easy remember remember that it's easy so we deploy all our uat brand new on top of that you deploy all your applications okay so now I have a brand new uat infrastructure brand new uat applications and again if you use deployment systems like octopus a lot of this stuff's a lot easier you try to do it manually it becomes very very frustrating very fast and no one will do it you have to automate these types of steps otherwise it won't happen people will start going down that path and they'll abandon it

okay so now you have uat and it's like your new Sandbox essentially right now you got all these tools you can point them to the uat because again remember the end goal here is that production is too late so you turn on your log ingesting right so all your uat it's going to have firewalls it's going to have networks you might even have ldap in there uh if not no big deal you can still pull the main ldap you can have your system on perfume on you're gonna have all these logs right so pull them into a uat index into your in your log ingestion system okay and those ingestion systems they can be set up via infrastructure scripts

right in fact that's the most common way those systems are rolled out because you're typically rolling them out to hundreds of hosts right so they already have a lot of that ease of use baked into them right your sim you want a uat Sim so I don't know enough if you want to have a second Sim or if you want to have a second dashboard second area but what ideally though you want kind of a different set of alerts different everything that's independent of your production and corporate environment and just this little uat Sim however you do want the same dashboards and same alerts because again what you're doing when you have stuff like the Sim setup you're not just

testing your application you're also testing your sim right that's what's kind of neat about having a uat environment you're kind of doing a double test right if you set it all up and you do some tests later and your sim has nothing well perhaps your sim is set up wrong right maybe your application isn't that secure maybe it is but it gives you another thing maybe you should look at your sim because again if you find out your sim is not set up properly in production guess what it's too late right observability uh there's a lot to install a lot to set up with observability I'm not going to get into that observability the concept is

basically you're watching a product from the very beginning to the very end you can watch the uh the user at a website level you can watch all that all the way through the application you can watch its calls to the database you can watch how that impacts containers you can watch how it calls out to other sites it's it's crazy observability systems see absolutely everything it's like a profiler that developers already use but on steroids right it's they're really really powerful systems and the reason you want to capture this I'll get into a bit more but this data becomes really critical in the uat environment which I'll get into okay your security orchestration automation response system

this one's a little bit iffy right you may want to copy some of your playbooks into uat but if you do you copy a potentially production playbooks into uat so if something gets triggered you might be turning off your production firewall so if you're going to move soar maybe it's easier to set it up from scratch I don't really know the best answer on this one yet because some of these sword products are still a little bit more immature so they don't necessarily have um some of the flexibility some of the newer ones those some of the more updates you can actually pull in like um uh playbooks from GitHub so you can have kind of a base starting point so so it

might make it a little easier but again just be careful with this if you decide to copy it then you do run the risk of actually running playbooks against your production versus uat okay all right so now you have some stuff set up everything's ready to go okay and that takes a lot of work don't get me wrong that's that's to get there is a fair bit of work but now that we're there what do we do well we'll start simply we can run our UI tests right developers already have these they'll call them synthetic users in fact there's even systems I will run these for you they'll call them integration tests essentially the most

common way of explaining it is the selenium test that just we fix a user clicks through the site all right who hasn't seen these super common most companies have tons of these already but your system is now instrumented as it's called and that's when the the hooks for the profilers for observability are in place so now you can actually look at it on a system like APM and you can see performance problems you can see errors you can see lag latency you can see it these process maps that you're seeing here these systems Auto build these you don't have to build that that's built based on the traces so you can look at that it's like wait a

second everything's working fine but why do I have two payment services one's called payment.old that one's probably supposed to not be called anymore you can actually see stuff like that it's working fine right that would never trigger an error but you can see that that's a problem by kind of looking at and investigating these tools in this case on the screenshot I have you can actually see that red circle and that's actually some errors that are occurring on a service and so you can actually but it's not always occurring so again you could have a scenario where some of your tests are airing but not all and you can see the little path through right where

the air is kind of coming from so you can kind of do root cause analysis further in these types of systems if you click on the map you can actually see what containers it's running on what versions you can really get a lot of detail and again that depends on having the observability of the collectors properly so it's really important next you can run your load tests right so now all of a sudden you're going to just just Hammer the Box what happens well again with these observability tools you can monitor your containers what do they do you can start seeing them scale scale up you can see he'll see them when it starts or not you start

they start getting color coded if there's problems you can click into them and see what's going on you can click into them and see what services are running on them those this container map and kind of the service map they you can kind of go back and forth most of these types of systems have something like this or similar there's also you'll see all the logs behind it so if you see a red box and you want to click it you can go just show me the details and you'll see the running list of events that are going so you can just look at all the events right there's a lot of stuff that's available in these and if you

kind of set them up right it makes it really easy from a graphical perspective to go through this stuff right especially load tests because I mean there's just so much information from Lotus okay you can run the attack framework so this one's pretty neat so I'm sure a few of you guys have heard of Red Canary right they have an atomic red team so they've taken the minor attack framework that exists and they've actually made an attack system you can just run it there's actually so that link there it'll take it to their main GitHub page and then there's a command line utility that'll run all of their attacks the nice thing is it pulls them from GitHub

so as they get updated and changed it'll just pull the next set of attacks so you can kind of point that thing at your system and Let it Loose now it will break things so kind of be ready when you do this like don't be like okay we're gonna have all our we're gonna have the users the actual end users are going to test tomorrow but today we're running miter attack well you'll probably destroy it before those guys get attacked so just be careful this but it's really neat because if you do this okay and I've done this on a small scale and it's just so cool well then your sim will detect this stuff okay most Sims have plugins

add-ons apps whatever you want to call them that actually monitor very specific attack Frameworks so in this case this is the Splunk Sim and we're looking at the the minor Attack app right and these are these they're always freebie add-ons for these systems right because it's meant Happy guys and basically what the miter does is it basically Maps All the known attack vectors to known blog hits right I guess that's the easiest way there's a log pattern which matches this attack but that's kind of what it's looking at so when you run your attack Vector right in uat this stem should just light right up right if it doesn't again does that mean the Sims set up

wrong or does that mean your application is stopping everything right if it doesn't stop it just doesn't light up at all it's probably something set up on the Sim which again so you're testing both the Sim setup and you're testing your application right and as teams go through this they'll start to feel it out and they'll they'll start to see when things are kind of one way or the other right the other thing when you run an attack framework if you have Source set up what did it do did it just start going nuts shutting stuff down right for some things it probably should have so again what's going on is it is it is it doing this stuff but this also

helps you test the infrastructure right the reality is most of this stuff should have been stopped before I made it so did that happen right so you have one attack happening in a uat environment and three teams are benefiting from this right that's really the kind of the key part of what we're doing here right so now you've done all your talks you've done some log stuff now it's the red team's turn let them loose you've basically built them a safe place to go break it go see what you can do right and they'll be like I said they'll be going to that little Windows folder on their desktop double clicking my little hidden scripts

and I'm gonna run right that should be in Source control for the whole team anyway they're going to run those they're gonna see if they get it right well again you can monitor this attack what did the red team do what what does what do we see when the red team's doing their things right did the Sim show up stuff again did sore go crazy like what happened the important things though for both the the soar product and the Sim products there's no one to expect right like like I said maybe you are actually expecting the Sim to see Zero minor attacks because you've hardened the firewall you've hardened all the access points there should be no way to get in okay

so if that's your expectation just make sure the team knows that as part of the attack right what are the expectations because that can be a little bit confusing if you're expecting to see a few things and you see a hundred right okay well maybe maybe we deployed the wrong thing maybe the infrastructure script's wrong so just kind of have some expectations as what's going on so now you've done a tax you've done load balances you've done all stuff you've actually collected a ton of data right so now it's a threat Hunters turn a better way to train threat Hunters than basically hey we ran a miter attack yesterday go see what you can find

right we'll dig through the log system right if you have a log ingestion system you should have all this stuff for them they should be able to go looking they should be able to start looking testing profiles and testing theories right so so let them loose the other nice thing about these systems especially if you've automated this stuff everything will be time stamped you'll know you're around the miter test from here to here on these times so if the threat Hunters think hey I think I might have found something without pestering anybody again if this is set up right they should be able to just look back at the little you know deployment tool oh yeah

this is when they ran the miter attacks it does line up with what I'm seeing I can go now bug whoever ran them and verify that what I'm seeing does does line up so again you've made this nice safe place uat has now become this magic sandbox for all that stuff right so uat think of it as no longer being for developers it's now for the devsecops right historically it's been developers off limits no one is allowed to touch it we got to get rid of that right these three teams have to start working working together you have to test all aspects of these things like when I say test every aspect like I'm not saying every aspect of the

site of the whatever the developer you know your Revenue generating application or website or whatever I'm talking about you need to test your infrastructure right you need to test your testing tools test your sim right is it actually finding stuff because in the day-to-day process you're probably not getting attacked so much that you're confident that Sims actually set up so you need to test those systems and automate when possible if you don't automate this stuff is really really hard so you'll get one guy that goes through it you think you'll get the next guy to go through it automated right there's tools out there for automating we're starting to get more and more of that in the operations

team please continue that the more you automate the more you get students and Source control the more you can make it repeatable right again I've had to do this before with software not so much infrastructure but if you have a release that goes sideways and everything has fallen completely apart to the point where we have to rebuild VMS and all that stuff how do you deploy the last known good version of your software if you don't have a deployment system you know what I was able to do I was able to go to my little dashboard and click deploy version three I'm not doing anything else I want to get the VM up and running I did

everything these systems are just Lifesavers like they're just amazing so make use of those systems okay all right useful resources okay uh just some screen or well URL against the miter attack There's the link um again they're just oh wasp there's a source code analysis tool the second one down that's the um that's invoke Atomic red team that's the tool that will run the miter attacks okay so if you guys want to get a photo of this please do I'll wait until I see all the cameras cameras go down before I switch it still still a few folks I'll get a drink of water spend spill some time okay all right next one same idea so

these are Splunk resources not just um not like the other ones were just kind of all over the place open source resources but so Splunk Lantern is an amazing place to learn not just Splunk but to learn these Concepts learn what a swords how does it work what's a Sim how do you just set up a sock the lantern is just it's gen like it's a general purpose amazing spot uh Splunk has a surge team that's her security response team anyone can sign up for their alerts you don't have to be a Splunk client anyone can um they're probably one of the first groups that released a blog post on how to find and mitigate log4j when that

then that happened they are a really talented team and everything they do is just kind of released it to the security world even if you are a Splunk user or another type of Vlog just type system user they will release searches if they find something they'll actually post the searches that you would use in order to find things so it's a really really good resource the user groups events events for students the events and the free training are probably really really good the events they actually do a lot of Hands-On workshops there's a lot of just marketing things whatever but there's a lot of workshops where you can actually they'll spin up like an environment in a

cloud world for you where you can poke through these systems and try them and Splunk I don't have it up here but but Splunk also does something called Bots I don't know if anyone's done a Bots event yeah they're they're fun right so yeah if you ever see on the event not Splunk and there's a boss event please I highly recommend they're capturing the flag event essentially uh but they always do something really fun they're always they're really cool stands for boss of the sock that's what it stands for so we think anyway I am out of time and uh yeah so thank you so much for attending if there is questions I can maybe answer one or

two before they kick me out of here anything no all right oh yes

my computer also we were all students so how high would you prioritize

okay so the question was for the students you know should basically should they specialize on a vendor product essentially is what you're kind of getting at right uh that's a tough one they're they're all similar enough that um if you can get access to one learn it the problem with some of these is if you're a student you're going to get the Open Source One and they're hard to set up like the open source one's kind of depend on well I have five other it guys helping me right that's not always the case some of them are pretty good but but that's why I had the the thing for the events the link and there's other

vendors that do these too but when they have those events they typically will have what they call workshops so that's the key word for a workshop and that typically means it's a Hands-On item so you can actually go in and play with those uh tools but I wouldn't get too fussed about um oh I only ever use this one or that one because when it really comes down to it they're all pretty similar really yeah do you have a question no okay anything else I think that I think that's it so thanks guys I really appreciate you uh attending