Attacking Modern SaaS Companies

so the only thing there is to know that your sex company so I know things we've got to do Wow last year we presented an attack on max pasture competition style attack just single thing just maybe choose different pastures and bless your client have a report earlier a man found such a terrible crypto invitations but today we saw free service so there's lots of software the service that you had we don't use Gmail TurboTax all the stuff it's super popular and it's a definitely new way software is being made it's huge industries about it's over 100 million dollars in a year so it's absolutely enormous itself of the numerator to be developed and I really liked this tweet

by Kambiz OB that in 10 years will be only three major operating systems and they'll be AWS Google compute engine manager which is kind of a weird thing to think about with lambda everything you know that's an interesting was that mean if there's only these three new operating systems like we don't have rootkits frame to me that's you know so what will that be so my goal for this talk is to kind of explain how we make sense off where how do we do continuous delivery why that's useful from a security perspective to know what what you should look for what what's typically don't belong this is kind of huge broad topic so this is definitely

an introduction from going breadth over deficit in any particular area so I start my talks with in conclusion so my conclusion is that if you have access to developers laptop engineers laptop you probably have access to everything if you have access to the build server you probably have access to everything the artifact server the pick management server and of sequel database the container the configuration the cloud API really if you have access to anything probably have access to everything and that's obviously a huge problem as a defender you have to try and make sure that this is not true does the attacker you take advantage of this so let's otherwise that's companies are different than your average hundred or

other company so it has to be some fast iterative development process they typically don't have like six months release cycles if tons of automation is very very common for a code that you write just be pushed out to the production very quickly you have a lot of empowered engineers interest rate because engineers are able to just write code and do things and improve your product very quickly you know a ton of brand-new very great powerful tools that let you install packages on all 1,000 of your servers or quickly and easily and there's also a lack of security culture this is pretty common to many companies but SATs companies in particular seems it's not here they're also weaknesses so

we take the the strengths each one of these drinks have a corresponds to weakness there are also the linchpin servers there is like the build server all this stuff is really really critical if you get access to that you get access to everything there's not much security monitoring despite the automation collecting logs it's pretty expensive small licenses are pretty expensive so they actually tend not to do much security lotteries and just to do a very narrow things empowered engineers really means there's no security planning or anything like to talk to security engineers that the SAS companies know like I didn't know my developers started on paredes that was listening on a public IP address with no off what's

like we just didn't know because had no idea because the fast iterative development process just did not allow them to do anything they're powerful tools means you can use them for evil later and a lot of security culture really means it was little to no budget for security so let's talk about how you lose that some things were companies per hour this products made so first engineer it's write the codes try to respect or something right they need to code it up they committed to some kind of version control get some version maybe I'll get up then it's news integrations going to build and run unit tests there's lots of different options for continuous integration of the most

popular Jenkins still Travis's use a lot of open source community and circle see iron ship or kind of kind of then we're going to play to from there it's going to be promoted to staging we'll just have a test QA environment sometimes it's automatic based on integration tests and then you can deploy it to production where your actual customers you can use it most companies that's still a manual process but really advanced companies like Facebook will have totally automated development processes so you can commit code and then a few hours later it's actually in production without any humans anybody cool so let's just talk about just how does this aspect so continuous integration Jenkins probably most people are familiar with

Jenkins you can put projects in here build whenever you very nice does build notification is that kind of thing how does that work first the build is trigger source codes are downloaded it's compiled tester wrong solvers package it's uploaded to the artifact server so here's an example of one way so we get back to where everything it builds it's kind of like a reflections on trusting trust I think if you can impact or everything you can easily so one thing I knew is everyone can spend a poor request of all the kid on projects right some people use Jenkins or they're going to github projects to build and test and I was wondering if you just submit the

pull request does it just run code on their Jenkins does if you go by the code of the Rankin's you have access to the bank design you can pack where they run so I found this tweet which is really awesome there are actually box on github that are creating pull requests just to mining bitcoins so all they do is they find projects that are running continuous integration they insert Bitcoin mining code into the tests and submit them in and as soon as they start running the test and obviously event you have to find that but so yes and answers absolutely yes I found the give up pull request builder plug-in and what this doesn't let your

github to Jenkins so but on this one unfortunately it makes you verify the patch so the admins have to look at the patch make sure it's not actually a Bitcoin miner so I want to figure out a way around this so I set it up yeah it's what happens is as soon as you make pull request the at the bottom sets in one of the admins verify looking make sure it's not a white wire I noticed it was five minutes apart in five minutes sounds suspiciously like polling if it was you know webhook event-driven it would be very quick so I said okay well let's figure out how it works it's every five minutes to get hub motor vessel plugins

gonna pull the Polaroids to see if there's a find everyone check to see if the offers whiteness did we trust this offer to not run Bitcoin miners or the pork west's accepts of the admin says that this is not a big if not post comment asking if so build the pull request amount of tests so this how we're gonna work around the we're gonna post innocuous pull requests like oh hang up the tests the bottom saying can the ad verifying yeah and you look say hey this looks great on remote tests tests displays that was in five minutes we do first push the different commit that is a new malicious commit and the reason why were force pushing instead of

just regular adding a commit is because document you send an email every involv saying hey there's been a new commit and you know what you should be checked if you force push the pump does not notify anybody so then you are going to be running a malicious code on their on their jacket as you can see you know successful finished and that this was with my special pushed Edition so this is what i did i just added a simple thing to the thing which is attacked and then in my tank ins you can see that this is building bullets and here at the bottom its attack so it was just very very simple this is you know pretty

basic and it it's very easy to do this sort of things if you ever find something planning a Jenkins migration it's always interesting you can get on that tank it server I take quick Google search to see how many people are using this plugin hooks in a lot so potential pentester it's sort of plug in or something similar or something that works so you don't have to use this bug so if you could get access to Jenkins you can write any file Jenkins controls so the credentials that XML this has a ton of useful credentials like it might have credentials tree or artifact server like load it might have a check for configuration credentials the job and two directories you use

maven and Java this is where all the libraries Linda so you might think my backdoor some libraries change the jars around secrets is exactly what it sounds like as much see your keys that kind of stuff and the workspace that's where all

that's different versions so the continuous iteration is going to upload to the server and the deployment software's go to download some examples these are the Java ones so today Nexus j4 Arbor factory these can be used with other languages it's not just job also dr. registry counts and artifacts of using or something so if you can upload to the artifact server you can upload a fact where versions of code so with just a simple command if you have access to this either you got this from Jenkins where you got it from the developers machine where you somehow got it from one if there are other machines you can just upload it's just HTTP it's very very simple and if you upload a new

version of their code when they download the code running whatever you want so how do you get the code from the artifact server they're machines there's a family two ways nowadays there is the doctrine or core OS sort of thing which is a virtualized OS which is not the same as VirtualBox VM whereas the operating system itself is virtualized and usually you have all of your dependencies included so you have your database volume only libraries in one docker container usually to use in future we build the whole image which is nice too then you can just pull it down directly configuration management software so what a lot of people use employ things chef puppet ansible salts cfengine

there's there's tons of these out there they can figure notes you can also deploy software and actually a lot of custom scripts out there and these custom scripts might have scripts to say every five minutes pull this as three bucket give me the versions of software doesn't download and people people do whatever they want to get their solos so just let's start my containers first speak to like all the different tools I found containers of stalker there's kubernetes Corvettes Bezos ATV Panamax retainer ranch airship here there's just so many of these I can't even keep track of code but it's just crazy and there's even more of my ops guy he's like you forgot these over ten of them and

there's just so many guys a pen tester don't worry about knowing every single one you're probably familiar with docker and super Nettie's making pooris those are the far away tables a popular one it's all these other ones you can punch as far as you go so there's a trick containers one real degree benefit is easy patching updating because it's just a container if there's a OpenSSL what you need to do is build a point everywhere it's great it's micro-services really well we'll talk about a little bit it's easy to automate building and employment of these there's tons and tons of great ones out there for building and automating docker images one important thing to know both

as offender containers but not security matters most the biggest danger with containers is that they're still so new that people don't know what security trade-offs yet they don't know exactly what you can do with your software so let's talk about some examples of potential or potential way to break through so kubernetes what this does is it automates the deployment of containers so you have obtain time to point your devastating production environments you want to orchestrate all of that so you might have an FCS Bastion host that you say well you have to SSH into this machine first like loot logging on it and have a good one inning of that so all nine years but Cooper daddy's lounging body past

that century completely and this gives you a show on be made on the politicking in this tunnels over four four three eight which is you know a little harder to fire along 22 and this isn't a security problem it just I tried to security we do know this and they were like oh I don't let my engineers ssh into production except they didn't know that they were running you know essentially a state remote show this and this is you know challenging to just you know challenging to restrict but it's it's very convenient for the engineers and that's kind of one of the big things you'll see with web stats companies things that are completely different

years doesn't use things are inconvenient like security so this is from a great talk crash course in genetics and security sector last year and just like I was saying like barrier so here's the example so if you run the doctor is so this first talker run is basically just saying start a docker container and I want you to map this users home directory to slash each dots and then run the batch command which the base is just copying that hash to each dogs would show which is now maps to our home directory and then make sure it's at UID for seven then if you look on the host machine as a stone goes between the root shell is

on my room and it said you might need we're going to run the root shell and root so there is absolutely not a security barrier whatsoever it is this is a design and we're attached to this so this is totally fine it's really trying to use containers that's just important oh it's something it's root in your daughter damn day we're under machine so you need to design a different security of course people just put chip on the Internet so this is for tenor this is a container management software I take portator in to show them and it's the first thing that came up this is a bad sign if this is on the internet because you have access to all of these

machines I can SSH into every single one of these I can start up new ones I can't believe all of these I tried finding the person with this wasn't contacting the identified can figure it out and there's there's tons of these if you just search for all these like doctor or sort of tools they're all over show that and this is obviously a huge problem for security if your engineers accidentally do this here's another one that scuba Nettie's actually got a bank it's password on this one too which is next level awesome not only you have an access to run anything but I also have access to their Jenkins so this is this a bad sign this is happening to you okay

talk about configuration management so pH management really solves this problem so your software your remaining we're developing things it depends on NTP running the time being correct because they could use it us depends of the recent kind of version because maybe I'm using Google alike so you need to do performance improvement and let's say you depend on the image magic hopefully not because of all the image traffic sort of things let's say this is what your software depends on how do you make this how do you make this happen are all 175 notes usually up-to-date how do you make sure there's software fine you don't like open SSL or something do they have all these packages installed how do

you peer review and approve changes all these problems are solved by configuration management even if you're using docker even a few beads and all these things you still use configuration management to make sure that the daughter hosts are running the right version doctor there's a few tools that do this the tool that's I think the most popular ones well that I use most of all chef so what this does is it kind of treats your infrastructure in its code so you just cry and what not how so how is basically all programming it's like you know programming and slightly less subscribers much so for instance in this example you're just saying 30 is going to be a directory called application

think it will be called it's called this it has owner service as this mode make sure you create it and make sure you create all the directories it needs and you can make really complicated objects like this application object and the same constructed and what's cool about this is you don't want you to worry about anything inside impotence so because you can run it five times as long as that directory is there and it has these permissions of customer shipping setup so people use this from sewing app packages making sure that their services are configured properly all that kind of stuff so the tool light is what you use to interact with chef so it has a big

file here this tells you where the chef server is and you have the User Content model which is a private art safety for nightly and so using these two things together with the developers machine or even from notes themselves you can now it was nice to interact let's check what can you do so you can list every note of the environment one thing with your pen down is you might have a list if you don't have a list every node ninth node list will just tell you every single chef server talks to it you'll find the database master install packages favorite machine so just I search a packages will tell you every single package that results you

can find versions of things you can find out of date you can find if there today needs to take some other add a new kernel version you do colonel release and this nice tool is just super super powerful it's built for late awesome system and operation site so every possible conceivable thing you wanted here one thing that I like is that you seminoles store secrets in here too so late if you just run night search and now just lists out all the different configure it where password and it may be something different some passwords in there but just looking through that whole command that's super useful because there's a lot of interesting stuff in there you can find you more

data life has of data bags and data bags are basically just like JSON walnuts and if you just do nice to have a list you'll give a bunch of named data bags like oh this is my SSL Certificates so if you do night to make sure what if sell certificates now you get their SSL certs you can encrypt data bags and it's best practice to encrypt them so if you come across Kurt's data bags you need to find encryption key first usually those are on developer machines or on the chef which means because there are all shared key you can run arbitrary SH commands this is like kind of the most powerful thing nice dozen this is like the killer

feature or five or six years ago for the persimmon you can just do nice SSH the star means which machines to run it also stores everything these drunk family problem this is this progress stage off and usually is the pen tester you don't necessarily have you have to make a compromise or know that yes this is useful for admins just to run quick commands stuff like that so if you want to back to where everything chef does the first thing need to do is you need to find the most commonly used recipe so I have this final recipes sort them make sure they're unique circle account so it looks like you see on this one is is the

most common it looks likes to do all these all these recipes are likely true so you should install make sure assistance both to do slack Handler all that kind of stuff so we get back to or any of these and we'll probably every single node because every single node so do is thanks for book download so the way chef works is it has cookbooks which are comprised recipes that's like kind of funny and nice right so you down the foot books which are comprised of recipes so we ran down the little cookbook and we're going to put our back door into the Z's recipe and we're going to just upload the cookbook back very very simple and easy so the

back door where I made is just very simple - back - or if you just run this every time just double you get some scripts that you've written and just you can make something that's much more chef compatible something that's like actually useful or something but this is just how you do that and then this will be wrong on every single no that has Asians gold which is probably everything so now access to so first what are they they're small services that do a few things the tree centered around some kind of business areas if you have a photo website you probably have a photo uploader of service if you might have a photo search service and you might have

a service and you'd be breaking them up it's like if they're early in my example if you're making to-do list out part of your app you have to do a service and usually to maintain exclusively the API there's no shared memory there's no data structures maybe you communicate via a database but usually it's rats hpsm JSON also common in section queues so rabbitmq Amazon that sqs and you know I like to make with micro services Morgan traveller does really like microservices anymore is that micro-services are like functions and typically we want our functions to be small maybe not looking for lives or small functions but musically we don't want them to be 800 900 complaining

monsters and instead they want them to be small and composable same thing that services you don't want your this is my company it's been one project it does hold two things because then developers start doing the funky things like oh I just spoke with this in memory I'll share this you didn't realize that it wasn't persistent so microservice is a very quick easy way to do development so why do we use it use informatics food all the time it's much easier to understand oh this is just the photo search server it's not to worry about uploading well that kind somebody just worried about search it's a logical separation very easily monitor rule which is nice you can say okay well it

looks like our photo search isn't running as fast as we want to but the photo uploader is working great and this is one of my favorite myths especially you know somebody who's running a service that needs to be on 24/7 the service for me personally helped during disruptions so the photo uploader doesn't work that's fine because the photo search still works at least part of your services still going to kind of gracefully degrades that's designed well and this is really important from my perspective it's very easy to test and release things automatically because if you only need to consider what happens with uploads you'll need to consider okay why do I need to worry about if it's going

to break the search is it probably won't it could if they decided poorly but most microservices much easier to test and reason about so here's an example so here's our user and they're going to get if they want to use it to do this application so they're going to get the front end which is either gjs or react or something like that they download that HTML or something CDN or some web servers they're gonna then along and they're gonna off to the off load balancer which is then going to talk to the odd service which is that going to talk to the account database that's gonna store user name password email student so the password of the master -

then once they're logged in they're going to talk to the actually to do service to create to-do items and to mark from done they're going to talk to it which then corresponds to the - to do services the to do services to talk to the office service to make sure that this person sending is the valid user and it's logged in usually this is cookies or something and said hey is this cookie still valid and they need to talk to the to do to a database thing to you know add something to do they might put something on a queue I couldn't really think of something maybe like it's some kind of like in the future we

need to do this some kind of batch processing a lot of services have this so you put it on to me cue and then the batch processing is can get to it whenever it gets to it and then it writes the database so as the pentester you also want to ask them this service the service all authenticated is that some kind of shared a is that rate-limiting on our load balancer how do we do input validation and also my favorite it's just not sequel injection upfront don't worry it's very familiar with sequel injection but sequel injection is usually tested like oh I have a new to-do item and all put you know a quote in the name or something but if you can

somehow get it so that it's actually in the back end of the system where it's not directly facing either from a queue or something a sequel injection is much more common the farther away from the testing surface you can get like way back there it's a very interesting place to do testing everything extra online on services there is to the left the line is kind of like the outside world and development actually pretty good and making that work making sure that's not totally garbage from a security perspective but once you get into the right side and that's where everything's that Google if you're on the network you're trusted and all like I said hard on the outside squishy on the inside

we've covered that analogy for like fire what else everything absolutely true for satisfying these so if you're eating them at work over there it's very unlikely that you can't get access to everything take a very very well designed sass that work that case the API if microservices communicating the API you need to figure out what the API assuming you don't have API documentation is baby tied up in tissue life's lot easier so obviously you can use there have a purpose week or something to figure out the domain or to figure out pads everything it's like for instance you'll often example economy to do the service stuff production example calm and you're gonna have a bunch of

these if my favorite thing to do is to use the web application description language in Java specifically there's a bunch of web application description languages they're very useful from a developer perspective because you can pull them up they look like this so this is the one produced by Jersey so if you had something like this so this is basically just an XML document describing what the API is supposed to do so we can see we're starting it slash and the resource path to do slash ID and we have a method yet on that and that corresponds to the Java things we call it get to-do item and then it's going to return a response which is allocation to

days and obviously your application not waffle is going to be much longer and more interesting and we're going to tell pen testers is that if you get this usually it's not useful to test like to fuzz or test outside of this so I'll try it / - do you know something else I'll make something else that doesn't normally work you have to kind of stay within the balance of this cuz that's your Jersey and Java we're really good at making sure they can put fits in this schema but as soon as you get into the schema thing that's where your developers are there and they've forgotten to handle the null case so they forgot to do something like that

they forgot to or they had a special ID with all the zeros that does something different that's very common because like to kind of go back to order themselves place to get this is usually just slash application model you can actually see this on some production services if you know where to look for it which is slash and it's it's often there and you get this very nice documentation so then you can point your herb sweetie there were some API buzzing if you want that it it's very useful so I won't talk about some more weaknesses of micro services from a security perspective so like one of them is this kind of connection between services so

one thing that that's always the problem with composable systems is complexity so while each individual attribute is simple like our search service is simple once they start interacting together it can get very complicated very quickly and there's this one micro service system that I know of that is booting up various other containers and everything and actually you can get into this state it's a kind of gossip protocol thing we're able to talk to everything that it knows it's like all I know these ten people all tell everybody it knows about the state that I have right and it's kind of cool micro-services architecture but the problem is it actually has these if it ever gets a

problem like we're one node things that it's down itself which is very odd in a bug it can start having these harmonic like things were half the fleet starts to go down and the other half starts a moment and certainly going up and down those things are very very challenging to do both so micro services are best in their life design very simply and they don't have a lot of complex interaction a lot of crud create read update delete that's that's like be kind of a key thing for simple micro service services service loss is a big thing that I did just speaking as a developer I know a lot of people don't know that you either

don't know or don't care to find out how to do the services service also how does the photos search service talk to the tagging service doesn't maybe needs to know what tags that users are allowed to search on how does it authentic as it as the tag servers know that the service service will have to do that the most common is just network level off if you talk to the service you're allowed in this is hard on the outside squishy on the inside I do share to custom auth is done the next most common so like somebody just can want some each to the inventor that's like X company off and they just set it to some string or you

said some shared secret leave a little bit of encryption hashing H max sort of thing on you do some kind of custom thing there usually it's the same key on every box if you get that key you can then do your services service off and my favorite is that I may pass through financials so you're acting on behalf of the user so if you talk to the service service it was the user authenticated with that and then the serve service was able I'm acting on behalf of this user so that that's how it's authenticated that's that's very nice because it's very consistent from a security perspective every service has to authenticate there's no special cases

there's a few other better ones like the spring framework has really nice servus servus off again but actually making food okay so the cloud what is it every notes Clemens points that means it what it is so excess keys so one of the best parts about the cloud is is extremely extremely fine grain that you can say this user is allowed to upload only this file on this bucket right a very very fine grain or this user is allowed to read but not write to this queue or this user LaVon to extort instances like ec2 instance they're not to look at them or they're allowed to only shut them down you you can do very very specific things

but of course because we want these to these these sometimes operations teams of dev teams just make super wide commissions oh listen to the company I should be able to do whatever I want so they have permissions that let them do anything they want and then Apophis if you get access to their credentials you have access everything so where the slope is usually trade of us potentials sometimes they're committed to the source control and then reversion so they're saying like oh I shouldn't do that and the way you find the best keys is just by looking through every git commit so this is just lists out everything you can commit hash and then using X arts we're just going to say

Chloe use get grant to find something that looks like a to speak and if you ever find anything that's it's useful to just try out the problem in abyss keys you usually can't find out what they're for you can just find out if it works and the great thing to do is if you've ever accidentally commit something to source control which is you can convert it if you want the thing you have to do is revoke that to me that's the person [Applause] one of the nice things about the cloud is there's a lot of this book that blogs out there because for every single API action is starting up an instance creating a cube policy on a user

everything creates a log so here we have a bunch of logs from a us so like Andrew did stop instance on this instance and then it looks like you started off that same instance later that Eric walked in twice you can see all that guy so it's very useful you put this in your Sam or something this is kind of really obviously a center from a security perspective I just want to put a little bit on your previous slide it over Ajax to grab for yes there's actually two red X's that are posted by AWS reliably finally because if you notice like to start shaking yeah mine your screenshot on the next slide it starts with yes yes

so you wouldn't about that key but great I try to repeat that so cool so I need to be this they have to read Rex's things was used as if I want to find any of crispies also point you can see on my my screen on the top left my it wouldn't find say it's a test okay I figured that I figured it was like a thing that was a hippie was keen once it's all that makes sense and it kind of made sense internally in my head so yes great one was I found this really great blog posts that are stalking about how to disrupt a business laws talk about that because they keep using this

for kind of critical security function you need to know if your attackers stopping you from looking at blogs the most obvious and I just want us to just delete the cloud trail thing this is obviously point to Jerry or logs and so he'd be kind of noisy and loud but this will just stop quite a trail from blogging anything any more logs won't notice what the attacker is doing at this point and the next most slightly quieter than that is just saying stop logging right don't delete the whole trail just stop maybe they're monitoring for deleting the trails not stopping water this is another clever one this is saying okay we'll only log in the region that the clio trail that's

running it and not in every region usually kind of trader wants globally into one place which is very convenient but if you update cloud trails they actually just log in u.s. East one and then if you're using you best bets to you won't get any logs over there which is kind of clever because it's still partially working so if they have any anything inspiring happen if you have access to that so you're going to get a little just store silently failing because well you have anymore if you want to be why if there is Allegra final one destiny lifecycle to delete all the logs that for one second hopefully you'll get you know there get her won't get in there before us

three life cycle rule just starts deleting all the logs after one second after that written and this is kind of the most clever one blog post so you basically say set up the Long's with this kms encryption key and then you just throw away the key and that they don't have a key or writing the logs but nobody can get yeah this is the blog post I found a problem disrupting Daniel goes like very very fun there's even more fun though things in there about chunking logging so like since it's totally the last few seconds about just who team and what I've learned from running a SAS company you really do need to restrict developer access even though

they it's great having empower engineers they can do everything it's not great if they can just ssh into production and start many things and opening up works and everything because you have to ask it's not just you that you are worried about but it's like your new intern that has production access for this new diet you don't really know well it's like would you give everybody an access to your house that you don't even know once you start getting a certain size and your sales team you have to start being like and necessarily trust every single tool best way to do this is just to automate everything the wrong way to do this is to take away developer access and not

let them do anything because then engineers get really mad if you squad lead everything I have no excuse and if you say hey you're not allowed in the production but instead right it's also a lot of rating engineers attribute like that continuous integration which despite those bugs in section 3 them in the incentive security it's very useful and I think it's really critical security function nowadays especially if you do security testing on the software during continuous integration a peer review is really really great at least we review 100 cent of our source code to make sure that there's no doors or anything they trust them when you're choosing new technology to choose slowly and carefully it's really

more choose that a new docker orchestration framework you have to know how do i configured it securely how do I make sure it's not you know opening on that stage board that was no aware of the lack of some things to be considerate about everything just installing stuff immediately and then you have to always ask where happens if this service is compromised so like your linchpin servers that your continuous aeration or your chef server what happens if your continuous integration server is compromised it's probably a master but you hopefully you can lower that to some kind of you know okay well if they compromised this then than this but I have this other process in there

lower that same thing with the chef server to get nice access to the chef server how are you making truth about back touring everything the artifacts or those specific really obvious ones like don't use the root account use rolls the access keys nowadays you can say it's ec2 instance is allowed to do this not key basement keys anymore and your main keys they can't get lost and it was planted and all this stuff have rules nowadays to really do it highly recommend using rolls and speeds obviously used to FA watch the learning logs and segments report so that network access is actually restricted if you're going to have a network of laws it's really nice actually saying if your

networks not everything talk to her and then you give audit your accident management regularly want to alert on pneumatics s keys new provision to users roles and groups obviously if your attackers adding users you want to know about it instances this is kind of problem though because some way some companies work they certainly instances every five minutes and this would be a super terrible way to go so it used to be kind of you might have to tune down that alert to make maybe it's role set up with it or something like that you have to be a little bit more specific suspicious console log ins this is like all sins sort of bread-and-butter thing

you know we've never walked in from Russia before that's worth a structured logging like make that one slide showed you could disrupt logging tons and tons of ways probably the easiest one is just if you're not getting as much long as they are you probably won't learn and there's tons of those times horrible to do one cloud lowest it's actually not a really great set of best practices yet unfortunately not to alert on various providers so you kind of just have to do your own research at this point that's it [Applause]

yeah so at one point you had the jenkins file structure kind of down did you just like download Jenkins and look through like where with starting all of its sensitive data so the session one is did i download banking's and figure out what we're restoring its data I use our Jenkins that we use with my company and I just walked in and I was like weird I know what these passwords are where are they and I just looked and I found they were there in all these places and then I found the Chinese user has readwrite access to everything so and then I found out that if you run tests you're running as a Jenkins and you have

access

[Music]