First Contact with Container Security

hey everyone thank you so much for joining me today in first contact with container security because when it comes to containers security when it comes to containers resistance really is futile i i want to apologize right off the bat like we say we shouldn't apologize but whatever i am so nervous right now because dollars hackers dallas hackers have become my family and presenting in front of family is so nerve-wracking so let's just go with it i've got allergies tech issues all over the place lighting issues so this is gonna be a lot of fun now i just want to make one caveat if you've seen this or if you've seen this talk listed at other conferences before it's

not the same talk i really should have put version numbers in it the first version was myself and malware jake jake williams just kind of shooting a breeze being friends having conversation we both watch first contact we're going like you know what would be a great analogy for this and what came out of it was just kind of this outline the syllabus on what you should learn that's not enough for me i love to teach so what i really wanted to do was develop a series of talks that led you through the journey didn't teach you everything but led you through the journey of things that i believe that you should really focus on so version 2 was given an rdu and this

really focused on the fundamentals of everything you know where you should start whether you are just starting out with containers or whether you're starting out with uh technology itself after that we had version three which is actually this one and you know it's actually a benefit to you because i'm recording out of uh i'm recording out of order just based on when conferences were and so you're gonna get a bit of everything pulled into this talk version 4 primarily focused on the attacks themselves and that was given at tcs texas cyber summit i'm going to do my best to get those recordings to you as soon as i find them i'm listing them at lopunk.com

speaking and i'll actually have a resource page for all of you at lopunk.com dfw now i've gone on and on about three minutes which is usually when you know people leave the room if they're in the wrong one but hopefully you're all still here so let me introduce myself my name is elle marquez and i am the linux security advocate at inazer actually i've been transitioned to just security so there you go and i get lumped in with the devrel crowd a lot which is fine they do amazing work but in order to understand this talk i need you to understand a bit of what i actually do my job is to work really closely with

researchers both at inazer and researchers that we collaborate with researchers that i've met and so hey if you are researching something right now and you'd like to present on it you're just not ready or you need somebody to help just bring it down to earth like reach out i love doing this stuff and what i do is i ask questions you know what are you seeing what got you inspired to research this like what triggered it did you see a new piece of malware a new apt what are you seeing it do or what are you seeing them do what are you seeing within the code like what vulnerabilities are they hitting what are they looking for and i while they're

all digging into you know the binaries and the libraries and the grains of salt inside of it i pull all that together and give them time to write their research papers their white papers because i come to you and i say this is what we're seeing and this is what we suggest that you do about it and some of them aren't really suggestions it's just like this is what you're gonna do about it though i have just a few more things i mean we've gotten started but you know if you can pull up your discord now and i have a few questions about containers for you first is how do you know are you one of these

people that use containers every single day you know are you day in and day out dealing with the security behind these are you actively protecting them put it in the discord now take a look these are the people that you want to make sure that you make contact with you know conferences are all about networking and so you need the opportunity to speak to people who are in the field these are going to be amazing these these people are going to be amazing resources for you i don't mean during the incident itself i mean through your journey yes i i've had that information pointed out to me before i don't want to know what happened

now how many of you feel that you know the basics you are at an amazing point if you truly know the basics then you know where to build out from there you know where to go it can help you know your troubleshooting steps and finally who's new you are at a great spot because hey you've got your advanced people you've got your people who are just a little bit above you with the basics make friends make contacts network network network why repeat mistakes that someone else has made like that's why i'm here and if you know me you know what i'm going to say next it's okay to be new the truth is when it comes to containers

everyone has to be new the technology is just changing way too quickly for us not to be new if you're not new you're stagnant and if you're stagnant you're stuck in the old ways and if that's the case guess what you're the vulnerability in our system and that might seem rude but i've never been one to hold back on the truth to truly understand containers though we need to go back to the original back to the origin i mean when it came to star trek like here's where we got started where we learned to explore and love the world we loved the characters and saw how you know different types of i want to say species and that's not

correct but really learn to interact with each other for a common good for a common goal you know to go where no man has gone before and to understand linux containers and really what they gave us right they gave us a way for developers to be able to work together for our applications to be able to work together even when something went wrong then i have a question for you one more time what's a linux container i really wish it wasn't awkward for me just to sit here for like 30 seconds and be like all right type your answer in but instead i just kind of talk that probably wasn't even enough time but keep typing you know and um

what is a linux container that's a linux container nothing there's no such thing like go and read the source code if you're so inclined go read the man pages read articles about it do whatever it is but i'm not wrong there is no such thing as a linux container because it's like asking you you know what what's a linux web browser or what's a linux i don't can't even think of anything right now the fact is that there isn't one what it is is an application that's loaded into linux to work within the system that is what a container is and it's what we know as containers now has really been built on you know the back

of other things you know we started up with the ess enterprise and we went to the uss enterprise nc blah blah blah right you see the chart and yes i'm gonna be making these analogies oh if you've seen a container talk before you might know where i'm going go and read this white paper it is by far my favorite white paper and yes i am that kind of dork i have a favorite white paper and actually most white papers i hate they're too complicated for me to understand this one this one's on point fact i met bill cheswick who is the person that wrote this and totally fangirled and he told me that he actually can now spot

when i give this talk because you see the spikes in traffic whether that's true or not i don't know but go visit them let them know i'm talking about them and the name of this paper as you can see is an evening with bradford in which a cracker was lured endured and studied you gotta love that back in the day when we had the term cracker and we were still using it and what had occurred at the time and this was around uh 1991 and there was a cracker attempting to really delve in and to break into bell lab systems now just a small note that you might want to know he was using the famous send mail

flaw the debug hole and was trying to get into the internet gateway machine why they were really needing to understand the why and eventually they came with the idea that you know what he's wanting a copy of the password file so chaswick was like well how are we going to find out why let's give him one and they gave him one to the system and for several months they really just kind of led a chase and it was he calls it a merry chase to find out more but obviously he didn't just give him like uh here's the system have fun he looked to create an isolated environment in which the attacker the cracker i'm gonna keep saying that

didn't know that he was in this area didn't know that he really didn't have access to the entire environment he created a contained environment for this attacker now really what was used were long of the variables i should say that became what we now know as charut you know in its infancy and he built gels in order for this attacker to be able to play i'm going to go with that gels intraroot are at the base the first step of what you need to understand to root by the way sounds uh sounds like it means or stands for change root what i want you to do i don't have the time to go into everything what i'm here is for you to

take notes and outline the steps that you need to take in order to make your fundamentals so you can understand and be better at container security so write it down now go look up to root and go look up free bsd gels now you can go a step back if you want but that's a good step to find when you understand these fundamentals you can understand the history of really how containers man they're older than i am to be honest with you and it's just a it's a beautiful story i'm gonna go with that like let's i can fangirl all day over technology the next step that you need to take get your pen out now

namespaces namestypes namespaces are vital to understand i have a talk around them called a containers not your mama's tupperware that's going to be on my on my side as well and by understanding how how namespaces work you can understand how the isolation occurs excuse me i'm trying not to sneeze how the isolation occurs within a container my recommendation and my slides over here which is why i'm looking at it are start with the pid namespace and if you don't want to start there start with the network namespaces these are going to be ones that if you are a linux user it's going to be just very natural to you if you're a windows user yay you get to

learn linux and they're actually really easy to use why because there's amazing documentation on there and you can go in and these are a part of linux a part of the linux kernel so you can start making segmented network namespaces and playing with them without even needing to rely on container technology like if you understand and you know how to use these you know isolated resources if you know how to go in and set up a network if you know how to mount a file system within it like think about it how much more are you going to know about how to troubleshoot issues with a container you're not going to have to rely on just this open

documentation and looking at stack overflow and seeing if anyone's already answered this issue next under namespaces write learn about c groups c groups control groups are a kernel feature that help control and limit the resource usage by processes or a group of processes it's what allows us to really present resources to a container and continue to have them isolated like in a sense you are giving a container access to the kernel now we do our best to isolate it so we can isolate what happens but it's still using that base kernel a lot of that focuses around cgroups and namespaces now how do you learn about it well of course i'll have resources and please

feel free it's all hosted on a get lab repository go and add more go i hate saying this because i feel so rude but go and read the documentation go and read the you know the man pages i hate saying you know go and read the uh friendly manual are we gonna go with that but it's extremely important that you do because you there's been a lot of time dedicated in order to be able to explain these things to you they are amazing amazing documentation have i made my point yet let's go to your homework read and enable with bradford it really is good i promise it's going to teach you a lot go look up at namespaces do we use all

the namespaces have they always been used could we create containers without some of them great questions for you to answer next go and read c groups can you explain to me how they're actually used what they actually do as a part of containers and don't just say you know they help allocate resources what's actually behind it you know these are going to be pretty cool questions for you to be able to bring up an information in an interview if you're just getting started just saying it's also a great basis for you to understand what it is that you're protecting this is where your homework will be available at if for some reason i leave something out or you need some more

resources just ping me most people can tell you that i am really friendly and really approachable so let's go back to what is a linux container okay so you know how much job has become a word and if you don't know that mcjob has become a word wool surprise it is according to marian webster's dictionary i'm not trying to trick you i'm not trying to get you to say you know okay miss l yeah you're calling me msl there's no such thing as the linux container well our verbiage has changed and so when we talk about containers we're really talking about um added space you know one or more processes that have been created within

this isolated environment that's meant to keep them from the rest of the system very similar to bsd gels or the tree root command and their point at this point is to be portable and to be contained so they can work from one system to another without any issues in my humble opinion containers were created to keep the sanity of our developers that's the fundamentals right we got those down now it's time to dig into docker not yet we're getting there i promise but there is more to containers that dockers itself like i said they were built on the back and the technologies of others and containers were built or docker containers were built on top of libsy and linux

containers once again libsy you can go read about it yourself i have to pick and choose on what i talk about so what's a linux container what also refers to lxc linux containers or lxd lexi and this refers to the linux container daemon you need to go play with this like seriously don't limit yourself to docker alone lxc and lxd are very interesting technology you see they're built to work with both containers and virtual machines themselves i'm not going to bore you with that slide that's like this is the difference between a container and a vm here is the hardware and if you don't know that like that's okay ask about it in the discord

channel contact me later lots and lots and lots of blog posts and information written about it but having a system having an application that is by the way it is written that is there is a word that i am missing uh but anyways i hope you understand by the way that it is designed it is you know meant to work you know hand in hand it is meant to be able to collaborate between you know uh virtual machines and containers because at this point it feels like we are doing one or the other i mean hell containers can run in virtual machines that can run on a server i i sound rude uh but i don't know some of these fundamentals

have just left the base of our technology and i'm also getting distracted by the weird lighting but like we said we're gonna go with it my homework for you is to go and get your hands dirty look lexie and alexey whichever you choose to call it their development team man they are on point i have met with some of their trainers and they are dedicated to helping people learn their lab environments will allow you to go and play without having to work on it on your system installation is actually really easy and i encourage you to try it yourself but they have great documentation they have great labs you have absolutely no excuse not to be able

to understand lexi like it's all out there and hey once again if you go into a job interview and you can talk about containers beyond docker and kubernetes like that's gonna make you stand out it hey i know it did for me finally we get to the point of container security issues now other talks really work around hey what are the issues well we have misconfigured uh docker images we have docker images that have malicious code within side of them and malicious tools and we have them running with privileged flags and we have there are a lot of issues but those talks are already out there and if you need help finding them just let me know

what i want to focus on and that's the major container security issue is we're human and we're new and that's okay but it means that we're gonna make mistakes and a lot of people are trying to solve this by going to the cloud relying on their providers relying on their technology guess what the tech the containers on the cloud the cloud itself it's run by humans like i keep sounding just so you know just rude but there's just these small bits that we don't think about we are going to make mistakes obviously the cloud providers the people behind it are going to make mistakes we can't have this inherent trust and later i'll give you some straight up examples on how

it's ultimately true and lastly or not lastly but the next point i want to talk about is the tools that we use in security when it comes to containers or just the whole holistic view of everything i uh here's an interesting statistic did you know that 70 of vulnerabilities are introduced or found inside of the application layer i talked to so many security teams that man they think they are on point they have the best in breed technology their perimeter man those are strong walls okay cool well you keep building that while the assassin's already in the castle taking care of things for you like we can not focus on perimeter-based technology alone i also have been working a lot with

developers lately and going to developer conferences and having conversations with them because that's a huge statistic we need developers to be a part of our team we don't need to have this adversarial just kind of relationship so what i hear from the developers is security doesn't understand the application layer like we rely on tools just to do our scanning for us and we come up with these reports and they tell me that all you do is drop it on their desk and say hey do the needful i hate those words just take them out of your vocabulary right now developers will tune you out they'll tell me like this report it was like you know three days ago we've changed since

then we've updated it these reports are pointless or they rely on things that you know the security of individuals don't really understand why these configurations are the way that they are and the fact is we don't have to we should really make an attempt but man we're buried under a mountain of alerts having to stop all these attacks having to do these configurations we can't know everything and we can't do everything on our own that's why we need these relationships with developers and operations like i always say screw devsec ops those are three teams attempting to work together what you need to do is you need to focus on being one team with one set of tools

and ultimately we have one goal that's to keep our jobs and continue to get paid by the way my apologies for the weird edit the lighting was just getting to me the dogs were barking i had to take a moment of zen to be able to you know really work with you all issues we don't face when we're in person all right back to the topic of working with developers one thing that we're seeing here at inazer is and honestly i've talked to other companies as well it's something that is being seen you know across a lot of companies is attackers focusing on misconfigured docker api ports there is a lot of times in which

configurations occur on channels that are not encrypted or really allowing communication from anywhere to everything without being authenticated and you may be thinking like nah not mine i take care of it i make sure that it is protected on the network basis i've been looking for some of these first of all if you've been looking on the server side wow you're a unicorn but it's happening so much so that like it's automated we don't even need to worry about the or the attacker doesn't need to worry about even doing it right he's written a bot it's getting done it's hitting companies every 90 minutes and that might not seem like a lot until you start thinking about how many

companies there are and the fact that they're facing it every 90 minutes i love to talk about this and if you've heard about it before i'm sorry but the way that i imagine it is an attacker you know just kind of sitting around eating at like mcdonald's panera wherever they're eating and their pager goes off and well their pager duty goes off and they kind of look at their phone and they're like huh this is kind of interesting like uh we got one and so then they just push the button or list the command or add the command on slack and then the automated attack happens like and they their job is so easy at this

point i feel that as protectors our job is more complicated like we have to be able to protect everything every single time and they're just pushing a few buttons and they have it done when it comes to containers one of the pieces of malware that i love to talk about and okay real quick caveat understand that when i like just go into this i am not saying that the attack is okay i'm not saying that the results are okay but on the research side this is fascinating material so the excitement revolves around that now let's talk about king xing i really hope i'm pronouncing that right king sing works by really targeting that unprotect unprotected docker api port

and it's running in ubuntu container now are you going to be seeing you know an ubuntu container as really strange activity within a container configuration probably not and then uh when it oh by the way when it's creating that ubuntu container it's um including a shell script called dsh so there's an indicator that you might want to be looking for now obviously it's easy to change and by doing so they changed the signature so here are some bypasses to look for kingzing keynesian is your friend it wants to help it goes in and it goes in and you well this clears your log so you know no pesky noise for you then it's going to go through and it's going to

disable any other malware than it sees for your system it's going to harden it it's your friend it's looking for crypto miner samples it wants to do the job that should have been done already all of this is done before it even loads its payload now i'm not going to dissect this uh chart for you but i will have it available with my slides at lopunk.com dfw so you can delve into it uh feel free to reach out for questions or with questions i'd be happy to you know contact our researchers to dig into anything that i might not know about okay i only okay my company i get excited my company is completely crazy enough to let me play

with malware like if you know me that might be a legal issue let me show you uh a few samples that we have of uh kaji i'm sorry that we have of um king zing now all of these okay so i'm gonna look over here because that's where my screen is if you're looking at the section which is the fourth the sha-256 these are different variants of uh like you probably want to say kaji of kingsing that we're seeing i want to draw your attention to the one right next to it virustotal now the first one the very bottom from may 23rd that i have access to seven out of the 60 anti-virus programs there were able to label it as malicious

like 7 out of 60. that should shock you this is a really big issue like traditional anti-virus it sucks it is not transitioning well to the cloud it is not transitioning well to you know shifting left and this continuous cidc pipeline and it's definitely not going well when it comes to containers now it goes from the bottom up so we see new variants coming up and oftentimes it's new abilities that have been added into it it's a great way to be able to bypass behavioral based detection so let's go up to hey the last one there september 10. we got 30. believe it or not that's really good to have around half of them under it though

8 out of 58 6 out of 60 7 out of 57 like excuse my language but what the hell like why is it this bad how many of us really depend on virus total for our information and not picking a virus total i mean it's only as good as the antivirus programs on it so just saying i'm going to be focusing around two samples right now the first is that may 23rd the one that i had access to 7 out of 60. and the next is the one from august 24th 7 out of 57 so that's gonna be a sample one at the bottom the august 23rd sample two being uh august 25th okay breathe out because i

i'm not cool with this so obviously i'm recording this before uh the date of the conference i went ahead and was like hmm i wonder what it's like today i mean obviously these are over a year old we must have better detection rates like it's out there there are a blog post and you know indicators of compromise already posted but well sample one's gotten better i mean 25 out of 62 after a year i'm also shaking my camera um i get really bothered by this and you should too if you're so busy working on your network and your uh host based detection is an anti-virus program or even what's going through your network like look i don't tend to tell people that

like you know negative news but if this is all you got you're screwed they should prove it i it's not cool so let's take a look at that second last second piece of that second variant which had seven base detection ran the search again we're still at seven percent like not okay folks not okay and just to be clear that's the sample because i know that they both had seven percent so i dug in sample one what is it that we're seeing please understand that this is not a sales pitch if you know me you know i would never do that but i'm also just at the infancy of learning you know my dfir and my forensics and so i can't pull up

hydra for you or gedra however you pronounce it and start walking through the code i'm at the point where if i push a button and it gives me some answers that's what i'm gonna use so what we saw in this variant is code that was you know down to the binaries and the libraries was labeled as the kings and piece as kingston as that type of malware and we see here that there's already like an exploit that was brought into it an exploit that was already known so this should have triggered something at least that said hey this is suspicious even if it didn't label it as malware labeled it as something that you want might want to look at but it was

only about 17 of the code and sometimes this isn't enough to make traditional uh detection systems traditional detection or traditional lord i'm missing the words uh traditional detection abilities let's just go with that y'all know what i mean to actually trigger traditional detection products that's where we're going so that's kind of where that layer defense you know defense and death's right overly used but it really needs to be in place with more than the tools that we're already using what am i saying humans need to be involved it can't just be all automated sample two when i talked about code reuse and the variance and the way that they're changing what they're doing in order to be able to bypass detection

it's not just that signature base we have all this ai right that's supposed to focus on behavior and be able to catch it attackers they know security they might even work on it so they know the tools that we have they know what we're looking for and so they change their attacks to fit around it i mean look at this we have xm rig miner which is actually pretty new and it was discovered and had really low it wasn't even like zero percent detection when it was discovered um and we've got our coin miner that was going in there we're taking functionalities from different types of malware that what we need and that malicious library factor

is really coming in we're seeing a huge trend of attackers using open source software that's already out there like mirai right it got like open source in 2016 it's been out there some of us have probably already played with it like not admitting to it but you know hey actually i am i have stable environments and permission to play with this malware we're seeing it in so many different types of malware and variants that we're seeing like we should be catching this but it continues to be effective all of the time like am i pushing this through so you all see what a huge issue this really is and you know what kingzing is not alone like this isn't an abnormality

some of the other malware that i like to talk about and understand i can only fit so much in this talk when it comes to it but kaji uh watch the video for texas cyber summit when you have an opportunity you'll learn a little bit more about kaji now i ran this search um well i ran our team ran this search in august 26 2020 and hold on that was the other one you know what just ignore what i said my apologies i get over excited anyways when we found kaji and the dates will be in the other talk it had zero base detection like enkaji was quite an interesting piece of malware because when it was first found

it was host-based it really was attacking you know the primary the fundamentals themselves but then they maybe i don't know maybe they heard about how easy it was to find these open docker api ports and they transitioned the attack to target those open docker api ports and to launch within containers like why weren't we catching that second variant we already knew what it did it was just that now it's in containers instead of in containers instead of the host we've got docky now i have talked docky about docu previously so i'll keep it short doki was able to bypass all pre-run time scans because there wasn't anything malicious in the image if you don't know

this over 50 of the images found within the docker hub have been uh published or have been talked about that they are either have malicious code with inside of them or even attack tools already there yet docker hub continues to be this trusted source i'm sure they're doing the best that they can but we can't overlook that fact so docu though no maliciousness it ran through the pre-run time scanning dead fine loaded up into a container did its thing you know what its thing was it pulled down the malicious payload and called out to the c2 server the commanding controlled server and it's like hey i'm here what do i do and then it was uh

kind of then you know it actually did do container breakout but all of this happened with a matter of seconds can your detection look into that container that matter of seconds that it happens and actually trigger upon it it can cool can it look for a signature that isn't in your database that's exactly what you're seeing here so i asked this question before am i saying that we're screwed kinda i'm saying that if you don't change what you're doing if you don't change your mindset if we don't change beyond this traditional you know network based just building walls around our castles and relying on automated tools to do the server side of it never working with our

developers like yeah you're screwed but if you open your mindset if you look at security in that holistic view and not just focus on container security is secure is container security it's all about the containers i don't know i've said it enough what you need to understand is we're really building on a foundation of sand now if we don't understand the history and the technology that led up to this if you could do all of the container security you could get that container as secure as possible and if i've already pwned your system like what does it matter i don't need that container i'm already there the assassin me in this case is already in the castle

we need guards to get off the tower and go look for it i speak in analogies go with me lesson number i don't even know at this point but i'm going to go with one right now server security is container security yep that docker 8 open that docker port open api that docker open api port go and look for it now i'm already telling you that we're seeing it so many times there's absolutely no reason why you shouldn't leave here today and go have a conversation with your developers on what's going on or maybe look for it first but yeah whatever do it whatever way that you need it to happen but both developers and yourselves and even ops

need to know how to look for this why is all of this occurring like why is the ground continuously just shifting on us that's because we have the changing landscape we're moving from on-prem to the cloud did you know most companies or majority of companies are using up to five cloud providers and security teams are supposed to be able to protect and know all of these no we can't and so when we outsource to the cloud we have this old shared responsibility model which i think is a joke and we just place our trust on something else like we just hope for the best and a lot of that comes with shifting left by the way what do i add him on here

because a lot of people really look at him as from voyager his first appearance was actually in first contact a little nerding out for you there when we're shifting left and like i hate the premise that security is shifting left because it's not happening if it's happening at all in a company as quickly as the development process is the more that we shift left the more that security is put on a developer's plate without the developer even getting the training the knowledge of what they're supposed to know and if they do get the training it's what like a day-long program in which they give them bullet points that they're expected to go read up more on even though they

still have like their deadlines that they're supposed to meet that's why the whole saying of fix it in production it's well known by everyone there are a lot of fundamental flaws really built into a our ci cd pipeline of development not just the code itself devs are a critical point in our security process we can't continue to sit in the basement in our dark little rooms and think that we can really protect our companies a way that i really had this shown to me and i've been doing a lot of outreach i think i already said to the development community is i went up and i spun this aws server up no okay i did my updates like that's

what you're supposed to do in order to patch your systems right dnf update gra app get update no more get apt update and i still had a vulnerability in it and it was a pretty high vulnerability why did that occur because patches aren't automatically put into the system there's a release cycle around them if it was automatically done and even with the dnf update we have like upgrade policies on it we have you know maintenance schedules the reason this occurs is because we can break applications really easy applications that are key to our production when we shift left we come up with this new you know new server new image every single time that's released

but once again that takes security out of security hand security's hands and places it into that of the developers and places it on ops to make sure that they're staying on top of this also and maybe i'm putting this out of order but it was really interesting to me that as i'm talking to a developer's team they're like look security has no freaking clue what they're doing they're just dropping these reports on it and you know what they did they actually automated their way around the security checks because they told me that that first security check was taking over half an hour you know how far that takes the deployment process back and keeps them from effectively

doing their job we have like oh i'm gonna play around because my code's compiling i'm to play around because the security checks are happening so they had a few deployments going through there and everything else was just bypassing these into production because they were just updating the servers themselves instead of deploying new servers nothing against them like i think after a certain amount of frustration we want to get mad at them but we need to understand so like i said what are we doing we're relying on cloud provider container technology you know we're using azure functions we're using lambda and this is seen as like this big change right a way for us to really be secure

did i mention earlier that cloud providers are human and that mistakes happen well in azure's research team like i said i worked closely with them were recently able to and i'll say it even though i keep getting in trouble for it we hacked azure functions we were able to create a compromise which they called royal flush now i haven't included every single part of the attack but i'll link to the blog post which you can go in and read and if they haven't fixed it yet you know i'm not telling you to do it do not go duplicate it even though you could if it hasn't been fixed don't do it do not become a criminal

what happened is we did this sophisticated attack i mean like our researchers are on point they went in and they ran this insanely hard tool they ran in map yes i'm insanely facetious because nmap is something that i learned on day two of training to be an ops person like if you're new go look it up you really need to understand how easy this is and yeah um let me go to the second slide we were able to get root it shouldn't be that easy and i'm not picking on azure i'm not picking on azure functions that's what we compromised my bad uh we weren't like it happens everywhere um in tcs i go into

how another group and i god i forget their name right now but i'll give credit there we're actually able to do the same within lambda we can't rely on cloud security providers that's not enough we need to understand fundamentally what's happening on these systems your homework go look at these blogs read them if you're advanced and hey maybe even if you're not go look at the code understand how simple these attacks were really done you need to understand you know attackers uh the way attackers aren't working aren't always these high you know just complicated attacks sometimes it's just the simplest thing that helps them get in we're almost at the end so what do we do what do we do about all

this like we need visibility plainly we need to know what's going on on the host we can't just keep outsourcing it to our developers we can't keep outsourcing it to the cloud providers and hoping that ops is doing exactly what they need to nothing again stops that was me but you know what i was overwhelmed as well we need a way to clarify the picture do i like the way that i did that so i have a few pieces of advice for you on how to do this and by a few pieces of advice i mean go and do these things they're not that complicated and you have to if you want container security and you want a strong container

security posture this advice isn't coming just for me it's coming from the researchers that i work with as well first trust no one i think i've made that point and it comes around that trust but verify idea it doesn't even have to be maliciousness it doesn't have to be that insider threat we're human we make mistakes we can't assume that everything is perfect from day one you may be saying well i don't cool then why if you go and you google data breaches do we still see so many there must have been some inherent trust into what was going on there just saying scan your images for known vulnerabilities with everything that i've been saying um some people have

taken it to say to me saying like we don't need to do the original steps you're gonna catch some things and that's good i mean even seven percent detection meant that seven percent of them were able to find it so let's get that out of the way please understand i'm not saying not to focus on pre-run time and your network these are important tools that we need to keep within our toolbox they just can't be the only thing that we rely on pat your host pat your host pat your host i don't know how many talks i need to save that in but you need to patch your host and a lot of times i'm sure that people think like l

i came to this talk for new and exciting ideas and things that i could do and cool i'll start giving those to you the moment that you start getting the basics down the moment that these happen i mean go and look for data breach or you know compromise because of a unpatched system right like go do it now but yeah i pay attention to the talk we're not patching i recently went to a talk uh on um industrial control systems where we flat out we're told these cannot be patched like it can't it will break things and we will lose critical services i'm going like why don't we get new ones the answer answer's simple it can cost

billion millions to billions of dollars to replace this infrastructure so what do we do we isolate it we create you know new vlans we try to air gap it as best as we can but we still need to work within it and attackers they know what's going on hell half of these systems still are vulnerable with eternal blue like what whatever i'll get off that high horse now it just terrifies me remember that initial cloud init issue why aren't we looking into this why isn't this being caught why is this still occurring like let's get a plan for patching beyond what we already do the idea ideas that people say is yo i got patching down right patch tuesday

i'm on it every friday we're doing this we're bringing in patches as soon as they come out first of all i really doubt that you are or you're gonna have a lot of down time because oftentimes it's going to interfere with what your application is doing but let's say that you are you're that rare unicorn you realize that patch still has to come out right and oftentimes that patch has to be configured to fit within the systems that we have the moment that attack comes out that moment that vulnerability comes out attackers are on it look we have attacker farms that we're seeing in other countries and if you want to know more about that go to my apt talk

because i'm not going to dig into that right now and start pointing fingers but we see them come out at 9am start their workday attack start around noon we see them take a lunch of course everybody has to take a lunch and at five they clock out and we can just start being a little more calm yeah it's their jobs to do this and let me tell you they're not under a mountain of alerts the way that we are your homework go and patch your host seriously just go come up with a plan on how you can do this better and if you can't patch your host then look you need to understand what your tools can actually do are your

tools monitoring the code that's going on on your system are the tools giving you the relevant information that you need to pass on to the other teams to deal with these issues or are you relying on an msp to give you your information are you relying on your analysis of what needs to occur or are you working with other teams to see what practically needs to happen say no to default passwords once again i will quit giving this advice when we're actually doing it and though we may know the default passwords when it comes to our tools our network-based products do we know them when it comes to all of the applications that are being used

it's not always password it's not always admin we can't know everything that's why working with these other teams is so critical you get the mindset of more than just one person you get the mindset of more than one team all of the teams become one team except that breaches happen this is paramount in your security breaches happen it's not if you're going to get breached it's one when you're going to get breached breaches currently take 288 days according to ibm's 2020 study to be detected 288 days let's just assume the attacker is already in the castle and go hunting we all love to hunt why wait for an actual incident in which everything is going crazy we're going down to practice

our hunting skills how many of you already know if you were an attacker how you would get in have you looked to see if somebody's taking advantage of this probably not and finally know what's running on your system at the core of every single attack lays one thing malicious or unauthorized code understand what should be running on your system and what's actually there look to see if this code runs you know parallel or has similarities with known malicious code with known malicious libraries go beyond relying on behavioral detection go beyond relying on you know your abnormal that was behavior i'm sorry signature-based detection know your host have visibility and with that you know what i am coming

close to end yeah like i'm learning how to do the whole binary thing but thank you very much if you have any questions i will be hanging out in discord and hopefully some of those subject matter experts will be there as well until next time you guys have a great conference