First Contact with Container Security

anyway now i'm going to do the intro for l here so el marquez uh really or ella punk uh that's she says that's so yeah uh well it's it's it's tradition we have to read the bio so l is a former sys admin cloud builder podcaster and container advocate and has always been a security enthusiast this enthusiasm and driven driven curiosity have helped her as she has become an active member of the infosec community leading her to explore the groundbreaking and exciting world of genetic sulfur mapping at enters into okay i'm sorry i said that wrong anyway l it's all yours [Applause] i'm i'm gonna do my best to keep it short sweet and simple um but i tend to

talk a lot if you've met me you know that's true so we're just gonna have a lot of explanation and then the last like 50 slides will be two seconds each go with it all right so to give you guys some answer on uh why i said it relates to my uh talk here the original name for kubernetes given by google was borg so and the name of the little blue docker well is moby come on you could have done some guessing there but anyways okay welcome to uh first contact with container security sorry i have a magnetic bracelet on and it's killing me um as you might be able to tell just by the

picture here and my persona i am not your traditional speaker so uh you know what just go with it you're gonna hear a story this talk is not gonna go where you think it's going i've got twist i've got turns and every time i give it it's different because i go on different time rates that being said let me introduce myself with the most important i guess explanation of who i am and that's an advocate for operation safe escape this is a 501c3 dedicated to helping victims of domestic violence in order to be able to escape their attackers when they're being targeted by tech if you would ever be willing to use your skills to help save lives please let me

know and we'll get you in contact we'll get you right to work and for those of you that might be in the situation or know someone who is please provide my information provide operation safe escape to them it is so important that they know that you know they're not alone in this okay who am i my name is el marquez and i am a warp core engineer you know in star trek when everything starts going wrong you know warp core is breaking everybody's leaving everything's been breached that's when i go in with innocent protect and i save the day all right who am i i am one of the biggest dorks that you will ever meet i am a

nerd i am not a trekkie because i can't learn enough about the star trek world to have them actually let me define myself as a trekkie but my job is a linux security advocate does that mean it means that i work hand in hand with our researchers i mean we're talking i ping them every day i ask them a hundred questions in order to take like this insane amount of research that they've done breaking down you know everything into binaries breaking down those binaries into assembly writing it into this huge you know hundred page paper and i read it i ask them questions and i bring it to you and say cool have fun reading that but in the meantime

this is what you need to do to keep yourself and brain breached there you go long long explanation okay so containers yay i've got a few questions for you before i get started because this is really going to define how i go with this talk my first is who in here uses them every single day like this is just part of your job y'all look at the people holding up your hands these are your subject matter experts if you want to go in depth yo i will go stand next to you while you ask questions and learn that's not what i do that's not who i am but as i'm learning that's why i'm here

so who here knows the basics hey we're getting there all right and who says you know what i'm still pretty new to the container world yay you know what i'm there with you and that's why i created this whole campaign called it's okay to be new this whole concept that even those of you who said that you're experts in it you use it every day guess what you need to be new too because the container world is shifting whatever it started out with six months every three months by the time we get exploits out every three days if you're not learning and you're not growing you become stagnant and you are going to fail just flat out fall on your

face when it comes to container security all right let's jump into containers when we start looking at container security we start focusing on kubernetes that's where everybody wants to go okay maybe you'll go open shift fine i'll start slow and do docker you're doing it wrong that is the first place that you have failed when it comes to setting up your container security because you're starting up so high up here that everything below you has completely gone away and the whole concept of what's a linux container and all of you are probably going oh god she's going to show me that picture of the hypervisor and how it differs from a container you've already seen it this is a linux

container no such thing there is no such thing as a linux current linux container go but there's a kernel but not a container go and read the source code if you absolutely have to read the documentation never is container mentioned in it and that's because what we now know is a linux container whether it be lxd or lxd however you choose to go with that or you know docker all comes from a basis of another origin my tie into star trek and it's you know we didn't end up with you know the star fleet we didn't end up with the original ussr enterprise it all built on top of each other and to me where containers actually started and

where you need to start your journey is with a talk a paper i'm sorry called an evening an evening with bedford in which a cracker is lured endured and studied back in the good old day when they were hackers and they were crackers um and so this paper was written by bill cheswick and side note because i have a lot of those i got to meet bill cheswick and i completely fangirled over him to the point where we were all embarrassed and i told him i bring his paper in every single time i do a talk on this and he goes oh that explains my network traffic you know spikes so make sure you visit it so he

knows that i mentioned him again but in january 1991 he was a researcher at bell labs and somebody just kept breaking into their environment and they couldn't figure out why like he wasn't taking anything they couldn't figure out what he was doing so he decided to set up a little trap lyric would eventually become what we know as a honeypot and guy breaks in he starts looking around and then suddenly he's like okay i need a way to really monitor this without him being able to do some damage so he starts shifting around the parameters he starts containerizing the environment i'm just going to lay that out for you guys which eventually became what we know as

bsdale bsd gels and charut if you're starting out with containers spend a good amount of time building out your rooted environments this is going to help you truly understand this then ask your friend to go break your true rooted environment and figure out where you went wrong basis of what you need to know to pin testing containers from there we move on up oh wait hold on i was going to say so my definition of uh of bsd gels is kind of really what sets the foundation for me and right when you do it you're basically creating several independent mini systems that all share the same kernel with very little overhead is that not usually what we

define containers and their benefit as then we pull in namespaces all right the original crew what really led us into becoming the next generation and now i'm not going to power point you to death i literally just put this up here because everybody always asks me for my slides and i'm like why do you want a bunch of pictures of star trek but so here is some good general knowledge for you to be able to read if you want to know where to start start in the pit process generally that is what we're talking about in just modern day containerization because everything else is pretty much done for us we just take it for granted

we also have c groups c groups are due with resource allocation i this whole first half and i think i'm almost already like 10 minutes into what i've been talking about could be a talk within its own and if you have time write it i would love to attend it i think it'll help the community in a great amount of ways but i'm going to leave that here with just giving you homework my job is not to tell you everything that there is to do with containers my job is to give you the breadcrumbs so that you can go learn and then learn how to how to actually make them secure don't worry about taking pictures if you

go to my website i'll have your slides homework i have links to everything i talked about this is a get lab project if you have information that i don't have on there that you think would be beneficial to people add it to it right all of this is open source you want to steal my slides and use it for something knock yourself out unless it's embarrassing then you just take my like name off of it all right let's get into container security my biggest issue and where i am just going to pivot completely from where you think i'm going is we suck when it comes to security we're building all of our security practices on this foundation of sand we're so

worried about did we set up our container firewalls correctly did we you know make sure that we containers this application in a certain way so that it doesn't have ability to you know cross-communicate with this within our docker within our kubernetes setup and oh we're going to bring in shift so we need and like i'm confusing myself just giving you this example and at that time we have port 2020 of 2012 yes 2020. we have port 22 open to the world you know our ssh keys are sitting on a github uh you know platform somewhere you can find all your um i your rsa keys on lord i am losing all of my words today

over on stack overflow that's where i was going with that you need to build from the bottom up we need to make sure that you really focus on securing that server in your old traditional methods whether that be you're still on prem or whether you're on the cloud and we get to the cloud that's where this all goes wrong i've gotten to the point where i start saying that when we shift to the cloud what we're doing is we're actually just shifting away from our operations team you know it's operation teams as a service we don't need ops anymore the cloud handles that all for us and as we do that we can shift left this is a

great time for me to speak because we had all this information ahead of time but the whole concept that when you said you know we need our developers to actually focus on development is unheard of if you're in a culture that does that you're a unicorn because the last statistics that i read was just earlier in 2021 is that we have 10 000 developers per one security person so that means we hire more security people right no that means that we give the security job over to the developers and we trust that as they're going through the process that they're going to be adding the security controls in place and okay fine security writes them

for them but we're going to trust that they're using them properly i've spoken to far too many companies who security wrote the you know the framework which they're going to be running it through in their pre-run time nobody actually ever looks at the alerts nobody looks at the results but they ran it through them so we're fine so now on top of that our developers are writing the code putting it in taking in new technology they're spurring up containers and we're trusting them that you know everything is going to be fine because we have this process in mind they're running on the cloud the cloud is secure our cloud provider is going to take care of it of course there's not

going to be any issues like here we have the security group you are all in charge of security in some form or fashion for your companies how many of you can tell me that you know exactly what code is running within your environment i'd honestly be surprised if anybody ever wrote i can't tell you how many containers are running in our environment that's because they're going up and down every few seconds if you don't know the code that's running in your environment you don't know how many containers there are what are the chances that you know the code that's in that container well let's start talking about what that code is pooling so cves wow that is

completely blurry you can see it on my actual website but cves are coming out at an average of 50 a day 18 345 cves were released in 2020. 400 of them being called the worst of the worst that's ever been seen i know that i try my hardest to actually keep up with cvs but i don't i try but i can't now i'm going to match those cves to what code is actually being deployed on the server and i need to know to make sure what's being called upon in that container and now i'm going to trust my devs to do the same all right well i'm just going to patch our system well i just broke

the application that keeps our company going i need to be able to have the conversation when it comes to our developers and going like hey how fast are you deploying what's i patched this is the environment going to go down okay i can't patch it right away so can we take that function offline can we take that deployment cycle can we stop those containers from automatically happening oh wait we're automated so now we have to figure out where that's going i mean like i know i'm talking in circles but i'm hoping you all start seeing where this foundation is going it's a foundation of quicksand that we're at so unless we start addressing these issues

it doesn't matter how you're configuring your container security policies another thing when we've shifted left is misconfigurations um so tls wow i really wish i'd made these bigger sorry so tls being configured on docker seems like a pretty no-brainer right that's what you would do when we first started founding and i'll talk about it later another version of malware called docky we went out and looked within one second we were able to find over 2000 instances of docker api ports being open to the world the more and more malware that we see this the more that it seemed pretty common why is it happening because we'll fix it in production we have a deadline that something needs

to be pushed by our developers are governed by that deadline so what's the easiest way to get code to run the way it's supposed to is just to leave it all open right we don't have to worry about any issues if we could find that many in like a second what do you think that attackers that can actively look for this are doing fine i'll get off my high horse and i'll go over to containers this is what i imagine containers right they're the holodeck they are a land of impossibilities made real that's really what companies are viewing containers at manager goes and reads a magazine he's like oh containers will fix everything let's do it

and one thing that as security vendors we know individuals we know is if there's a way to get in properly there's gonna get a way to get in the way you're not supposed to i have to make my star trek reference here it's like when the borg were able to get into the holodeck and go in and attack captain picard because the safety measures had been taken off the holodeck and they could actually get hurt nobody seems impressed by the fact that i knew that but that's where we stand within containers when containers get breached we're all like what happened there's no way i mean literally go google cloud breach or ping or thing or whatever you do duckduckgo

cloudbreach and you'll be able to find that this is an every day occurrence and this is kind of where i get the most frustrated is when i started out as a linux admin there was a linux is secure by default mine point and hopefully every one of you just cringed but now we've gone to the cloud is secure by default containers are secured by default therefore containers in the cloud i mean you can't get any more secure than that and that makes us let our guard down and like because you're in this room you know that container security is so important and you know that you want to research it and you were probably hoping that i was

going to give you a point by point and exactly how to contain your environments and i hate to tell you but the way that you do it all starts with it depends it depends what the culture in your company is like and that is not on the people front but on the who owns what security do you own network security is that basically what you're doing you're monitoring the perimeter you're acting in response time every time you get hit are you working as opps are you responsible for the network security and the server side themselves are you responsible so the security of the cloud where do the devs fit in it's an extremely taxing situation which i wish i could give you

the answer to but that's where your whole devsec ops relationship comes in the first step to container security once you've actually worked on your host is figuring out the lines of the delegation what are you actually in charge of what are your devs and your ops in charge of and that's the only way that you can actually build out that attack plan so the malware that i was talking about called docky now docky is a fascinating piece of malware when it comes to container security because what do we rely on when it comes to containers we rely on that pre-run time scanning right we need to make sure it's coming from the docker hub we need to make sure that

there's a malicious code in it we need to here is our checklist the security that we want the devs to do before it deploys donkey came from the docker hub came from a tested source completely clean absolutely nothing maligious malicious in the container image itself until you run it then it's able to use the curl command to pull down its malicious payload and it's gone in seconds okay well you know we should probably look looking at runtime and seeing what's going on except for the whole part that i said it ran in seconds and because of that donkey was able to run for at least seven months in actual attacks calling back to its ct

c2 servers for over seven months with zero percent detection when it came to virus total there are many variants i don't have to tell anyone that malware changes within variance and my company is crazy enough to allow me to play with malware if you sat with me at dinner last night you realize why that's insane if you're talking to me now you probably realize why that's insane but these are the samples that i had access to and i really wish that you could see them better i will fix my slides but on the side it gives you the portions of when it was able to detect it so uh like the very top one 24 out of 58 of

the virus total search engines were able to detect it over a year that have been released that's kind of telling us that we're failing somewhere when it comes to our modern or our traditional um security endpoints right we're not being able to lift and shift to the cloud the way that we're supposed to and even when it comes to on-prem these anti-virus programs these cert engines were really created for windows environments they're not as effective in linux so now we have linux cloud servers running containers and we're expecting them to pick things up just as quickly you might say l because you guys talk a lot of you can hear as your quietness say l give me a minute you know what it

was a new piece of malware not everything catches up we have to give some time for everything to shift well that's great we're doing that while our environments are being compromised and it wasn't a completely new piece of malware because inside of it it had code from the abc backdoor code there was literally malicious code that we already knew about being reused in this uh within this image and what it did is it gave it the ability to actually execute commands like that was part of the malicious payload yet when that was run inside of your container we didn't pick it up and then the commands were executed and it broke out of your container

and we didn't pick it up side note because i like to go on tangents abc backdoor when it was discovered zero percent detection rate and i was talking to some windows guys last night so fine you had his beat when it came to the windows variant none of them picked us up so what is my whole tangent on this section about can i say this i don't want to say it screw your antivirus programs there was a time and there was a place where you could rely on them and that's out the window am i saying i want to make a windows joke there um am i saying you know don't use them ever no but i'm

saying you can't have blind faith in them that can't be what your security posture is around specifically when you have so many layers going up from the server up to the container out through how many environments i promise i'm getting to the point here well they're all the point but i'm wrapping up um so the cloud providers are secure by default i actually loved what was brought up in the keynote because i didn't know that we'd actually already had that azure breakout as of 2016 going up to 2018 and guess what earlier this year the original version of this was called we hacked azure functions in which we were able to go in and gain root within an

azure function container breaking out of docker must have been so complicated right these are advanced techniques so much so that i can tell you what happened so we went in and ran in map and then we were able to find open ports all right then pretty complicated and then we found one for nginx and we're very excited except somehow that one actually didn't have like things that could be compromised on it so what do we do we do recon we went to github and we found an image from a microsoft employee that kind of showed us how things work and we were able to change that image to edit the pseudoers file tada we're root

that's it if you don't believe me there it is i have a link to the actual blog post you can actually go in and read more on the technical analysis and how the code actually works what they were able to do um so like i said originally it was written as we hacked azure functions stuff happens i'm sure you all know how that works when you're disclosing things so the name of the article is different but just click on the link and you can trust me just click on the link so that's your homework read it seriously like i am not trying to promote innis or whatsoever it's on there but like our researchers do some amazing work and

this one was done by a guy named paul and his mind just just blows me out of the water because he took something that we see so difficult and just broke it down into pieces the other talk that i give is about advanced persistent threats transitioning to the cloud so much so that in 2016 the government was like hey you guys might need to be careful like apts are targeting managed service providers 2018 they were like okay they've breached it they're in be careful 2011 or 2011 2021 september earlier this month they came out and they're like okay so um they've been breached you need to accept this these are some of the steps that

you can take to take care of yourself so you've shifted to the cloud you're figuring out your cloud security now you've gotten rid of ops because who knows them and now you know what just to make things easier we're going to use the cloud providers containers at what point do you as the security team have any control over this how much do you get to talk to microsoft or azure or gpg google cloud about what their security posture is and how you can integrate with it the land of cloud security is getting borderline impossible so my solution to you is that you as the security teams need visibility right it's not unless you can actually see

what's occurring that you can do anything about it i've given you more problems than solutions so my solution plan is going to be pretty simple for you trust no one don't trust your devs don't trust your images don't trust your neighbor don't trust your cloud provider the whole trust but verify you need to be in the middle of everything you need to go beyond hey i'm correcting i'm correcting protecting our network perimeter you have to be the the busybody that's actually trying to figure out what's going on in your environments with that yes please scan images for known vulnerabilities i'm not saying you shouldn't do that just because one managed to get by and it was way more

than one doesn't mean that you shouldn't do your pre-time spread scanning like yes still do the things that you're doing that work for some be willing to expand your imagination on what attackers can do next bet your host pat your host pat your host some of you are rolling your eyes going like duh we knew this you thought you were going to give us new information cool then why can i go on google right now and find 15 instances in which in this week you were able to be breached because the host wasn't patched and i'll hear the whole monolithic argument of well this application only runs on this version and our company can't survive with this application so

we can't patch it so what you're telling me is that the vulnerability in your company is the only reason you have a job like there's a big issue in this so you know how we solve it we're going to lift and shift that monolith put it in a container put it in the cloud and we're secure by default at this point i hope you know that's laughable but that's about 90 of our container strategies at this point so yeah there's that picture that i showed you again earlier that you can't see the one reason i brought this back up is because we can't rely on dnf update yum update whatever it is to do our patching for us

i literally spun up a server did the patch on it and ran our scanner again and there's still critical vulnerabilities on it it takes a while for a patch to happen and then the patch has to be actually brought in by your linux distribution which then if you're using the cloud has to be brought in by them trust no one so there's your homework don't forget it excellent is harden your host no that is not the same as pat your host and it's also continuously flowing as your developers are bringing in new code as they're making changes are they taking care to make sure that everything is updated on the host side it's like i said you know you're i went

completely different with what you expected this talk to be but your first contact with container security is your host like that is the end all and everything of all of it and second of all is know what's running on your server if there's one thing that every single cyber attack shares that at some point unauthorized or malicious code has to run on your system i love you all security folk but because we focus so much on these high abstract things is the reason that it takes 280 days on average for a breach actually to be detected so start here you've been breached assume it what the fbi says i don't even say nsa or whatever government agency want to go

for says it you have been breached your job is not to focus on that one container that is spinning up now your job is also to focus on the 500 that spun up earlier and what consequences they could have had so i tried to keep this short and beautiful so everybody could get back on time with that i am a binary freak feel free to hit me up with one of our t-shirts but overall i'm saying thank you feel free to hit me up after questions so we can get out of here for everybody and don't forget your homework thanks very much [Applause]