Pentesting Cloud Sandboxes in the wild

to our conference talk pen testing cloud sandboxes in the world for besides munich 2020 uh in august um i will we will present our content and before we start we introduce ourselves um first of all who we are matthias introduce yourself please yes hello everyone my name is matthias i work as a platform security engineer at salesforce taking care of our platform as a service offerings um very happy that we can do this talk together we're actually sitting next to each other yeah even though we live quite a bit away from each other so yeah very excited um dealing with cloud containers almost full time for the last four years now i still love the topic and i'm very

excited for the talk yeah thanks mathias uh yeah my name is jan i'm a kind of itc security specialist and i love to play around with containers and orchestration solutions and nowadays um working as a freelancer did worked in this area and as a the last few years as consultant as a penetration tester and whatnot yeah okay so enough to ask let's have a start with the content um we will start with a short recap last year you hear from andreas cripps on besides munich 2019 a talk about containers he introduced you to what's about containers how do they work the history what's docker what's the docker engine and how it do how are these container images

composed how does it build up and so on in general we have a container and you can see on the right hand side a container is building up on a legitimate host system how does it exactly work we will have an idea also one second if you're not super familiar with containers yet we would recommend that you that you look at this talk and watch this talk from last year because it's a really good container introduction so if you're not familiar with the container basics we will not cover those so before watching this go ahead and watch the talk from last year thanks matthias for the hint and this situation um just to get a short recap

what we are focusing on so first at all linux containers are just a group of processes and they are bound together based on linux kernel features what exactly kind of kernel features we have in background linux namespaces that are used for isolation like username spaces networking file system what not and what else of kernel features are used in in special case of docker we use they use c groups to eliminate the resource of the system and even the capabilities are um reduced and we even have um introduced some second filters so you have not all access to the kernel they are limited by dropped capabilities and filtering and so on um what kind of container environments do we have

actually we have the run c system containerization tool alex c is the old one maybe the old guys from the hood know um and then now it nowadays we have like systemd or docker or even firecracker to name some new one um actually you can have multiple containers running on the same host well it's just a group of processes that are stick together and bound together and um normally in terms of containers you'd not never have a virtual machine but in some special edge cases you have some lightweight um virtual machines like cutter containers or firecracker where we have a microkernel started as virtual machines but we will come to this in a later situation what i

always find super interesting is that the term container is absolutely not defined it's very often the case with any like industry driven uh i don't want to say standard because that implies that there is some definition or standardization but it's not the case for containers so with all those container runtime environments like lightweight vms or down here on the left run c lxe docker and so on so you basically need to just look at the code what they are doing and that's what a container is there is some degree of standardization coming in with oci the open container initiative so that's pretty much the most standardization that we have but it's still rather new and not not the right it's

not arrived everywhere yet so okay um get back to the talk what we will cover this time we will discuss at first some container breakout techniques that are possible and what you can perform on normal containers and the general attack surface of containers um this may be an interesting topic but um to be honest it's almost kind of a misconfiguration issue and and when you talk about just about your container security situation it's normally not all of it the relevant risks are coming with the control pane management of the control pane the api supporting services that you used to build up containers um or even your supply chain these are the real hard and tough things

to cover but before you can solve these issues you must get an idea for the basic stuff like container security in terms of attack vectors on the host file system from the container perspective we can share only just directories with the container or even the whole root directory of the host but in if you start a container with an insecure configuration you may have shared the whole root host directory and if you have access to the directory from a container you're still root on the host file system to escalate from a container to the underlying host you can use in this term in this situation last just normal linux privilege escalation techniques for example you create a new

user and connect by sh by deploying a key or even create a new cron job and and wait for a reverse share connection something like this there's something what you can perform in this situation um another big thing everybody's talking about and you may have here is just um the tech factor by privileged containers um if you start researching and looking for some details about privileged container container and container security there's always the hint never start your container privileged you said well what does it mix actually mean a privileged container gives you direct access to the linux kernel from the underlying host and to the hardware of the host system um what do we actually have in the the

security measurements that were introduced by docker to have somehow a separation between the container and the host kernel is by [Music] only a limited access versus this file system you know this is a fire system for the linux kernel where you can access it directly and normally use capabilities are dropped and in terms of a privileged container you have access to serious capabilities like sysadmin or sys module for loading kernel modules or something like this um or even access to the host devices what could probably go wrong yeah you know you can if you have access to the devices you can just go and mount the host the volumes like like the hard drive into the container

and modify it so it's like similar to the previous attack case or even just from one of the last um blog post that was published with like access to the proc keys where you have the kernel key ring access to and read out the um details of the key ring but in this case i would recommend you to to dig into the blog post it's amazing um what kind of attack vectors do we apples have uh actually we have in some cases that you start your container with mount docker socket the docker socket why do you actually need the docker socket in the container the docker socket is the control plane to control your containers on the local

system and if the control plane is mounted into container for sure what do you can do is just use the control plane and start new container and these containers can be started in any flavor you like for example you make a superpower container which is privileged you have access to the host network you can directly interact with the other processors by ipc and even modify the host namespace and if you just the last step to get out of the box is switching the mount namespace which you can use unless the system command it is enter and then you are in with ns enter t one and dash m then you are in the mount name space of the hosts

of the init process and then you actually are owning the system that's also an attack vector so um since we are talking about cloud this the previous attack vectors were more likely to be for local systems if you have a local docker container that started but if we are in a cloud environment what actually is uh the dangerous part with the cloud interfaces the most interesting one and if you're looking into bug bounty for example there's always an educated and talking everybody's talking about server side request forgery though you have access to the metadata api that's the control plane of the cloud and just having a container start and stopping is just one point but if you if you

decide to go into the cloud with your containers you normally leverage features from the cloud what exactly does it mean you have service accounts for the metadata api just access tokens for this metadata api and from there you can control the workloads you may have access to the storage that you have configured like in f3 storage or something like this um or in in in worst case you can even go and can take over the cloud account depends on the configuration of the access tokens we have compared common cloud platform provider that are out there to name a few uh to name them we have aws from amazon azure from microsoft just local docker um a new cat from the

block is a fly i o and even gcp and heroku as container providers um to to get an idea from the comparison in the default case you have in in case of aws ecs the metadata api um in this part we have a remark that you have a custom cluster that has to be started exclusively for us in aws fargate in comparison you have a shared host so you can see it based on the second filter for example that there's a separation furthermore um in in case of fly iho we have what i've told you in forehand a micro micro light right 4m little machines like firecrackers uh so you don't have x actually not access to the linux kernels

underlying curl um there is a an abstraction layer for the kernel by the firecracker uh in case of google cloud run we have a zg visor or in heroku zelus use under the hoods lxc and so from this derived we have different measurements a detailed comparison is on the github side that's linked below the image and that can find all the different capabilities and second filters that are introduced okay the summary of the attack vectors matias so jan was giving us an overview of the different attack vectors so basically i would summarize them into mounts from the host system which opens like possibilities for all the different traditional linux local attack vectors we can have privileges assigned to a

container it can be either fully privileged you can have some capabilities some system calls might not be filtered or accessed to certain unnames-based and entities components might not be available like for devices then you may have sockets mounted the most prominent example is the docker socket and you have like control plane access by that we mean for example access to the cloud metadata api but it can also be other control planes as we will see in a moment um if you want to perform in a text surface analysis or like pen test cloud sandboxes uh container environments in the wild as the title of our talk suggests there are actually two amazing tools already out there there is

a bob with a silent team by chris leroy link is here on the slide and then there's mi contained by chess frasel both of those cover slightly different aspects of those breakout vectors and we will go through them right now also i have a i have a backlog of things that i want to add to the tools jan already like had some pull requests merged into them so we are really big fans of those tools so if you want to automatically analyze those attack vectors um first we will have a look at bob one of chris's hobbies is looking for linux sockets and find fun things that you can do with them so there's a lot of options available

for that so there's also an auto pwn option which is basically it um moves your own container into the host namespace with like everything namespaced in the host so basically your current process is running as a regular linux host process now this is abusing like an exposed docker socket and the auto pawn option basically just does what you usually want to do get a privileged environment you can however also look for domain sockets that are available as i said the docker socket is just the most prominent example but there might also be other sockets available we've seen cloud environments where the hosting provider exposes services into the container for example logging services or other control plane services

via linux domain sockets and since this is usually local they often don't worry about authentication or something like that so we also have an option to like hunt for any type of sockets that might be available in the container and chances are that those are quite interesting actually then um there's two control plane options in bob so it's extracting mounted humanities secrets if there are any available and it is also parsing for meta data access to cloud services this is the contribution that jan actually made to the tool yeah right so this is what bob does there's also some other like fancy features in there for example there's like a ci pipeline used if you

if you have a build environment that needs this type of analysis there's a ci mode involved you can also like upload the scanning results to an s3 bucket and such and yeah then we have mi contained my contain gives you an enumeration of the other privileges that your container might have it is also checking for the docker socket as we see at the very bottom but this is only like the standard location of the docker socket if you want to perform like a more comprehensive search for any docker sockets that are available you want to use bob what mi contain gives you is a good analysis of the enabled privilege separation features going from top to bottom

basically where the namespaces are available and it's here checking for the pid namespace and the user namespace so we see usernamespaces are not being used here this container is also not running with an app armor profile assigned we have certain capabilities that are limited and we have certain system calls that are blocked so the interesting part is if you would want to do this manually you could via slash proc slash pid status actually identify whether you have second filtering enabled for your process however it only tells you is it enabled or is it not enabled if you want to know which system calls are blocked you just actually need to try and perform all system calls and

look for the return codes that you are getting this is actually quite interesting since on the heroku platform we often got back bounty submissions that oh you're not doing system call filtering but certain tools are only looking for certain error codes and we return a non-custom error code for our filter system calls so be careful when you perform this analysis there so these are the the tools that are available for automatically assessing your container environment and i think this is also what you used for creating the overview of those environments so um also what can you do about that if you want to ensure that your containers are properly hardened from the runtime perspective we will come back to that but as we said

this is only one aspect and i often refer to this as container runtime security so if you want to harden your environment and you want to mitigate all those potential problems um what you have to do depends heavily on your runtime if you use a public platform as a service runtime environment there's usually not that much that you can do they basically won't allow you insecure configurations with regard to breakout techniques i mean that's in their best interest to maintain isolation otherwise they could not offer a multi-tenant platform so if you want to definitely host your own environment which may be required by um some security policy or some consideration or whatever um you have to harden your own run time

obviously so there's basically two options that i see you can build your own abstraction layer for deploying workloads so you basically offer like a small web service where you can submit workloads to and then you basically build your own runtime container yourself with the own runtime configurations that you choose and the abstraction layer that you have basically just allows for submitting a docker image but how this docker image is started and how the runtime is configured is completely up to you as the infrastructure provider if that is not an option for you for some reason you can still basically analyze the deployment artifacts that are being submitted to you and there are already several tools out

there when it comes to docker file there are actually two linters out there that gives you very good results when it comes to security or best practices in general but also security best practices for docker files for example don't run is rude don't use add with like don't use ad at all if you can avoid it so there's two two options available for docker files for docker compose files um we're not aware of any tools that do that however i also don't think docker compose is super relevant anymore at the moment personally i like docker swamp a lot but everybody else has jumped on the kubernetes bandwagon so for kubernetes there's actually many many options available how you can

hard in your runtime environment this is only like a selection of tools that i like the best so the top one is mkhit the managed kubernetes inspection toolkit that does not only analyze your running containers but also your cluster configuration there is styra which is a paid service by the open policy agent founders i think or at least some early developers that basically now offer you managed open policy agent configurations for your kubernetes cluster for all types of things something very similar is k-rail it's a very similar concept that also like hardens your cluster and also continuously monitors it it's a very good tool and then like a more of a you know like audit-based to this cube

score however when it comes to hardening your runtime environment it's actually also not that hard to harden your kubernetes cluster yourself q any of these clusters have something that is called admission controllers and you can basically integrate port security policies and opa they they have a big overlap in what they can do but some things can only be done with opa for example so you determine your the hardening that you want to perform and then there's actually good guidance available how you want to design your port security policies and how you can actually tie that into an admission controller so admission controller is a feature of kubernetes that as the name says controls what is getting admitted

to the cluster so any deployment is pushed through a chain of admission controller hooks so you can also write your own admission controller and just hook into that chain and then basically give a yay or nay upon deployment however all those admission controller or any web hook controller in kubernetes clusters are very so basically they are they are blocking if you want to deploy anything and one of the controllers that are being configured fails nothing works anymore you can configure your cluster or can make any changes the running workloads they just work fine but any changes that you want to do they need to go through the admission controller chain so if one of those fails in there anything is going

wrong um you have to troubleshoot and debug your cluster so they are a little bit of a double-edged sword they are cool things that you can do with them but at the same time they are also mission critical because they can be blocking changes to your cluster so um summing this stuff up um containers are an amazing uh thing from an engineering perspective we're both like huge fans and i think we use like docker containers for lots of daily tasks almost everything yeah right um and you can definitely operate them securely um as i said i'm working for like a big platform as a service provider that is whose isolation is mainly based on containers and

we did not have to release any any press release about failed multi-tenancy issues uh yet so you can definitely harden your container properly and also enforce proper tenant isolation with them you don't want to misconfigure anything actually all the many containers come in quite a yeah many container run times come in quite a secure default configuration i have to admit kubernetes had some ugly outlet layers in the past but basically if you wanted to run a pot as a non-root user you could configure that but kubernetes was not doing that i think second filters are by default or also just finally coming to kubernetes so you can import them by part security policy you can do that but the default

deployment does not leverage those so with kubernetes you definitely want to pay some attention to your hardening but if you have a default docker runtime the state is actually not too bad yeah and then it completely depends uh obviously i'm a big fan of platform as a service providers that offer you container services but what you select in the end um might heavily depend on your company policy or which golf cart um your company xx are going to so have a look at your cloud platform provider you have now some tools to analyze what they are offering you from the security perspective and this should enable you to properly harden your own environment wherever it

is running some uh further tasks that we have on our mind some tooling that we want to do for example the cloud metadata scraper for all the big crowd cloud platform that basically just dumps all the metadata content uh something that's not available from my perspective yet the one project that i want to do since a long while is a container privilege footprinting where you basically run a container and monitor which system calls and capabilities and such it is using so you can actually minimize the available privileges to it we also want to extend bob to scan for binaries with assigned privileges so those privileges the capabilities that we talked about earlier they can also be assigned to binaries

not only to users but also to binaries so this is a very interesting thing and a good way to backdoor a system absolutely yeah and then personally i also want to extend the mi contained documentation so you can actually also read about what mi contain does what the impact is and make it a little bit of like container privilege knowledge base you can always read the code but we all know that can also be tedious at times and now thanks for your attention and we're more than happy to take your questions in any form yeah thanks for your audience while you gather your questions here are some references that we have in here and also

shout out again to chris and jess for the amazing work with bob and am i contained