k-rail: A Tool to Manage K8s Securely at Speed

thank you for coming along you may have seen me from previous shows such as talking about privileged containers I mentioned Cairo this is Cairo so let's jump into it cool so the things we're going to talk about today I will give you a general introduction for those unfamiliar with kubernetes that's okay we'll give you an off knowledge to come along for the ride we'll introduce the general problem that we commonly face from a security perspective with kubernetes will demo some of the attacks in some other ways that both can go wrong and then we'll also introduce the solution for the we were built for ourselves which is Cairo now we'll give you a demo of that and then wrap up here

is your typical who am i slide done slightly differently if you face that into a terminal which should spit out similar stuff as normal who am i I'm Sam confusing you also called Frenchie infrastructure security engineering manager cruise at cruise yeah we're a company makes help driving cars and I'm Dustin Dekker I'm a senior infrastructure security engineer at Cruises cool and get the the important stuff out of the way yeah cruise we're building the world's most advanced self-driving vehicles to safely connect the people places things and experiences we care about and where aren't cool so similar to the main talk we'll do the the usual test who here has never even heard of kubernetes complete

background never heard of it give me big pirate yar all right okay cool this is for Ian if you're in the audience I don't believe you cool everyone else who's maybe use it once or twice but not super confident yeah dabbled a little bit but give me big yar okay that's yeah like maybe a little bit more what about people who've used it a lot feel pretty confident yeah operators or maybe even you I call contributors to kubernetes who's in that boat give me a big yar okay I'd say most of the audience so far has been probably in that boat second cap and then finally who's just confused and from the shipping industry wondering why we're

talking about containers give me yeah one one person great I'm very sorry whoever that was but we'll break it down will give you enough background knowledge for those unfamiliar so for the shipping industry you should be along for the ride so you may have seen something like this when looking at what is kubernetes you know in description such as yeah it's an open source container orchestration platform for automating application deployment scaling and management with a diagram like that that you look at and still have your heads yeah scratching at the end of them so we'll break it down to very very basic first principles the only knowledge you really need to know for this talk here

we have a nerd this nerd would like to make a deployment and so they make a deployment file which is typically a gamma file that describes you know this is a pod this is the image that we want and yeah here's you know some to metadata around it such as how many replicas we want and then that deployment is passed on to the the API server and that's that's power deployments work with kubernetes effectively one one important note about the threat model of the way of thinking about today is that we're not talking about malicious administrators people who have access to admin the Kloster have access to admin across that that's expected we're also not talking about

people who are completely unknown stright we're not talking about kernel OD yeah cribben itísö days or any ways that you can relate escalate we're talking about cases where yeah commonly red teamers who have creds who want to try and break clusters or most commonly by far is well-meaning developers who are just trying to do their job who don't realize that they're accidentally deploying something that's insecure and we want to be able to provide that feedback to them at the earliest point in the deployment lifecycle for them to prevent a security issues so a little bit more detail particularly related to how how Cairo works so what is the API server well it's the coordinator of the

system so it receives this request and then talks to the other moving pieces such as nodes so nodes are effectively the servers that you're trying to talk to here I think ec2 GCE or yeah any compute really and they talk to them very little thing called a coup blip which is an agent that runs on those nodes and then talks out to the API server for instructions means receives those instructions effects the change which is the pods we were talking about before which is you know groups of containers with certain configurations how many replicas you want the image and so on and the power of kubernetes really comes from the fact that you can run many many

many nodes so that's that's that's it in terms of the nouns that you really need to know what are the verbs what does it actually do well you can think of kubernetes as a state machine one critical thing to understand here is that it takes an ideal state and then realizes it using a resource specific controller in this case here we show a simplified deployment controller that you know it takes a request and then make some pods pod memes not dead yet and then yeah but it could do other things as well such as your ingress right where I would take an ingress deployment and then pass it to the English controller which would affect

change on save load balancers make some ports open and so on and one critical step here which is important for Cairo is that there is a an admission controller right so when new deployments come in you can check them with the validating web book or a mutating web book and then either yeah accept reject or yeah mutate the deployment as appropriate so with kubernetes we have this you know powerful tool that provides planet-scale capabilities however it's very complex and tiny misconfigurations can sometimes lead to critical vulnerabilities so really kubernetes is the death star in this talk we hope to demonstrate how a talented pilot can fire proton torpedoes down some of the common exhaust ports

and then also show you a tool to how you can prevent this over to Justin thank you Sam so I'll cover some of the security issues that we've seen give some demos of them in action and then I'll introduce the tooling that we that we made to address this so compromising vanilla kubernetes is easy but securing it is difficult even if you you create the cluster with best practices like role based access control and mutual TLS between the notes there are still some routes of privilege escalation through standard kubernetes api s-- and we recently released a blog post to cover some of this in more detail that you should check out on medium cruises medium and yeah so some of those api's

that kubernetes offers for pods about you des specified by mounts and this can get access to the hosts file system host networking which can be used to bypass a security feature which will cover soon running privileged containers which is something that sam Amaya covered while yesterday and that's not even talking about kernel vulnerability server 0 days so ultimately these features can result in host compromised or cluster compromised and when that happens they can often obtain secrets they need to pivot through the rest of infrastructure one quick thing on that image I absolutely love is yeah have you seen a container like that where like you just look at it it's like oh it's a Thai fire inside a container that image

was just perfect a custom fabric yeah a lot of surface area which can help facilitate these attacks externally so here's a demo of using hosts paths are also called bind mounts to mount the hosts file system and it's used to enumerate every processes environment variables which can often contain secrets have you spotted one is a private key in there another common practice is escalation through a docker socket that's mounted this is commonly mounted for building containers using the hoe stalker daemon and this is that because it is full API access to the host docker daemon which can then be used to spin up new workloads and escalate access and then cloud providers they take care of many of the basic

security best practices when bootstrapping a cluster but they can also add some surface area for attack in this example the GCE metadata API includes a cube and endpoint which is used for the nodes to bootstrap access to the API server so they grab this credential and they ask the coop API server for their node certificates and the node authorizer and kubernetes allows a node to access resources that are scheduled on it so if you request node certificates for each node you can request all those resources such as secrets and Brad keys Munro - slick exploit which automates this process for all nodes in the cluster and gives you all the secrets that you need to pivot your access here's the entry

point of it and all it does is it spins up an evil pod copies his script which is what automates the process and that script will enumerate all the nodes in the cluster it'll grab the credentials from the crew them endpoint then it'll use that credential to retrieve the node certificates then with the node certificates 10 tickets as each node and pulls down the resources assigned to them and then finally it'll package them up in a nice tar file and give them to you on your system so let's watch that in action it only takes a few seconds on a small cluster you can see shortly all the secrets dumped out in a directory so we

we saw a lot of these issues some of them we were familiar with some of them the red team brought to her attention and some of them Brad Keys men brought to her attention when he is contracting for us so we knew that we needed to solve this problem we needed a solution that enforces security policies to prevent some of these myths configurations ideally should reduce complexity for engineers or where I can't increases visibility and helps educate them and it shouldn't break things that are already running we don't want to slow down their velocity too much because they're trying to deliver a product but we still want to have a good impact where we can and finally it needs

to be easily extensible these are some security issues that we recognized today but surely there will be more down the road and we also wanted to support some operational use cases that can help make the clusters more stable and performant and there's some prior art to many of these projects were started around the same time or a bit after but none of them today still solve all the requirements that we set out a common myths requirement is that they they support giving feedback for pods that are applied to the cluster but not higher level resources that ultimately implement pods like deployments daemon sets staple sets and cron shops some of them lock a monitor mode which will run

passively and tell you when a policy would have been enforced and all but the PSP the pod security policy lock flexible way for adding exemptions which is a very important feature when you have large clusters for diverse workloads so introducing Cairo Cairo is the workload policy enforcement tool that we entered to solve these problems and it includes features like telemetry which enable you to see what is enforced or would have been enforced if you're running a passive mode it includes many impactful policies out-of-the-box it has easy and flexible policy exemptions structured violation data that you can process with other tooling to help you make decisions and then one of the biggest features is the real-time interactive feedback and

we'll dive into some of these features now but first how did we decide the name internally we called it security validator which was descriptive because it uses a validation web hook and kubernetes but is too vanilla and we wanted to use a carp on so we went to guardrails but that was taken in several ways by existing security tooling and also a security company then we settled on Cairo which is another carp on and they're also known as jersey barriers which are a divider for traffic used in the states and it has K which this Lita is a kubernetes is the closest we'll get to a dad joke in this talk as well so does Cairo work

well we have a bunch of hackers and nerds hopefully mostly benevolent hackers and they want to apply resources to a cluster when the what validation web hook is configured or a mutation web hook it'll create an emission review request to Cairo and then Cairo will process the context of that request so the research specs and who is applying the resource and where it's being applied and then it can make into this a decision whether to allow or deny it and that comes back in a response from the API server and this all happens instantaneously so they get immediate feedback and the feedback pointed up for is one of the most important features to us this is what gives real-time feedback

to the engineers that are applying deployments to the infrastructure or systems that are applying it and in this example they're applying a deployment which is a high-level resource that ultimately implements a replica set then finally it pods and they get feedback as soon as they apply the deployment in this case it's denying the request for a few reasons that they're now familiar with and that feedback going back to norm for security focused engineers isn't really valuable you know I'm mounting the dock recycler submit I know exactly why it was was not and we shipped today with many policies supported out of the box and when you encounter a violation whether it's enforced or not you get a JSON blob and

standard out that you can use for processing so let's take a deep dive through them on what they actually do so no bind mounts prevents by mounts also called whose path mounts to which mount part of the host file system into the pod the node docker sock mountain is a subset of that used mount the docker socket API which we touched on before no helmet iller prevents deploying helm tiller which is often an API that provides unauthenticated access to deploy to kubernetes and becomes accessible from the entire BBC host networking is a feature that allows the pod to use a host networking stack rather than its own and this is a bypass for the GC metadata API host PID sharing

allows the pod to use the host PID namespace and this is dangerous because it exposes a number of kernel to Nobel's but it also exposes all the environment variables for every process running on the machine and they often contain secrets no new capabilities which is something that Sam and Maya touched on in greater detail yesterday but this essentially prevents people from adding additional sis calls that they can use to potentially do nasty things no privileged container which is a superset of that it includes all the capabilities and also a bunch of other things mutable image reference in enforces hash pinning on the images and this is important because the tags that people often use can change under your

feet and this can be problematic if a malicious image was replaced or just from an operational standpoint because the software version changed and this one's slightly different here where it's doctor on Tomcat instead of in genetics can anyone see what's wrong with this case anyone the audience nope so in 2018 docker user one two three three two one notice that there was no default image for Tomcat on docker hub and so if you docker run Tomcat like a lot of people would do it would actually pull down this backdoor image that docker user 1 2 3 3 2 1 put up and we actually was equipped a minor instead of the official Tomcat image that people expected so

they've got 5 million downloads over the period of a few months before I was pulled and so in this case yeah trusted image repository don't pull from public pull from a trusted GCR instead yeah generally don't want your engineers pulling random docker of images and decide to use turnover repost and said so this enforces app and the request ingress exemption policy is useful if you have an ingress that publishes services publicly and this requires that they get an exemption ahead of time so it's white listed and prevents accidental as public explode public exposure of applications and then recently a few more have been added some of these who are contributed but other open source users and the first one that

no pod exec prevents exacting into pods and that might be useful in a production environment for example and there's a work-in-progress operator that's that somebody is working on that will actually garbage collect pods that have been exact into after some period of time so once they've been exact into they are painted you don't trust them and this will ensure that they get deleted and replaced it with a fresh one the default set comp policy just ensures that you're using the container runtime default set comp policy and that's important because kubernetes actually runs containers unconfined by default for every container runtime the override image pool policy was an example of an operational type policy that we've added

and we added that because we're bumping into GCR rate limits with GCP for some of the common and it containers we used and so we we just overrode the image public policy to only download the image if it wasn't always already present on the machine and then finally empty directory without size limit this is another sort of like operational reliability policy and prevents the host disk from filling up on the note from temp of host mounts it's also worth noting here that some of these policies we added but some of these have also been added by the community so we have been sourced this tool in October of last year 2019 and since then we've had

a great response from the community so yeah thanks to the people who've contributed the AHS on this front yeah and there are a number more in progress right now too and so if you wanted to add a policy you have to implement this interface policies are written and go so you have the full flexibility to go but you ultimately need to implement these two methods in it and this has changed a little bit since then because we've added support for mutating resources as well if you want to change certain aspects of them as they come in but it's still pretty much this but the benefit of using a cheering complete language for this is you can put in a logical

like you could say hey this is a deployment after 5 p.m. on a Friday we'll only let it in if we can also query the Dominos API and see that you've already ordered pizza to say Sargent team hypothetically and here's an example of one of the policies that's implemented this is a subset of a no host by announced policy and all it does is it iterates through the volumes in the pod spec and if it finds one that's a host path it'll add a violation and return that and these are some example exemptions that you can use they have several fields that you can specify and they all use string globbing and you can specify like this particular

resource and a certain namespace is exempt from some set of policies or maybe all policies so it's very easy to use and it's actually more flexible than the role based access control way of enforcing pod security policies so what you've been waiting for a demo of kay riddle we'll run this coop and exploit again with a GCE metadata concealment bypass which is the host networking plug but this time with K rel enabled so we're running that Auto explain script and immediately it gives to reject rejection that the host network uses forbidden so that exploit and no longer works now back to you Sam Thanks so we'll share some actual data from our experience in rolling this out

so this is one of our busiest errs with about 18,000 pods at when we did the analysis and that's just a snapshot point in time a lot of these are transient very rapidly changing which makes it hard to do an analysis up right if you just dump it once and then have a look that data is already out of date so we run payroll and all of our past clusters and this is just one that we picked up for example so here is a dashboard and so we mentioned telemetry so across the top there you can see there's the different policies you know pod no bond mount no privilege containers etc and a red box in that

column basically means that their own violation the name space is on the Left we blanked out but this was self-service an engineer could log in and have a look at this and be like oh how interesting mine is violating on these three Oh why's that I'll go have a look and then the yeah we ran that for a few weeks in monitor only mode before setting up enforcement periodically we applied some exemptions ourselves in cases were like oh this makes sense this should be you know doing whatever it's doing but yeah we certain watch for a couple weeks but once once we release this that's when the real magic happened so when we shared an announcement that we were

going to put in place enforcement but people proactively went and fix the problems themselves when provided with the right information and tooling most devs will do the right thing and here's a couple of paraphrase comments from team members which we particularly like the one on on the right there where you know picked up on a particular thing and we noticed hey this is a security risk yeah do you mind yeah is this intentional and the response was like wait what happened this stupid oh geez I'm just gonna delete that so a lot of times the the operators were completely unaware that these things were in were there and they didn't need to be so they

just deleted and got rid of a lot of stuff so yeah that was a that was a pretty big success story then once we put in place the the enforcement mode we also provide another dashboard for violations themselves and so it really only took us about a week to roll out enforcement across all of our production clusters after went running in monitor mode and proactively putting in those exemptions with effectively zero impact of engineering productivity which was which was pretty awesome excellent cool so to recap we provided a brief overview of kubernetes we demonstrated some of the common problems with decoster as as they are and then we also some of these security implications that this has when we proposed the

solution Kol that fixes some of these issues it is open source of course we would love if you had followed down how to play around send an issue or even better send sense and PRS and we've got a maybe time for one quick question if someone wants to throw it on slide Oh or yell add a dad joke in the audience alternatively questions on Twitter later yeah question yep I can repeat the question as well as that's exactly yeah for sure yeah so the question was the policies are written in go Oh PA has an alternative what it is off the top of my head Rhaego well greg oh yeah record city SLO OPA uses and

that might be easier for some people that aren't used to programming but it does have a lot of restrictions like you can't order Domino's Pizza or you can't query the kubernetes api for existing ingresses so the go provides more flexibility and and for somebody that's used to ride and go it might be easier as well because you can use to go tool chain and if you are familiar all you have to do is implement that interface that we that we showed earlier but it's not for everyone there's definitely alternative tooling out there to consider yeah and even if you just want to toll with that necessarily adding your own policies there's a bunch of powerful posters it

come by default so you don't have to write go you can just deploy it for those that do want to deploy there's a one liner and the readme where you can just try it out on a demo cluster yep you can always make a feature request for a policy that you want to that's a way we're pretty quick wonderful I think we're a time but thank you all so much for listening if there's other questions have an answer in the hallway or on Twitter later thanks so much thank you [Applause]