School of /Proc

back folks here we go it's like like conference over that's it oh my gosh all right um I guess uh yeah I guess we can get started and then I will drag people inside slowly whether they like it or not um all right our next presenter today is Alexander wise who's a technical account manager at verica and his talk is titled School of proc kubernetes security and the post PSP world just give it up foreign thank you uh yeah like every other speaker has said so so cool to be back in the conference uh setting in person seeing people I love this uh but I'm I'm Alex I'm here to talk about kubernetes security I'm here to talk about the the

recent past and near future of kubernetes security uh so about a year ago back April 2021 uh with uh the release of kubernetes 1.21 the kubernetes team also announced the deprecation of pod security policy API and kubernetes pod security policies were the only built-in way to configure your kubernetes cluster so that uh a workload could not be submitted that that could trivially escalate to root on the Node um but but kubernetes deprecation is just an initial first step it's initial early warning that says hey this is going to go away eventually um you know start making plans to come off of it find something else that will work for you and there really are no plans to remove

pod security policy until kubernetes 1.25 which is the next kubernetes release 1.24 came out last month um so the next uh the next version of kubernetes probably will not have pod security policies in them um there's still no reason to worry if you if you might be using pod security policies because you have most most folks are not on the freshest Upstream release of kubernetes as it comes out and you will have at least a year of 1.24 being supported before that eventually goes away so you're not going to be forced to force off of pod security polies a lot of folks are not on are not using pod security policies it's a we'll get into

it it's a little bit difficult of a of a tool to use um but yeah we will uh we will get into that later um a little bit about me um I am my name is Alex wise um I've been doing kubernetes stuff for a little while my first production cluster was uh 1.6 a little while ago I've since then secured kubernetes workloads a lot of different places I've done for SAS for robotics for ML workloads um if you want to ever nerd out about development environments or build tooling and how to secure that let me know I'm currently at a company called verica we build tools to help you secure and find the the safety margins of your

kubernetes clusters of your Kafka clusters we also do work around town for the software Freedom School I teach for them write some classes for them helping beginners learn how to use why to choose open source software so some folks may have seen me for that my you can find me on Twitter AWS Architects so that's me I do want to lead off here with a couple quotes from the co-chairs of Sig security for kubernetes the first one um indicates what what value pod security policies gave us uh what how people used them what was the threat model they were trying to stop um the other one we'll talk about the design decisions of philosophy where

they want to move kubernetes to why they're deprecating pot security policies and what the replacement needs to do better so the first quote here is from Tabby Sable from her talk from kubecon last year PSP replacement past present and future

permission to create pod should not mean permission to be root on every node in the cluster so this is the problem that pod security policy existed to solve kubernetes is a is a system administration tool that wants to do system administration things it wants to help you run privileged workloads perform upgrades things like that so if you're able to run those kind of privileged workloads there are a thousand different Paths of attacks ways to to escalate your privileges out of a container way to perform container breakouts that mean that there's really a very fuzzy line between create pod and kubernetes and root on the Node and we'll talk a few of those in a little

bit but but that's that's what PSP's solved that's what the replacement needs to solve that's why we're talking about this the next video is from Ian the other co-chair of Sig security Tweet back in 2020 saying kubernetes emphatically not secure by default um and and we should not paper over that fact that that what kubernetes needs to do better is to be secure by default um you shouldn't need to be a secure container security Guru to to be able to to run a secure cluster so that's where we see PSPs we're falling down and what The Replacements May hopefully do a little bit better it's really important I'm going to put it here we'll probably refer back to it but

kubernetes needs to be secure by default so wayfinding slide here we just finished intros we're going to talk about what is slash was the Pod security policy we're going to dive a little bit more into what those rough edges were why it's going away going to talk about pod security admission it's replacement um and then if pod security admission doesn't work for you or if you're if you're a security research or an attacker out there trying to see what you might be up against a in a cluster we'll talk about some of the other security controls that solve this problem in kubernetes all right so diving into it what is a pod security policy anyway

so pod security policy was an admission control plug-in it was it's the only uh security system like this that's built into kubernetes natively and that that's important right it was it's pretty long in the tooth it was G8 in 1.6 and what it does is it lets you configure requirements for the security context for any pods that run there and those are extremely granular we'll take a look at those some of those but you can be very specific about the secure the security controls that you want on your uh on your workloads it's also extremely flexible you can configure your pod security policy to to default you to secure settings to to to secure or just

validate and say no you're trying to run as privileged we won't let that on the cluster um you can set these policies up so they apply to to all namespaces and everything cluster wide or you can scope them to specific privileged privileged name spaces so there's it's very flexible and very granular which which creates some foot guns for for folks um who may not have expertise in this so uh this is some of the granularity this is just a sampling of some of the the controls you would have uh with pod security policies setting those security contacts you can see things like running privileged containers um you can see in Linux capabilities any sorts of of volume mounts you might be

able to add there's um SC Linux modes and app app armor settings that would be configurable here um and if you see there's a an allow privilege escalation Boolean which is kind of my favorite Boolean like set this to false and hey we've solved security right we we did it we fixed cyber um yeah maybe um anyway so how does this work in practice so uh so pod security policies that they build on the the existing R back and workload identity apis of kubernetes essentially every pod has a service account associated with it and and that says what permissions that pod has so PSPs just sort of build on that and say okay if you if you have this identity

apply this policy to it you might have these extra permissions or you might be running you know very very restrictive but pods map to service accounts map to either cluster roles or roles and then those map to PSPs if you're if you're mutating you're going to get one PSP if you're if you're validating you might get a whole array of them uh the drawback here is that if you're not using workload identity in um in your kubernetes clusters so for instance if all of your pods run as the default underscore default service account in a namespace you might have some heavy lifting to do before you could start to use pod security policies but in theory what I could do here I

could make a a really restrictive cluster role that is cluster-wide that everything sort of gets by default um and then if I have more privileged workloads if I have security tooling if I have I have operational tooling or or Legacy workloads then I have a role that's specified to a single namespace and gives me just those perms there so it can kind of do this high this really cool hierarchical uh setup where I I can get my least privilege right so that's what this kind of looks like here on the left we have a cluster role that's cluster wide it's very restrictive on the bottom there that says resource names that is the name of your pod

security policy the block all the things policy on the right and your lift and shift Legacy namespace you've gotten all the perms pod security policy so that you can kind of do this this really cool thing where you have have hierarchies and you can run the things that need permissions um just sort of as with those elevated permissions without making the cluster as a whole um less secure pretty cool so I think I can do something like that how do I turn it on all right so PSP uh you would turn it on just by by passing a flag in this enable admission plugins flag um to your the cube API server start command um you'll actually just be a big long

string of comma separated plugins that are that are also on by default this will typically live in your cluster build out scripts or your your infrastructures code that kind of automation um so I I can do that I can update this right and the idea is I'll turn this on and then I'll go in I'll start looking at my workloads I'll start crafting policies uh similar to what that previous slide was and and set this up right and so I'll do that and everything breaks because uh pod security policy is a security control it's denied by default if I turn it on and I don't have my PSPs created uh no pods can be scheduled everything

gets denied everything gets shut down and everyone's mad at me um so I need to know I I need to know exactly what my policies are going to look like before I turn it on but I I can't turn it on until I know what my policies look I kind of I kind of have this chicken and egg thing right I I can't quite uh uh I don't know a priori what my PSPs need to look like and I I can't really find my way into them but what I could do is I could like make a really permissive Global policy then turn it on and then start incrementally um you know creating policies for

different workloads and finding these privilege I could do that there are some caveats um you know it will take some some trial and error to get this right probably depending on how complex I need to be with it and it's not a great default I mean we can just say that right this is this is this is uh from a from a defender's perspective um I I have to do a lot of maneuvering to get these turned on to get these running in my environment and and if we're trying to be secure by default that we're kind of falling down here all right one moment cool any questions so far in that nope all right so

we're going to dive in a little bit deeper into pod security policies um we're gonna say okay I have done everything I said in that previous section right I've set up these these cluster roles I've got everything running I think I'm running pretty secure fairly least privileged I'm feeling good about it I'm probably in the top 10 percent of secure kubernetes clusters operators because a lot of folks don't use PSPs because they're kind of difficult then I come across a blog post and it says kubernetes insecure host path mounts okay okay um I dive in I dive into this blog post and I think oh duh right cool so kubernetes lets you mount a server's direct a directory from

the server inside your container as a as a volume inside that container similar to how you would do with Docker right well if I just Mount the root directory of my server inside my container I can do all kinds of shenanigans uh this blog post has you just rooting into into that directory and taking over the server that way but you could you could cut Etsy Shadow you could steal SSH Keys you could steal the the kubernetes node search right there's a lot of Mischief you can have if you have full access to the root directory on a Linux server but I can fix this right I can block this PSPs I have them set up they give

me the tools to to fix this and it would look something like this right I don't I don't even I'm in the cloud right I don't need this host path I don't need to be mounting directories inside of containers from from my server my my servers are mostly immutable there's no there's no uh State saved there um so yeah I'll just I'll just I won't allow um I won't allow host paths I will update all my pod security policies on my cluster and say you know these this is the list of the types of volumes we allow these are sort of kubernetes native things are Cloud native things we don't we don't need host paths we don't

want to touch them and I feel good I feel super secure [Music] um until I see this tweet from Duffy and that's a that's a heck of a tweet uh this can run this on any any cluster that's not running pod security policies and get get root on a node and it's a heck of a tweet right um but I I'm a container guy I'm a kubernetes guy I can I can so I can parse this right there's nothing here that I can't parse so Coupe GTL run pretty standard the image doesn't matter because we're overriding it okay host PID true inner Center all right I think I see what's going on here and I I I remember

something I heard about about Linux containers so Linux so containers aren't little boxes right I remember Alice goldfuss's talk here and she had a mantra about Linux containers containers are processes anchored to namespaces controlled by c groups uh so so container is just a process it's got It's got a pids uh it's in a namespace and that says what uh the Linux namespace says what other processes it can see and what what devices it can see and the c groups control what resources has access to so if I do a Docker run okay I started Docker container yeah I can see that it it has a a it is a process just on my server it has a PID

um I can do some Linux internal stuff and see the namespace that it's running in and I can use that NS enter command and go into that namespace um you know I can also see this information about the C group it's in so so containers are just processes and namespaces so going back to Duffy's tweet all right so running host PID true that means that I'm running in the hosts process space the host name space and I'm using NS enter to to go into that namespace that means that I can see every process and every device on the Node I can touch every file so that I'm rude at that point um everything in Linux is a file so if I do

that I am essentially root on the server but I can fix that right I can I can I can I can use my PSPs to make host bid false and and privileged false and host Network false and privilege escalation false I can do all that to stop it but it's not very secure by default I've had to be a container breakout expert to understand these Paths of attack so that I could craft these policies so that I could stop it and and that is not a place that kubernetes wants to be um so wouldn't it be cool if kubernetes bundled up everything that it knew about container escalations and privilege escalation and container breakouts into

a policy right because I talked about insecure host path mounts and host bid takeovers but but you there's mounting the the container runtime socket inside the container there is running with cap BPF and capnet raw to to forged packets there there is uh unmasking the slash proc Mount there are all these sorts of container breakouts all these acrobatics you can do to get out of a container when you can run privileged workloads and wouldn't it be cool if kubernetes bundled all those up for me so now we get to the PSA on PSAs pod security admission is the replacement for for pod security policies it is also an admission controller uh entered beta very recently uh 1.23 still

behind a feature gate pod security admission is is significantly simpler and then pod security policy uh it it works at the namespace level you essentially uh in kubernetes labels are key values that you that you assigned objects so you're assigning these to the namespace object and kubernetes and basically setting it to one of three security levels and any any workloads any pods that run in that namespace will need to conform to whatever that that that level is um so what you're getting here you're getting a loss of that granularity you don't have that that tool kit that you had um that you could use to craft really specific uh policies but you get better defaults if you're if you're not a

container security expert you you can um get a default secure setting and reasonably trust that that you're not there aren't uh Paths of attack that there aren't privilege escalations that could be performed so I like this um this first sentence from the Pod security admission Docs uh kubernetes pod security standards Define different isolation levels for pods these standards let you define how you want to restrict behavior of pods in a clear consistent fashion security restrictions are applied to the namespace level when pods are created so this is all about pod security standards so let's let's dive into what those are I said they were there were three levels right the first privileged basically no restrictions right if we're

not going to stop every privilege escalation um you know we will assume that the privilege can be escalated there let's let's not restrict anything Baseline is your your trade-off between compatibility and and and this is key here the known privilege escalation paths um it it wants to allow as much as possible to run without opening the Avenue for create pod to be root on the Node um some some of the things that are enforced in a baseline you can't have privilege true there's a list of specific Linux capabilities that you can have but none of the really good ones like capsys admin um no host paths at all we saw host paths were really problematic

um and there's there's a list of others but those are those are some of them the the Third uh mode that we can run in here is restricted um and that's that's enforcing every known pod security best practice so things like your containers can't be running as root um any capabilities have to be have to be dropped any learning Escape abilities uh whatever the the host SC Linux mode is has to be enforced things like that um also there's one where where no child processes that are spawned inside the container can have more permissions than the container at the the parent PID things like that so so those are our baselines um very simple buckets that we can we

can put our workloads in and it's behind the future gate right now but it's enabled by default we are moving closer to secure by default and kubernetes with pod security admission all right so let's let's take a look here what do these labels look like I said it's just two labels that you would set on on the namespace object to set this up so the first one is this mode label um so mode you enforce audit or warn the only difference between audit and Warren is audit is put writing to your audit logs um and Warren is just writing out to the console um and then you would the level is one of those three buckets the privileged

Baseline are restricted so you would say something like pod security.kubernetes.io enforce Baseline and in that namespace now every pod that that could be scheduled in that namespace must comply with the Baseline security standard so no running is privileged uh you'll you'll get rejected you won't be allowed to run in the cluster there the other thing I want to call out here is the addition of that warn mode is pretty awesome um what that does is I can set that and I can start looking at my logs I can start looking at my console to see oh you know am I getting warnings and will I need to to move things to different namespaces if I switch to

enforce will things start breaking it gives me a road map to go from Brownfield to secure cluster which was huge it's that chicken and egg problem that we had with PSPs we don't have here because we have a worn mode that we can we can start using right away uh by default out of the box so um the other thing to keep in mind is is the life cycle of these of these versions right a baseline uh security isn't set in stone um we could imagine that the Baseline security standard might need to change in the future you know if one of the researchers out there starts finds the way to to leverage one of the the

allowed Linux capabilities to do some new fancy container Breakout Baseline needs to get updated there um and so operators need to sort of have a version in mind and there needs to be a path to go from from version to version um and six security has decided let's just tie it to the kubernetes life cycle kubernetes release is basically quarterly um so so you can label here and say my enforced version is kubernetes 1.24 and you're enforced uh you're enforced uh your Baseline standard that will be using the kubernetes 1.24 version of that Baseline so those are the two labels that's how you use it that's that's pod security admission in a nutshell um pretty straightforward

um we get we get better defaults we get uh we get a roadmap we get that worn mode we get that road map to be able to turn it on to see what might break um and it's just been a a an easier end user experience for I want to be clear but the the built-in because this is the built-in mode of running kubernetes securely they wanted it to be usable by by folks who were not container breakout artists um all right uh but but PSA won't be usable by everyone um you may have you may have compliance regimes or security policies that don't map to to one of these buckets right you may not have your workload split up into

namespaces in a way that really lets you leverage this um you may need to get to get more granular you may have custom hosts that have to run that SC Linux uh configuration so what do you do then uh what what is out there for you if you can't use pod security admission and pod security policies are going away well I said I said at the top that pod security policies were an admission control plug-in for for kubernetes pod security admission is an admission control plug-in for kubernetes so that really begs the question here what's an admission controller in kubernetes and I've stolen this uh this diagram from the from the kubernetes docs um so imagine you run your Cube CTL apply

command you're submitting a new pod a new workload to run in the cluster it enters the left side of this diagram here as an API request to the API server I mean there's a series of different validators of different uh of logic that's applied to say okay is this is this workload that has been submitted is this good is the per is the person who submitted it allowed to submit that and and is it well formed and there's a there's a ton of these they're not they're not all security related um but these admission plugins for example um there is one that will check and say if you if you want to run a pod

uh are you trying to schedule that in a namespace that is currently in the process of being deleted because if you were to do that then then we would get into an infinite chain of of deleting things um and the circuit breaker for that is an admission control plugin so there are lots of these um uh more than a dozen um and they really fall into one of two types they fall into over here on the right we have the validating admission plugins that those just say thumbs up thumbs down is this workload allowed to run um hopefully if it's a thumbs down you get some sort of an error message that gives you a reason why

um a little bit further to the left you see the mutating ones um if you were to run your pod security policies in a way that like defaults everything to privileged false that's your mutating admission web hook um I think Nathaniel when he talked about service meshes a pretty common application for for mutating Mission web hooks is to automatically insert that service mesh container into every workload uh or or a logging container sidecar into every workload so that that's um that's what you have as far as admission control and kubernetes and kubernetes is extensible it's flexible and there are three of these plugins that are enabled by default that uh that give you extensibility and give you the

ability to write custom admission logic um the image policy web hook the validating Mission web Hook and the mutating mission web hook you would set these up they're just a series of kubernetes objects you would apply one of those is a is a pod is a web server that's running um when a user goes to submit a workload to kubernetes it reaches out to that web server and says does this pod spec look good thumbs up or thumbs down and gets the response but that's your custom logic that you can run and and pod security admission if you can't enable the feature gate they give you all the the steps and all the code to to run this separately as

its own separate validating the mission web hook um it is it is a very valid way to run it um all it is under the hood is a validating Mission web hook it's just the one that's included by default with kubernetes as well and you don't have to to write this these custom web servers these custom applications right there there this is a paved road if you if you want to do this there are lots of big mature projects open source projects to configure custom admission logic in kubernetes so all the things you could do with pod security policies you can do with these projects and you can frankly do them a little bit easier and you can be a little more

flexible do them a little bit better I just threw up before here that I'm familiar with there are more out there um but for instance you can you can do more with these than you could do just with pod security policies by themselves you can you can get all the Pod security standards and all of the PSP configuration all of that security context granularity but you could also do things like require labels on all of all of your workloads so so any pod would have like an owner label would map to what what team is the point of contact for that for that workload um if you're using Valero or or Caston or some other sort

of uh kubernetes backup uh vendor backup software you can you can help configure your backup policy uh using admission control I talked about adding sidecars for logging or for or for service meshes um you can do things like all images that run in this cluster have to come from my private Docker my private registry nothing from Docker Hub um can run and there's all sorts of security controls extra security codes above and beyond even with pod security policies provided that you can do like like um mounting the container socket inside a container or or doing like um uh engine accent this was this last bullet point here nginx Ingress is a is an Ingress that runs

um if you you can apply custom nginx logic to your to the load balancer there if you pass in an annotation to an Ingress object and that opens the door for things like HTTP Smugglers so you can even like uh you know enforce some controls there that you couldn't before with PSPs so so it's a little bit more flexible there's a lot of cool stuff you can do um and and yeah if PSA isn't for you um then then maybe you want to to look at some of these some of these projects here um so yeah takeaways oh man um so pod security policies going away in the next release probably August um but one two four won't go away

forever it'll it'll you'll have at least a year there um PSPs were pretty challenging these out of the box and really to to craft them you had to know what you were protecting against you had to know how to perform container breakouts which is a pretty high bar for for operators for security teams um you know you shouldn't need that specialized knowledge to use the most popular container orchestrator in in the world um so hopefully PSAs do that a little bit better if you weren't using PSAs because they were complex maybe you can use or if you're using PSPs because they were complex maybe you could use PSAs maybe maybe that's a better a better

method for you easier to get your hands around um if not other other Solutions exist um that may even be more powerful for you so um one other thing I want to call out is that pod security policy deprecation is not the only enormous change going through kubernetes right now and it is a big change it's one of the biggest changes that's ever happened in the history of the of the project um but at the same time we have we have now ga'd uh dual stack networking as of 1.23 and that's been A Five-Year Plan I think that started in 2017 to come off ipv4 and then go to dual stack and then go back off dual second basically write

the whole networking stack um Dr Shim was just deprecated as well in this most recent release um that's a huge change that was that was talking in the community hey shouldn't this be a major version bump for all the things that we're going to be breaking and changing here shouldn't this be kubernetes 2.0 so a lot of a lot of big changes coming through the kubernetes community right now coming through the kubernetes project so if you are an operator of a kubernetes cluster or you know an operative Community kubernetes cluster have some empathy for them because they they might have a lot going on um got some references here the tables PSP talk Ian Coldwater and Duffy

Cooley's abusing kubernetes default talk and then some other blog posts there and that's what I got for you any uh any questions yep um awesome with the uh pod security that's awesome um two two questions really it could go to three we'll see um so with uh the first question uh if you have sounds like we're just kind of uh formalizing in the stacks you know you have your your enforce and your audit and you're born um is there any way to uh enforce and warn at the same time you can use multiple modes uh or is it one or the other um I haven't tested I've only had like one cluster with the feature gate on I

believe the the the the idea with that schema is that you can't have uh you could you can warn on a restricted and and then you know enforce a baseline or something right um so I I do believe that but I have not tested it so that's a great great question um doesn't work automatically warn or write some things to the log I mean so has logged been elevated within access logging yeah well you'll get um you'll definitely get console responses you'll get a reason that it did not pass admission because of because of the the label um but then um yeah and you'll see that in in event logging as well if you're capturing the

event the event logs which users submitted it yeah um I'd have to look at the autolog I believe the autolog format would have that not necessarily the event format um it would have had to be done by an authorized user um the the audit logging would have that level uh but once it's once it's in Ed CD it's it's once it's written at CD is something that should be applied which is on the other side of Mission Control then you would you wouldn't have that but if you're saying you're stopping an admission control you would see the apply and you would see the stop okay thanks so um so sv-1x and apple both have you

know lots of tooling to parse the audit log and find custom policies and have our profile so you have you know minimum permissions you need your right side to make your application run design exists like that for PSA or is that tooling that's still being builts um so the the security contacts in kubernetes so what what PSP was enforcing does let you um just defer to and refer out to any app armor or SC Linux controls that you would have um uh uh oh so your question was on on the visibility of of seeing what would just like if there's any tooling right now um permissions or changing PSA logic I guess not probably counters yeah no

um I'm not sure about the about tooling that exists there um you know it would it it depends on your on your audit logging setup there you would get you get responses back um the admission control side of it you would see um you know that you tried to submit a workload that that was privileged in a place where you couldn't submit a privilege but as far as like if you submit uh an application that needs a Linux capability doesn't have that capability um you know it might crash it might have weird Behavior I think you would be dependent on the application to log that out or tell you in some way that hey I

don't have capnet raw here and I need it yeah um yeah all right can we default back to PSP if we wanted to or is it completely gone PSP will be gone the so as far as as far as we know 1.24 will be the last version that has PSP in it um and and that was an oh [Music]

um yeah I mean so so and the the Sig security has said you know the story for PSP doesn't end until the last user um stops using it um the idea is that all the functionality that PSP gave you you can really get by these these uh these other projects um and to if if people need that flexibility that functionality to defer to those out of tree projects why why diverge

um so so the story of kubernetes yeah yeah okay so yeah the um so the the and you know I don't represent like the cute the kubernetes project or the community or anything I um but the the story uh over the last five or six years has been let's pull out um logic and feature that that isn't core to uh orchestrating containers and let let's pull out um you know the the drivers for specif for for creating volumes in specific clouds for instance so we'll pull that out of the main kubernetes project and and that will be owned as a separate project that the cloud providers can can then take care and will be pluggable and

extensible um so so this is more of a continuation of that for for the built-in security for the built-in thing we want something easy to use out of the box um but but if you need that flexibility if you need that granular that extra tooling look to your other cncf projects um that will be plugged that will plug into the the workflows that are available in kubernetes

any last questions folks yep so if you if you are using this 1.24 version and you have these uh security features set up and you're using them currently and you upgrade to 1.25 what happens um what would happen if you wanted to oh if you had PSP objects um on a like existing in your cluster in at CD the API would be gone you would get deprecation you're getting deprecation warnings now but you would there would just be nothing there there'd be no plug-in available to to apply those um so they would they would exist but it wouldn't um yeah yeah you would have to label the namespace so you're not you're if you're not putting the labels on namespace

you're not you're not actually using the the the um the Pod security admission controller

cool thanks for the time y'all [Applause] all right don't run off too far uh we'll have one more talk um in 15-ish minutes so