Kubernetes Security: A Developer's Nightmare?

welcome to my talk kubernetes security at developers nightmare my name is andreas falk i'm working for a company called nova tech consulting located at stuttgart germany i'm mainly working as a consultant developing custom software applications for for big enterprise companies like insurance branch or in a motif branch um i'm also a member of the obas community and on the open id foundation so i'm always very keen on application security topics as well like authentication with or out in almighty connect combinator security and security awareness in general the first question that always arises at my talks is what about the slides and where can i find the code so the good thing here is that you can

find all the stuff in my github repository now with corresponding practical demos which unfortunately i cannot show live in my presentation today due to time restrictions so that starts with the path for secure development on kubernetes so we'll start at application security the application security is the basics of all the things and so if you have an insecure application then putting this application into kubernetes cluster does not improve anything kubernetes does not not uh make you safer for from uh authentication failures authorization misses or having secret injections so you have to make sure that your web application is developed in a secure way and is authenticated authorized in a proper way prevents secret injections by

using prepared statements prevents cross-site scripting and glossite request for cherry attacks and also protects your data by using good cryptographic algorithms so the 101 on application security is use input validations for all types of inputs you are using uh and getting inside your system use output encoding to prevent all kinds of cross-site scripting to avoid sql injections use your hibernate framework spring data that is using prepared statements out of the box to avoid that enforce authentication authorization on all endpoints for example by using spring security that secures all that out of the box never implement your own crypto graphic algorithm or your own session management use frameworks like against spring security and also always check your third-party

dependencies for vulnerabilities so even if a bad code does not arise in your own code it can be brought into your system by just adding some third-party dependency and automate all the things by using static and dynamic application security testing as part of your cicd pipeline that's also really important if your application is secured then the next step would to be to put that application inside your docker container which brings us to container security here i start just with the basics to make sure that everybody is on the same page what is the difference between a virtual machine and between a container basically a virtual machine has some some kind of hypervisor running on physical hardware if you have a type

1 virtual machine on it and on top of that hypervisor you have running your different guest operating system so you can run a linux a windows system on on that hypervisor and above that guest operating system you can just install your guest applications almost the same is true for the type 2 virtual machine monitor with the difference that here the virtual machine monitor is not running directly on physical hardware but for both approaches you have a complete isolation between your different guest operating systems controlled by that hypervisor or virtual machine monitor so if we now look on containers and the big difference here is that containers really run on the same base linux hosts the same linux kernel

actually so they share the base linux kernel they share the same linux technology um they are just isolated via what is called linux namespaces um as you can see docker containers or rkt containers so you can also use other vendors apart from docker are completely based on linux technology so that's why it's really crucial for you to also learn linux basics to understand all the important mechanisms of containers as i have already mentioned linux kernel namespaces are used to to separate the different containers running on the same linux host so you have separate process ids for each container you can have several separate network stuff on running on your different containers you have several mount points to two to five systems

separate user and group ids and you can also have control groups for each container what is a control group a control group basically uh puts resource limits on your container so it's a bad thing to exhaust all the host resources by just running one container that consumes all the resources that would make it impossible to run several containers on the same host system but that's why you should use control groups to to limit your cpu usage your memory using devices processes and network resources if you're using java to develop your application then you should be aware that you should at least use openjdk8 update 192. this is the first version that is really container aware

um older versions just ignore the resource limit set and will just use the complete cpus complete memory from your host system you don't care about setting that resource limits my recommendation here is uh also not using java 8 but also going forward using java 11 so java 8 is already quite old quite outdated so so go forward and just update your applications using java 11 so then you're safe and you are container aware by default next mechanism on linux which is really important are the linux capabilities in asian times before having linux capabilities available you had basically two choices to set up users on your system either you have a user that has complete privileges

as a sys admin on the system or you have the other user that just acts like a standard user without any privileges but now using linux capabilities you can fine tune all that things and just that the capability kept net admin to a user for example um so only sets the really required privileges for users and running the processes with docker you can also drop all the unnecessary capabilities like an example a command line on the bottom see a usual use case you will need for for running containers like like apache http server or engine server which typically requires to bind to port 80 which is uh not allowed for standard users to to bind to only uh ports above 1024

are allowed usually so if you want to bind to that port 80 then you need the capability netbind service so you drop all the capabilities not needed and just add netbind service that's all so if you're building container images usually you write what is called a dockerfile you include a base image that you want to use like alpine for example um then you put in your dockerfile additional package installations that you need for your docker image and then you also include your application that should run as part of your container image when you build your container image add it to a container registry just upload it to to dockerhub for example and now if you

issue the docker run command then docker just fetches the container image from the container registry and runs it on your docker environment and still remember docker containers just share the same linux host in the same linux kernel by default if you just run a container out of the box with only standard file statements then it probably will run with a reduced user with a standard set of capabilities set to that use but the easy thing uh to change that behavior is just to add a user to your in your dockerfile to your container and then with the user directive um your your your docker file that it should use that user instead of a root use

and also remember to to restrict the capabilities of running that container image so if you're writing a docker file then i already mentioned that usually you you include a face image um there are lots of well-known base images are available on docker hub like node.js postgres engines and the bad news here is that all these base images have known vulnerabilities inside it is a bit better if you use the the reduced uh variance of of such images like like for note gs10 that the slim variant or even the alpine variant that have a drastically reduced amount of vulnerability but always be aware that these images tend to have vulnerabilities inside and also always check where you

load your base images from so always load it only from trusted registries and also only use well-known images not exotic ones where you don't know if they bring the keylogger in from backdoors by default because of that vulnerabilities inside container images it's always a good idea to to image security scanning a typical image security scanner scans for for operating system vulnerabilities both on base image level and also as part of the packages that you installed in your dockerfile and it also scans all the third-party dependencies on your application for vulnerabilities and luckily there are three scanners available on the market open source scanners like trivia like like anchor or clear can be just easily installed so 3b is a

really easy to install tool that can be run on the command line and can be easily integrated in all the cicd pipelines for container security 101 really learn linux security basics and load images from trusted registries only so so restrict to to only loading from from specific registries is always a good idea so lots of companies also only run their own private registries and and even build their own base images and just load all the images from their private registry and always can images for vulnerabilities automate that in your cicd pipeline they know to root also run with an additional command line parameter for docker in that case to prevent privilege escalation and do not hard code secrets into your

container image always mount these secrets as recommended by docker and always limit your resources to not exhaust your host resources and also lean use linux security modules like ammo s or s e linux so if you have made your security precautions on container security level then we just move on to the kubernetes part when moving to kubernetes if you already are a bit familiar with kubernetes then you probably will know already that kubernetes can be a really complex beast so that's why i always recommend not running your kubernetes cluster on your own use a managed one that is provided by a big cloud providers like like google cloud microsoft azure or amazon aws so if you use that

managed one then you don't have to care that much about the operational security part of kubernetes so you still have to deal with the kubernetes development security parts like you see on the top of that picture here so you have to be be aware that putting your containers your applications as part of pots on the kubernetes cluster should be also secure so what does that uh what does that mean to you as a developer actually um so you have to take care about things like role-based access control putting resource limits uh on your what's um maybe using constructs what is called port security contacts port security policies or open policy agent what are these constructs the first resource

limits we have already mentioned that on the container security level so this is the same approach based on containers always restrict the cpu usage and also memory usage and the next thing is kubernetes already provides mechanisms to also restrict running with root privileges to also restrict a privilege escalation uh to restrict uh writing on the root file system and also fine-tune the capabilities so in that case here as you see we drop all the capabilities so we don't need any special capabilities in that case the bad thing about container security context is that you have to specify this security context for each and every port deployment yammer you want to deploy to your kubernetes cluster

so it's very easy for developers to forget to put in the security context to each and every deployment jammer but that's why there's another concept called the pot security policy which is still in beta um so here the idea is not to put it in every pot deployment terminal but just to install that port security policy central on the complete cluster so that it can be used for all the ports um but the bit more complicated thing in that case is that it's a bit more effort to configure put security policies as you also will have to use the role-based access control mechanism but that's why here is the basics of the role-based access control

of kubernetes on the right-hand side you always have a subject which can be a user or group so basically all developers administrators are users and can also be be part of of groups a service account is always bound to to some some kubernetes resource like for example a pot but these subjects are now bound to roles or cluster roles by using role bindings and a role or cluster role then again points to a target resource or target resources so basically a resource is anything that can run on a kubernetes cluster it can be a pot it can be a service and also a port security policy but this is a typical configuration used for for

bot security policies um you have a subject in that case a service account that you have to specify as part of the deployment jammer in your pot and that subject then points by a role binding to a role and that role then points to your target resource which in that case is a port security policy and in that way you bind your pot to use at a specific port security policy and restrict it from from running as root and all that stuff another approach to restrict your pots for from doing bad things maybe is the open policy agent together with a kubernetes gatekeeper um this agent takes care of restricting behaviors on your kubernetes cluster so it

it installs an admission controller on your kubernetes cluster and now if your developer tries to deploy a pot that contradicts some rules that are defined by the old policy agent then the deployment is denied by the admission control so you can really fine tune here all the rules that are enforced for all the pots running on your kubernetes so for kubernetes security 101 just follow the container security basics as i've shown learn some linux basics and always use a managed kubernetes cluster it's really hard to run your own kubernetes cluster especially if you want to be up to date with your kubernetes cluster avoiding all that vulnerability is happening there from time to time and also enable audit locks so that you

know what your developers what your administrators are doing on that kubernetes cluster or even be aware of some attacker doing bad things on your cluster always enforce uh authentication on kubernetes you can also use mechanisms like out of mighty connect for authentication there and enable road based access control so it's a bad idea for for giving all developers cluster-wide admin access to your kubernetes cluster then use pod security policies or the open policy agent with the gatekeeper to restrict behaviors on kubernetes and if you're using helm then it's a good time to to upgrade to helm version 3. the helm version 2 had some component called tiller and tiller basically also played root inside your kubernetes

cluster so version 3 removed tiller so it's a good thing as well to upgrade to the latest version and one really important point is always monitor your kubernetes cluster so if an attacker managed to to uh access your cluster and and put bitcoin miner inside your kubernetes cluster at least you recognize that with monitoring it the last part would be about kubernetes secrets in kubernetes so secrets are spread everywhere inside your world so it's it's spread into your applications in properties it's spread into your git repositories in clear text and it's spread what is called kubernetes secrets that are stored in the lcd database the bad thing here is that all kubernetes secrets are stored by default only as base64

encoded values so not encrypted um so it's a good idea to always combine kubernetes with what is called the key management system which is luckily also provided by the big cloud providers so you can easily connect kubernetes with a key management system that handles all the encryption and decryption stuff and as well it's a good idea to also combine that kms systems with your git repositories to avoid storing your secrets in clear text always encrypt secret data at rest and in transit also be aware that in cd in kubernetes secrets are only basic c4 encoded by default always also restrict interactions with the secrets api using role-based access control and then also mount secrets into your your pot instead of using

environment mapping to do that environment mapping always has the risk of of leaking into locks for example that brings us to the final summary of the talk a kubernetes is a really complex thing so always check alternatives so really check if you really need kubernetes so if you only have a simple application then maybe a better approach would be just to put it into a virtual machine on amazon ews for example or check other alternatives like hashicorp normat that is also much more easier to to administer than kubernetes also follow application security 101 make sure that your application is secured preventing secret injection gross side scripting all the bad things happening that are stated on the overs

top 10 list then also follow container and kubernetes security 101 as i have described in the last 30 minutes so ensure your secrets are encrypted in kubernetes that's also very important and also never store secrets in source controller your kits or subversion control and then the last point check out the demos that are available in the github repository you find practical demos for all parts that i have described as part of the slides unfortunately only in theory so so the practical part can be found in the github repository i invite you also to just check them out try out and see how that behaves in practical so that was all what i had to tell as

part of my talk for today so i'm happy to answer your questions and thank you very much