Kubernetes Security: Attacking and Defending K8s Clusters

okay um hello everyone thank you for joining um my name is magnu logan and i'll talk about kubernetes security attacking and defending uh kubernetes clusters i'll be available during the talk to answer any questions via the chat okay so let's start um just a little bit about myself i work as an information security specialist for trend micro in the cloud and container security research team i'm also part of the cncf6 security team and uh i i'm not a i don't consider myself a kubernetes security expert i'm still learning but i'm going to share with you what i've learned from the past year about studying kubernetes security this is the agenda for this talk we'll try to cover all the topics

in this 35 minutes talk but that's why i want to talk about this topics right now and and we'll go right into them and just uh before we start i just want to highlight this uh awesome kubernetes security list project that i created on github uh there is the link there and it has a lot of content and and uh tools videos and papers and everything that you can could use to learn about more about kubernetes security so feel free to check it out okay um so what is kubernetes right just to give you an overview if you don't know yet but kubernetes is an orchestration a container orchestration solution right it's basically an open source system

for automating deploying uh deployment scaling and management of containerized applications right so basically containers and it was a region developed by google and released in 2014 and today it is maintained by the the cloud native computing foundation okay um so this this this shows here on a high level the kubernetes architecture and we can see that we have two major components here uh we have the control plane on the left side and um the the worker nodes on the right side so we have three worker nodes on on this uh diagram here um basically uh you have five major components on the control plane side on the left side here so you have the cube api server which

handles uh exposing and and and receiving api calls right you have the sd database with which where kubernetes stores all its internal information um you also have the cube scheduler who decides where to run the pods on the top left corner there you have the cube controller manager which is responsible for several internal tasks in the cluster noticing when a plot does not respond and maintaining the correct number of pods etc and also on on the um you have the the cloud controller manager there which interacts with cloud providers managing resources such as load balancer and disk volumes okay on the right side on the worker nodes you have the cubelet which is responsible for

managing the container runtime by executing containers when necessary and collecting execution information right you have the q proxy which manages network communication allowing different containers to communicate between each other also uh responsible for external requests excuse me and you also have the container runtime right which is the one that creates and executes the container themselves um by default it was docker but now that's being uh deprecated since uh docker shim actually docker shim has been deprecated but uh you can use either container d cryo or podman to run your uh your container as your container time engine okay and and just before we start uh just a quick uh uh note here um on kubernetes you don't

deploy containers you deploy pods right so pod is the smallest unit that you have uh in kubernetes it's the smallest deployment unit and it can contain one or more containers so for example i can have a a pod that with two containers one is the like the main container and the other one is the sidecar which collects logs and and metrics and other information right okay that's good okay so moving on um so today we're going to talk about attacking and defending clusters right so we need to talk about miter attack framework which is basically uh a framework for uh development of specific threat models and methodologies in many different sectors right so it's just

basically a a a globally accessible knowledge base of opposing tactics and techniques based on real-world scenarios right and it was developed by mitre and we're gonna see here first hand the new miter attack framework that was uh published um last week for the first draft was published last week right but before we go into that so microsoft created their own version of their uh miter attack framework which is the they called the case kubernetes threat matrix right by microsoft azure they released that uh last year in april and uh has a lot of good information there and and and some tactics and techniques that we can uh we can see and and highlighted here are

the ones that we're gonna uh demonstrate today in our scenario right but this is not an official uh uh mitre uh attack framework this was uh uh just released by microsoft and but it was very helpful and and then used as a baseline to create the the original uh the official version of the the attack for containers and kubernetes as well which was released last week as i said uh february 18th by mitre and we've collaborated with them and helped them based on the uh real world scenarios that we've seen in the wild based on our research and our honey pots and everything and and so uh it's nice to showcase here that this is being

developed this is the first draft and and it's always uh uh it's it's going to be improved and it needs feedback from the community so if you have uh any data or information about other real world scenarios feel free to reach out to me or check out the blog post by by mitre and and reach out to them directly so this is the attack scenario that i i created um based on the the microsoft one because mitre wasn't released yet right so so basically here i'm describing uh an attack scenario that a malicious user or an attacker would do to compromise a kubernetes cluster right so basically uh i won't go into all the details here

about this diagram but basically you have a web application that's running and it's exposed to the internet and this web application is running inside a pod which is inside a kubernetes cluster and it's exposed to the internet and it has a vulnerability there has a vulnerability of remote command execution vulnerability right so an rce which when exploited the attacker is able to get a shell inside inside the the the pod in this case right and and then from there you can do uh uh he or she can do uh in either like uh lateral movement just just some basic internal recon uh privilege escalation and and from there compromise either the the whole cluster or even the uh the cloud environment if

it's running on on a cloud provider as well right um so one of the the most common things that we see when attacking kubernetes uh and and and docker as well we have a lot of articles published on this uh topic is that um the attackers usually try to uh automate the the scanning phase and identify um open uh or vulnerable clusters or api servers and basically when they exploit those environments either uh via a vulnerable web application or an exposed dashboard or exposed api server um they try to they try to compromise and uh the cluster and spin up uh containers to mine cryptocurrencies right so either either they spin up containers on the

cluster or they try to if they can have permissions to do so they try to spin up a new instances uh bigger instances on your cloud environment to mine uh cryptocurrencies usually monero cryptocurrencies right so just a quick note here uh so while doing this experiment and testing basically i use this uh vulnerable drupal uh version of the of a web application right it's a cms web application excuse me and and so we exposed that to the internet via a load balancer on the cluster and this application has uh well-known rc which is the uh uh seven 2018-76 zero zero cv the drupal gadon ii um and and and just basically um in less than 24 hours

they they were able like the attackers were able to to find that um url and uh compromise it by mining uh uh cryptocurrency as well right so just just while doing this experiment and we also have seen that uh happen uh oftentimes on our honey pots as well so so how do you how do you find that right so how do you exploit how do you identify that uh um and now web application is running on a kubernetes environment right so one of the first things that you can do if it's running on a managed services like eks gke or aks right um basically uh if you can find out the the url of the

of the load balancer that might tell you um that might tell you that uh uh kubernetes cluster and you're gonna see that by the the url on the next line right um another thing here is uh exposed dashboards so uh kubernetes had a head a a dashboard enabled by default which uh before a few versions ago was enabled uh and and didn't have proper authentication um another issue is with the cube api server where um if that is exposed and and it hasn't uh doesn't have the proper configurations then you can reach out to the api server uh sometimes even without uh authentication right so by exploiting this vulnerability here the cve we get a shell inside uh uh inside the

cluster inside the pod right okay so right i'm able to reach the api endpoint externally here are some examples uh by default on an unmanaged cluster uh kubernetes uh uses the port 6443 for the api server and on uh on managed services sometimes is um is the port for for free by default right and you can see here that by the url you can see that that's using eks uh on amazon aws on the second url right so that's easy to to to know that the uh the applications running here are using a kubernetes cluster right so um so that's that's one of the ways to find out about that what uh where the application

is running and after you exploit this vulnerability you get a shell inside one of the pods um either via an exposed dashboard or an rce right uh just a quick note here um i think tesla was uh hacked a few years ago 2018 because of exposed kubernetes dashboard where the attackers found that dashboard uh url and from dash that dashboard they had permissions to uh deploy uh new new pods and and compromised the the cluster environment they deploy the uh pods to mine bitcoins cryptocurrencies right so usually monero um and and they they basically did that and and thousands like got a very expensive bill on their cloud account right um okay so that's one of the things that

we can do uh another tool that's interesting to to use is cube hunter cube hunter hunts for security weaknesses in kubernetes cluster was developed to increase the awareness and visibility for security issues in kubernetes environments right and it currently detects uh 37 vulnerabilities on a kubernetes cluster i think it's developed in python and you can it's open source right you can contribute to the project as well um okay so i'm inside the cluster i got a shell uh inside a pod there what what can i do right so um one of other things that you can do it's an internal uh we call internal recon right so okay i'm inside the pond how do i know that

i'm inside a container uh i'm inside a kubernetes environment if i didn't have the the the proper uh url that would flag that to me right so uh one of the first things that you can check are uh um is the environment variables right so by typing that command there you can see if there are environment variables related to kubernetes so that might show the kubernetes api server the ports and everything right so you can see that from inside the pod you also have uh the service account token which every pod on the cluster has right which they use this uh token that's inside that uh that directory there to communicate with the api server

right so to reform the authentication and communicate to the api server so you can use that to impersonate uh uh the pod and and make requests to the api server as well so uh um a kind of authenticated request so you might have more uh more permissions and get more information out of the api server right another two is the mi container 2 which is called a container introspection 2. you can find out what container runtime is being used as well as features available there so that's also gonna show uh uh if you're running inside a kubernetes cluster or in a docker environment as well right uh other things that you can do right that basically you should usually

the the name of the the hostname of the of the pod or the container are specific or they have some uh uh suffixes at the end that might might give you give away that uh that's running in a in a container environment containerized uh environment right and and other things are um [Music] are the the number of processes right that that's running also in the environment you can check usually the container has a few processes usually uh uh one right so and that's not the case for a virtual machine or a bare metal environment right so okay i got the the the i did my internal recon i found some information there i identified that i have permissions to

um to exploit this this this cluster i have ex permissions to create a new pod on this cluster right that that's great right so now we go to the privilege escalation phase right and and one way to do that is by performing like a pod or container escape via a privilege pod right so here's a technique posted by duffy cooley uh from vmware on 2019 where we're we're creating uh um a pod a privilege pod here but it has some override or arguments as you can see there has a json uh to uh overload the spec with uh partial set of values parts from the json right so just a quick overview here on the

right side i have the json uh formatted better for for visualization um so basically first here we see the the host pid set to true right so which breaks the the most fundamental isolation of containers letting us see all the processes as if we're there if as if we were on the host right um we we have uh the end center here command which um which is used to uh to switch to different mount namespace right and and since that's uh guaranteed to to be the host mountain native space right um and the privileged security context here at the end uh it's necessary to prevent any permission errors when accessing that um that path here on the mount

right so this was also used on a workshop from uh from kubecon 2019 uh uh the source is uh secure kubernetes.com so uh just make sure you check that out that's very interesting uh uh learning about kubernetes attacks there okay so let's go to the defense part right so um defending kubernetes like how how can i protect my cluster uh from attackers uh isn't kubernetes secured by default right so where do i start so the first thing here is protecting your your api server right so make sure that's not exposed to to the internet and accessible by attackers right so if i get uh if i just make a call request here to my api server i got this

response although that gate gives me a 403 like a forbidden uh since i don't have permissions i didn't pass any any uh authentication tokens right this already shows me uh that this is a kubernetes cluster that's running there right so this is the default one uh um so and so if someone can reach reach out to your cluster like that that's a that's already an issue right so uh that that should be limited uh uh through uh either like um a vpn or firewalls or security groups that only allow your your developers and your uh um your employees from your organization to access that cluster even make make this simple request right okay so one of the first things that he can

do to protect your cluster is also checking the cis kubernetes benchmark right so the kubernetes benchmark is a prescriptive guidance for establishing a secure configuration posture on a for kubernetes right it has over 120 security checks for your cluster right um it was created by rory mccoon and release rice and many other contributors as well and uh there are specific ones for eks and gke so uh there's a lot of information there it's currently on version 1.6.0 which was released on july 2020 and it covers uh kubernetes cluster versions from 116 to 118. okay but how i'm going to do that like manually right so if i get if i if you take a look at the

benchmark it's a lot of pages a lot of checks that you have to perform and um and it's hard especially if you have multiple clusters right on your environment that's that's going to be hard for you to check one by one right so um there is this open source two that was released by aqua security as well called cubebench uh they basically check whether kubernetes is deployed security uh validating it against the the cis kubernetes benchmark um it's developed in go and it basically checks all the the the items from the benchmark and gives you a pass fail or or warning response so as you can see here uh this is just one of the results and you can see

that my cluster is not very well configured with a lot of uh issues there or it wasn't able to detect that if it doesn't have the permissions but you can see that the master node it checks the the api server at cd the worker nodes and and other configurations as well so i recommend you to take a look okay one of the other things that you can do right before you deploy your uh um your cluster and and your container images right is do uh what we called image scanning right uh there are many tools out there to perform that for you um so basically you check your your container image for uh known vulnerabilities malware

uh misconfiguration issues and and other and other set of like uh permission issues that may cause problem if your container is compromised um some of the tools here that you can use are clear smart check sneak and trivi right and also there is a now there's a scan command that you can basically run if you're using docker uh you can run the docker scan command which uh under uh the hood would uses the sneak database to check for uh vulnerabilities and and on your container image and dependencies there as well so i recommend you to check that out okay uh you check the images you scan them they look fine you did the the cis kubernetes benchmark

everything looks good and but now you've deployed your cluster your cluster running and it's string production right so how do you know that your containers are not uh have not been compromised and are not being doing uh something funny right uh uh like uh having an open as a sage running there or or or executing some files or uh changing the state of the the the the container image after it is deployed right so which something that's a technique that's called drift detection right so choose this tool called uh falco which was developed by sysdig in 2016 and now belongs to the kubernetes uh the container cloud native computing foundation right uh which is basically a cloud

native runtime security project right is these or as you can as they call it the the de facto kubernetes uh threat detection engine right uh it's uh an incubation level project it detects uh unexpected unexpected application behavior and alerts uh on threads at runtime right so basically it parts its parses linux uh kernel c skulls at runtime and and based on on some rules engine easy like uh uh yamo uh file rules engine that's easy to read and and to create uh that ruse engine uh detects the unexpected behavior in your cluster and generate alert space on these threads and and you can integrate that with other uh other tools as well so i recommend that you check that out

okay that's good and and what are the things right what are some basic stuff that you can also do on your cluster without needing uh um third-party tools to protect yourself right so uh basic basically uh you have the pods right as i said pods is the the smallest deployment unit on the kubernetes cluster right and and and so um just like containers you have to limit the resources the cpu and memory otherwise uh um otherwise it might the the pods can start consuming all the resources available on your node and and leaving the other pods with performance issues right and pos possibly causing a denial of service on the node so the way to

to fix that is either setting the uh limiting resources the cpu and memory sources specifically on the on the on the container on the pods back right on the yemo file or you can use this resource quota that applies that configuration to all the pods on your cluster okay one other thing is creating and applying a security context right so you can uh there are some uh privilege and access control permissions that you can set on a pod uh here just a few examples that you should uh enable that on all your containers to avoid any uh privilege escalation issues uh any any uh any issues with writing to the root file system or running at the containers root

right which is also a big issue there um also there are some linux kernel security features that you can enable some of them here are second app armor and selinux that also protects your your uh your container and your cluster against some attacks right okay um moving on um there was uh uh there is a a uh something called uh uh admission controller which is a piece of code that intercepts requests to the kubernetes api server prior to the persistence of this object on that cd right but after the request is authenticated and authorized uh one of this admission controllers is called the pod security policy or psp for sure right and it basically has a uh

it can control most of the security settings mentioned before on the security context but now on a cluster level right uh unfortunately this uh psp will be deprecated on night the next version of kubernetes the 1.21 and will be removed on version 1.25 probably uh next year so what can you do there right there are some alternatives that you can use uh two other cncf tools are the oppa gatekeeper and kiverno that you can use some policy uh controller for kubernetes as as oppa gatekeeper or kiverno is the policy engine designed for kubernetes as well right that he can use um other options there are still in discussion by the the kubernetes uh sig off team

are two uh two new implementations of the pod security policy one is called the psp plus plus or the container by boundary policy and uh the bare minimum psp uh which is intended to capture more common uh uh use cases and provide the baseline security so i linked i linked those uh documents there their student discussion they are not official and they're not implemented yet but i think it's important for you to be aware since that might be an option for you to replace psp in the future okay yeah okay let's see you've secured your cluster you're protected uh uh you've implemented some of these tools already right so but now you're you're you're uh

considering considering like an insider threat right so there's some employee that has permissions to your cluster and and and they might have too much permissions and compromise or do something uh uh that's harmful harmful to your cluster either intentionally or or unintentionally right so on kubernetes you have our back which is called a road based access control it's it's basically very similar to access control on most web applications right by uh role you have like you can have the the cluster admin uh you can create new rows uh and and permissions for each of those rows right now it's enabled by default on the latest kubernetes version you can check that by checking that that setting authorization

mode on your cube api server and basically what our back does it has four different objects row row binding cluster row and cluster row binding right and and those objects basically uh contain rules that represent the set of permissions within a namespace so so row is is restricted to the namespace level right and and row binding basically binds uh uh this permissions from the row to one or more users right so a set of users and the other two objects there cluster row and cluster rule binding are basically the same thing but uh uh um they're not tied to a namespace and they will be applied to the cluster level so all the namespaces on the cluster

and and so they are more permissive so be careful with that uh xd as i mentioned before is the main data storage for your cluster right all the cluster objects are saved here and and and and i we've identified that there are 2600 exposed cd on children today right so recently uh this is a recent search and and just uh be aware of that that if your ad cd is exposed it also has an api that you can interact with and and and if you if an attacker has access to the lcd they basically can uh do anything on your cluster right if they have right access they can create objects delete pods or or anything else right because

kubernetes is always checking at cd to maintain that uh a desired state which reflects what's an lcd should reflect what's in the cluster right so two quick things here about lcd basically the uh uh encryption in transit is done by default you can check that by running this command here and checking those files the certification file and the key file right but the encryption at rest is not by default right so it's not done by default everything in that cds is started by blend in plain text including your secrets right which is an object that is usually used by kubernetes to store a sensitive information right so to fix that you have to create an encryption configuration

object that uh you select what should be encrypted on your ad cd and that will be encrypted at rest as well but that's not done by default so be aware of that and and for network policy uh also by default all pods can communicate with any other pod in the cluster so basically that's a flat network and and if you uh uh if you have a network background you know that's not a good thing right so make sure that you create a proper network policy for your cluster right so like ask questions like does the front end really need to talk to the database pawn right so what if an attacker can access the pods on the cube system namespace

which is the main namespace for the kubernetes cluster where where the um the control plane pods are are usually uh running right so um psyllium which is a a a network uh a cni for kubernetes right network interface uh we just released recently a network policy editor and that's a very nice that you can visualize uh who can access uh what on your on your cluster and then it's they already give you uh uh the code the emo file uh for the configuration that you set there for your network policy right so that's nice to check it out the link is there in the slides and just to finalize here uh another thing that's important for

kubernetes to uh kubernetes security for protecting your cluster right is the kubernetes audit logs they are not enabled by default right so it's highly recommended that you enable them for security and also for troubleshooting right you basically need two things uh uh for enabling them a log path and a policy file right so specify where where the the the location of the log and which policy so what you wanna log so there are some policies you can log just the request or the request and response or just the metadata right and you set those up on the cube api uh server configuration right if you're running on a managed services basically for example an eks is just a

click of a button to enable those logs on aws and they go to uh cloud watch and to finalize here the basic stuff of of protecting your cluster right um so don't forget about those those are also important so update your kubernetes environment version early and often right the latest version is 120 and and kubernetes usually um updates their uh their version at like at least two to three times a year so almost every quarter so be aware of that uh don't use cluster admin to uh for daily work so treat it like a root uh user on your system right if you can use the many manage kubernetes service like aks from azure eks from aws or gke from gcp

because they handle the master node for you so you don't need to worry about the security of the control plane and cube api server and all that stuff just be aware of their default settings for example like eks which has the the the cube api server uh url exposed uh publicly exposed by default and check the cis kubernetes benchmark document for more secure best practices uh they're always updating that to cover the latest version of kubernetes so i think it's important to take a look and with that i conclude my presentation and i'm gonna be uh answering questions on the chat okay thank you you