BSIDES Cape Town 2018 - Lightning talks - Anomaly detection in container sandboxing - Christo Goosen

I was really hoping Barry was gonna stay outside but let's do this so a very long sentence and I usually explain things like this let's see how it goes I also haven't slept and this has been a long day but it's somewhat based on what I'm gonna research for my masters this year at Rhodes and it's partly around something called that and that so who am i I've got a little figurine now a lot of people a lot of people call me goose so call me goose in the day I work in insure tech startup it hasn't launched yet I did production infrastructure yesterday before b-sides so let's hope that works I'm also part of a wasp Cape

Town and besides he probably saw me around today I wore this crazy shirt so everyone could spot me if there was a problem so or it's payback for me calling someone the dev ops Casper the 3sat dev upstairs so it's all pretty much around sandboxing so we've seen in late if ever we had a question with or not docker was for security we've got a couple of answers the last couple of years here's some examples when Aero miners running in docker containers I literally talk to someone at devops days who said he knows someone who runs a kubernetes cluster with 15% of the resources running Manero and no one's ever noticed that I don't know how that

works in your kpi's when you're doing your job review but hey so we still have a lot of problems as running code in docker people actually download my sequel that's not the official one run all kinds of crap and generally what we used to do is run a whole VM and abstract everything in the cloud on our own machines in a data center but then we realized that it's somewhat inefficient and we want to squeeze more out of our applications as we see with Intel speed is everything and then we started looking like this so much easier to get to the kernel all the applications are just side by side horizontally so we kind of see this kind

of thing way more more people are looking to either hit the middle way so like the docker daemon itself by TCP finding vulnerabilities and Girling itself docker or the application and this is especially relevant in multi-tenancy environments that we seeing popping up a lot like this one that beautiful word we all love serverless which just means actually something like this patchy as a project called open whisk and it gives you idea of what it would actually mean to build server lists is you schedule things to run and a whole bunch of stuff in the background like console and Kafka make it run on docker hosts you can spin up multiple machines multiple darker instances and fill up as

much as you want until you run out of resources so what we tend to see especially in AWS is you'll end up with your application in someone else's application running side by side which is a new and interesting problem for us to solve so into G visor and Gophers because it's DevOps and everyone loves going so G visor is a statically compiled golang binary which you can add into your docker settings so in the daemon starts it has a additional runtime that adds a shim in between the application the host kernel so a little VM a mini VM so we're back to wasting some resources but we're protecting the host kernel by limiting the syscalls and also

networking calls so part of this would actually be protecting the docker daemon itself it has two little processes that run and it proxies Siskel's down to the house kernel it has limited system calls I'll get to that later and it also limits the networking and you're actually able to set it to debug and see the actual calls going on and how I came across this in the first place is that Google released Python 3.7 for Google Cloud recently and they both G visor to help them actually isolate the host kernel obviously Python doesn't have its own sandbox so they wanted to send box Python code or any code really and then they released this with the Python 3.7

news and it implements about 200 of the 400 x86 Linux of schools or what I mean is they focused on x86 it's obviously for servers acts as a file system proxy it breaks things up into separate namespaces it has its own network stack written and go and it also actually allows network pass-throughs if you want to go past all their implementation and adding it to docker is pretty much as simple as adding a runtime and then when you run the docker container telling it which runtime to use and it's a single binary like I've said just make sure that you've got the log directories and the write permissions where to run so what what's great about this is they've

actually made it super easy even if you have to tell your gloss I developers that they have to do one more extra thing it's not too hard to implement that why am I actually talking about this is that we have new things like kubernetes and you could run multi-tenancy on that so they're busy building more of G Weiser to actually support cabinet es so their own version of it already protects the stack but they still some limitations so it is a work in progress and part of why this is a problem and needs to be solved as much of the service side of things with kubernetes we we want to protect pods and also protect the overall system from being

attacked by a malicious container especially if you look at this it kind of explains quite well the general thought you've got a API server a key value store scheduler and a controller and at the end of user so you might have multiple users running their own containers across multiple machine so you kind of need to start thinking about the security especially if it's multi-tenancy so to some extent what I want to look at is applying sandboxing and monitoring within a container and then look at kubernetes and applying some of the zero trust concepts to locking down a user and certain containers with anomalies so kind of just using the idea of an access control engine and access proxy in beyond cope

and start limiting the the container and the user based on inference or anomaly detection so I mean we're running untrusted code so we already in a difficult environment but we keep locking down the axis as we go so it's still kind of a work in progress on kubernetes itself but I want to focus on G visor and the container itself and then see if we can build rules and apply a kind of zero trust limitations to the user as we see more and more anomalous behavior and why I think it is possible that there's good work is that G Weiser already supports all of that so they've actually both tests and they've run all of this and

found that all the Siskel's match up you'll see Postgres equal actually isn't here because it does magic process IDs that they don't like but potentially we can use these applications and runtimes to create kind of a baseline to then find anomalies base of that and people have already started looking at sound boxing malware with G Weiser it also allows checkpoint in and restoring so that you can look at a container later on and this company started working on honey trap to build a honeypot with G Weiser and dock potentially kubernetes and that's pretty much that that felt very crazy but I hope that was interesting and part of this is so that you can come

and ask me questions or tell me if I'm crazy I think Barry already thinks I'm crazy but they'll knock a thesis out of me but I would love some insight from everybody and that's pretty much it questions just one comment on the AWS side and the multi-tenancy for the service stuff and separation is provided between customers by virtualization so it's not just multiple containers from different customers running on the same box and you never have to customers containers running on the same instance in the background that's always separated on on different instances in fact this week we've publicly released firecracker a VM system that we actually using internally which is pretty sweet pretty that anybody can actually run it yeah

it's pretty cool but this is very cool maybe to add to that initially I thought of this was to actually look at supply chain and monitor new code and dependencies in sandboxes before it actually went out so multi-tenancy i looked at because it might be easier to find data on it than supply chain which is usually quite a protected thing in a company but I have actually looked at firecracker quickly it's KVM with some shim also just to add to that um firecrackers which is recently released that ya recommended so my question is if I'm not misunderstanding something what is the advantage of gee visor over just traditional using like control groups that the kernel provides for like

kernel space isolation yeah is it I don't know is the cause of like the sauce holes oh yeah it's mostly around the Siskel's and the ability to terminate a running container within some of those Siskel's they still all very little written about it like it's an internal documentation but it's mostly around limiting Siskel's and proxying them in a way to protect the kernel but better and namespacing things so that everything's not running in the same namespace which by default a lot of people tend to do that on co-creation oh and the network side of things as well they've isolated some of it so a couple of vulnerabilities at docker with found around the TCP daemon

so you need to protect the the containers and the actual dhaka diamond and then I think a lot of its around kubernetes as well to try and protect the API as well