What Should and Shouldn't Scare You: Kubernetes and Containers

welcome everyone we have Connor here he's gonna be doing this talk on KU brunetti containers he can take it away [Music] alright so as you can see the IMAX projectors off but we'll just make you my name is Connor I am a product manager I work with a company called stack rocks and we do kubernetes container security I used to be an engineer so I like to think I know something still know I sort of stumbled into kubernetes maybe 2015 when it was nice and beta and fun and and the learning curve was still getting replication controllers and that was all sort of introduced it to one company and then about four years ago started

building security tools so now I've been kind of like seeing how kubernetes goes out in the real world now if you're coming to this talk you probably heard something about kubernetes the slide is suppose to be up is is the CN CF landscape has anybody seen that before so there's a nice tweet from Dan Khan about how is variously described as useful overwhelming and the hellscape but has like you know 6,000 github stars or something now what it is is a map of here the people in the front can see it but it's a map of all the cloud native ecosystem and the point is kubernetes is just one little tiny box and there's something like 100 200 other things on

top of it so I'm glad that you're here we have six hours the doors are locked we're gonna go over each one yeah okay so no we're not doing that what we're gonna talk about what the slides are not online yet I should have done that yeah probably not a good time to try to do that but I'm sorry that would have been a great idea anyway so what we're not going to do is go through the old landscape where we're not going to do is give a best practices talk there's a lot of you know stuff that I've I've said a lot of people things other people have said about what you should and shouldn't

do that's not this what I'm gonna talk about hopefully in about 20 minutes - or five at the start is what it means what does it mean for us whether us me instead of security obviously that security represented here but what does it really mean for us and because often we sort of debate like is this good is this bad and we don't think so much about all the nuance and you know I got the nuance beaten into me and a thesis program so I'm not going to try to inflict that on you either but I just want to go through kind of some of the details go a little bit further into you know what it all means for us so how

we're gonna do this is first we're gonna go through the building blocks of kubernetes then take a sort of deeper look at different aspects of how it actually works in practice and and what some of the implications look like and the implications that has on books a ways to work together for how to actually improve security without losing all your friends and then how to learn more so that's where I'll go and then for each of the things we're gonna look at will sort of go like is this new or not is not everything is new and then sort of is it tilted more in Devon security forget to each of that building blocks and each of the features I'm

wondering if I should try to just like get those slides online right now unfortunately I don't do link sharing but maybe I can do it I'll time box it to like 30 seconds here and we'll see if we can can get that done just copy it over I was really hoping I could show you all these animals so giant share with myself

and if this all shows up this will be fun might be able to sweet it out before we move on copy

what I am NOT in the B side slack I don't know I don't I don't have time for slack I'm sorry I have enough slack at work okay now I'm not even signed into my personal Google so I'm just gonna keep going sorry folks all right so let's go into the building box I'm glad I have this like paper I thought this was stupid but turns out where the IMAX theater is the analogue theater today so first thing right now hype isn't new we're gonna go through the new cool for dev cool for a second I kind of a shrug gun coolness I have some graphs of just like google trends of kubernetes kind of up and to the right

OpenStack a little bit waning off other sort of cloud stuff OOP Con attendees coop gun has grown to like about 12,000 people at North America 25,000 people overall so lots of people talking about it and did you know that every chick-fil-a restaurant at least as of like three years ago was running a three node cluster and they're in the restaurant so something like a six thousand no deployment federated across the United States that's kind of kind of interesting maybe the hypes maybe there's something there there but uh let's talk about some of those neat things you know various about so the first thing when you might think of kubernetes is containers right kubernetes thought I oh right in the

home page will say kubernetes is a production grade container orchestration system now let's sort of look at all of those things now if you have a container if that's your cookie your image is actually actually your cookie cutter and there's gonna be a giant docker file like 18,000 feet tall but the cool thing I mean these are sort of new I mean we had image formats before but I would argue that just the lightweight thing and the standardization of both the construction of the image and the distribution actually is is you know enough to qualify as new and also pretty cool for dev and pretty cool for security because what you've got is actually a format that describes in

detail all the things that are going to be in the file system when we start out when this container starts like what's what's exactly gonna be there all the way from slash up to whatever you've got and then also some of the important runtime stuff so like what's the default user ID that you're going to run as what kind of environment variables are you gonna how are you how do you like to be stopped what kind of signal would you like to be sent what's your default command yeah it could be really easy to just sort of docker run you know nginx or coop it'll run nginx and another really neat thing is for the last couple years at least

we've had content addressable images so we've got hashes of absolutely everything in there with a sha-256 digest and you can kind of know what's in know it's actually running it on the field and you know compare it to something you pull from the registry and if security is partly a config problem and partly a day to join problem really nice to be able to look at your configs with a shop and you know use the SHA to join all your data later so that's that's images to start then containers themselves so I paid my royalties to the shipping container industry stock photo industry because that's the metaphor that's really useful for containers is not like a lock box not like a

super-duper tamper proof thing but just like a shipping at dinner you can throw it anywhere throw no chip throw it on a truck sort of train gets where it goes that's the metaphor that I like to use now our containers knew well sort of there are a lot of existing primitives there's a Julie Evans bark comic about what containers are and in like what 15 lines of bash you can actually build your own little container you get a file system give yourself a process namespace you raise all the file system in space bid name space network name space that's kind of it you sort of just put these all together and that's pretty neat because for dev

now you've got like this nice reproducible runtime environment but security may be a little bit more mixed because this is new thing and also because of the practices that we started using so a little bit more mix for security now if I were talking about containers and security and I didn't mention escapes I would be promptly like yanked off the stage with a hook so one the thing I want to say about container escapes is you know they're not new all the technology that we use to run different apps has you know different kinds of escapes and ways to get in and out and also we tend to think of like a container escapes being CVEs

but that's not the full picture heater because they're a lot of ways to get out of a container especially if you take down most of the walls maybe you saw Frenchie and Maya talk about that there is one flag that sort of just takes out all of the containment that you might expect so you know not super new and not cool for really anybody honestly but as with any technology it's a sort of a risk that we accept for some business value now one interesting thing about using containers is that we do have the options to sort of build out different you know packaged up different isolation techniques so their sandbox is like G Weiser VM based solutions where you can

use that same container form factor but sort of run it in different way to have this better or worse isolation so that's escapes and then orchestrators so like I mentioned krit ideas kind of an orbit is called an Orchestrator what is that well once you start running your containers they start eating other stuff they need to talk to each other and they need to expose themselves to the outside world so that you can actually do something for a user you need to end up you know getting a disk or a TLS key or something an Orchestrator is sort of conduct you all need to start playing music now right and I'm conducting you but that's

where you bring orchestration in and then declarative immutable infrastructure sort of best handled with a slide but you know there and may be best handled with contrast this is a practice that we use in kubernetes where we that's not new to kubernetes I literally gave a workshop in a room called terraform on Saturday so believe me I know this is not brand new way thank you now let's let's find where we got to all right cool and so to clear it if immutable let's go back to middle school you know grammar and think about this what's the difference this sort of declarative versus imperative I just say what I'd like to say I don't go do it and then

mutable which is immutable like I don't go change what's running so basically like increment IDs you end up just saying I want this to be true and you send a yeah mole or more like you know six animals through the API server and it kind of just does it for you which is you know not a brand new concept but pretty cool and then it's neat for dev because it sort of reproduces remove this it's also cool for security because you have so much information more on that in a second so let's move a little deeper I mentioned you know you've declared of immutable infrastructure sorry this is this is a I hope nobody's up Olympic so this is actually I think

something is pretty new is having standard API is for like basically everything down from the image which is identified with that shala we just talked about to you know we're in the in the system the workloads running annotations people have given all the other things that the orchestrators doing for you that are really importantly they're given to you in a standard form and they're discoverable so that can be super helpful if you're trying to understand but like what's running and with what kind of risk context and also if you're trying to understand where you might need to go fix something and I have a little quote well I did from the creative security assessment just to prove to you that

security people don't you know miss all the value for death does kubernetes the streamline difficult tasks relating to you know maintaining and operating cluster workloads okay now one of the downsides obviously would be like the new attack surface so we've got the API this all-powerful API which has external exposure and internal we've got the new components like the couplet and the - bored the helmet iller rest in peace which are sort of definitionally new may be cool furtive its kind of neutral but not cool for security because we actually need to figure out what all these are and figure out what to do with them and just them existing is different from you knowing how to configure them

in a nice way to a buddy of mine actually still hasn't deployed our back as of when I talked to a couple months ago and suffered his broad outages sometimes because of mistaken automation and that might affect our friendship I reserve the right to use that against them so one of the other kind of one of the other things about having all this standard api is you've got granular linux permissions so like capabilities user ID what kind of file system access you might have now new primitives neutral dev maybe but there because there then this discoverable format I think that's actually a pretty big net win for security for networking we actually get some very granular controls

as well so exactly like which pods can speak to which pods which containers construct efficient containers and this level of control is almost like a whole industry in RSA some companies have flamed out you know buying all the ads at RSA just to talk about their segmentation I'm not sure that devs necessarily care but for security this can be a really powerful thing and having it all in a format that you can you know iterate on review and deploy it's pretty neat so racing against the clock with this AV now leaky abstractions another sort of boob magic can be a problem so again kind of a new thing that we have to deal with this is

not the most magic thing to be honest it's just what I'm pointing out is this annotation here where it's like oh if I just add this like sort of freeform annotation I end up really changing the security implications of exposing my service from like being world readable to actually just only being on my V PC so they're just these things where you might write a client that understands what a load balancer is and then all of a sudden you've got an annotation that kind of comes in and and messes with that and broadly we're actually you know smashing a bunch of differ thanks the other cloud Linux containers orchestration everything together and where the abstractions are that's where

the security fund can be now defaults also I mentioned some of those now great things so you run a container it actually does isn't exposed by default increment of these you've got to actually go and opt-in to that and the sort of the easy way is to be immutable to assume that there's no storage and to you know write an app that can be restarted but they're a little more like chin scratching things as well so like you'll end up running as root with like no network access control a writable root filesystem no sat-comm actually worse than dr. defaults for a variety of historical not super interesting reasons and we don't have time for the audit log

but basically talking about how these have conflicting semantics some are opt-ins some are opt-out and a lot of it's just based on backwards compatibility so are those typical well yeah actually the number of clusters I've walked into with network policies is probably on one hand and I've seen many hands worth of clusters are people better at this and you think well some are there are people like maybe people using a rail I got you man but most people aren't so sorry some examples would be expose dashboard just hanging out there some of these are old so maybe people are getting better is a Shopify bug bounty you should all read my friend whose friendship is in jeopardy and then

most containers are running as root so people started paying attention when run see had a CVE but but you know before that six months nine months ago like people mostly didn't do that now the one cool thing before we kind of zoom out again is enforcement workflows so this is an example from a blog post one of my colleagues wrote but basically the cool thing here is you can give feedback right when the thing happens using built-in primitives like a Mission Control so you might have seen the K rail talk earlier that's one option for Mission Control you can build your own you can buy a product like mine but the cool thing is that you can give that

feedback right away and keep bad stuff from coming out because the easiest but your security remediation effects is one that you catch before it happens now let's wrap up so here's let's just do an eye check but really the cool when I show you is the dev column and that SEC column there are a couple things where where they're both green right and those are things that you might actually be able to do a bit easier because we all have you know different incentives and the best organizations are the ones where people are all working together they don't feel like different teams with different incentives but if you're maybe not in that nirvana state yet maybe you can go

for those kind of you know to to credence at once might be easier to convince people to do so I think if we if we zoom out I'm trying to give you a point of view here which is the building blocks that we have or super exciting we've got jaws and images we've got standard container formats we've got orchestration with all that data available for us but we do have more work to do in the details both individually like what we can do to secure ourselves and as a community like what we can do to secure the ecosystem so one thing that I've talked about recently was you know you got to run fast to do community security and that

led someone to say well yeah run faster just run faster than the other person the bear won't get you but that's not really satisfying as an answer I think as a community we actually need to all work together such that the last person isn't actually getting eaten by the bear like let's all let's all you know get to the front of that and by putting all this stuff in declare respects and adopting these practices and shaping our orgs to match that I think we're in a great place so I think that's that's a that thing I get in the high sign there was supposed to be one slide here about ways to get started I'll post the slides

after but basically like start with the easy stuff then you can go for more self-contained changes piece by piece and then app by app even because the ways the defaults work you can actually use those against your security problems and but just keep going after that I'll stop for you thank you so much Connor thank you for being speed grab these sides SF 2020 everyone give them a round of applause you