How to Lose a Container in 10 Minutes

Sarah young how to lose a container in ten minutes so hello first start I need to say this IMAX screen is freaking me out and how do I know well I did know it was this big but if I'd had known that my really bad Photoshop would have been blown up that much I might have made more effort anyway hello my name is Sarah young I work for this small startup that you may have heard of I don't know called Microsoft it's also a giveaway this t-shirt that I'm wearing I am a cloud security architect or specialist whatever you want to call it I spend I'm actually based in Australia I'll talk about that more in a second

but I spend a lot of time outside Australia and when people see me they're normally like hey where do you live again because I never see you in the country you will leged Lee live in it's true I don't spend a lot of time there I only joined Microsoft quite recently and I'm very excited that this is the first time I've given this talk and I get to add in all the bit drawings cuz they're adorable so there you go this is my full job title Azure cloud security and compliance global black belts yeah I know I'm not joking this is genuinely my job title so I tell people who that you know I fight hackers like literally with

the black belt there that's not true but yeah basically what my job is is I advise customers for Microsoft about cloud security and how to do security in the cloud well and hopefully if you're a company that cares about compliance which lots of companies do nowadays that's how you have to do that better as well just a quick note word of caution please google my name with caution I have a very common last name and a very common first name if you google me you may find seryeong the Christian author I saw her book in Hawaii last year and I didn't buy it because it was thirty dollars at CVS but I took my photo with

it I'm cool also the one that I can't put a picture up of put put a picture up is I'm sorry the late 80s porn star now I'm not joking so don't go and google that and think I'm joking and do it on your work machine because it is true just in case you don't know where Melbourne is it's down here at the bottom I always like to think I've come further than everybody else when I go to conferences I try and do that just because I like that strange badge of honor I'm not sure if anyone's come further than me for b-side San Francisco they may have done I'm not sure but yeah that's where Melbourne is

if you don't know and because I get asked this a lot when I'm in North America I'm just gonna cover this one off now this will kill you in Australia this will also kill you a funnel-web spider will kill you jellyfish will kill you the crocodiles will kill you and this really really ripped kangaroo would definitely kill you that's real as well that's not photoshopped he's called Roger and he died about a month ago and it made the headlines in Australia of course yeah I know but big male red kangaroos are really ripped like scary ripped it's weird and also cockatoos drink beer or whatever I don't know so the point is just in summary everything in Australia

is trying to kill you I know is it's dangerous to live in Australia no because if you live in a city you'll literally never see anything like this but if you want to come and talk to me about it later please do anyway I think I've wasted about four minutes of my time by just talking about Australia and where I come from and so what am I actually going to talk about today container security so you know containers orchestrators in particular Cuban Eddie's you know all the thing of the moment so I'm going to talk about good practices for can secure in your containers and for securing your Orchestrator now when I'm talking about orchestrators I'm gonna mostly talk

about Cuban edits because in my opinion that's kind of becoming the de facto standard now I know there are other orchestrators out there and if you want to come and talk to me about other orchestrators I have my own rants about them and a lot of the things I am going to say are applied to any other orchestrators but I will keep saying Cuban Eddie's also a lot of things that go hand in hand with this another thing is moving to the cloud now it doesn't matter which cloud you're moving to I'm not here to push any cloud in particular but often when you're moving an application into the cloud you'll be container izing it so

these things kind of go together they're not mutually exclusive but I'll be talking a lot about that too so yeah we're talking about good security practices for containers kubernetes and related tools now I do use a lot of gifts or gifs in my presentations they may or may not be relevant if they're really weird and irrelevant I will try and explain them if you don't get my sense of humor but I've broken this down into five bits which is protecting your data caring for your OS and your orchestrators checking privilege shifting life of containers and a little experiment I did which was where the title of this talk came from which was getting my containers owned now I think

we probably most people in here will have guessed where the title of my talk comes from it comes from the rom-com from years ago How to Lose a Guy in 10 days now if you haven't seen that film whatsit it's about a lady called Andy or Kate Hudson who decided she'll make all the common dating mistakes that women make to try and drive off a guy in 30 days so I thought I'm gonna make all the common mistakes people make when deploying containers security wise and see if I can get my container owned in 10 minutes now when I submitted to the CFP for this it was in the very very early stages of the experiment so in the

we'll come back to this but I'm just caveat with we're not going to log into any live containers there's this afternoon because it didn't work out quite as planned and so there you go and before I go any further by the way did anyone literally come to this talk because I promised chocolate on Twitter by the way I'm not wearing my glasses because I left my contact lenses at home and I'm just being vain so you have to wave if you're any further back than like the first three rows like you really need to wave a lot for me to be able to see you because I'll throw you chocolate right now you can be honest

oh I if you're supposed to be at the front you can come down and get it later if you want because I do genuinely have chocolate here that's shaped like a koala anyway so this thumbs up container and Cuban Eddie's security for me not monkey emojis but as most of you probably know this is monkey see no evil hear no evil speak no evil when people are moving into the cloud and container izing applications they just seem to think with security well we already have some security practices we already have we already have some tools we have some processes we can just apply exactly the same thing into containers and lalalala it's fine we don't need to think about

it no vulnerabilities I see nothing I hear nothing it's fine anybody relate to this if they've done something like that in the organization again you really need to wave quite a lot for me to see you okay whatever so let's moving on to our first one which is good data protection practices now in the words of Mary Poppins this isn't actually a Mary Poppins quote originally but it's the only place I know it from well we're gone is half done now when this is not contained as specific and I'm going to now espouse you what every single security person will have said to developers time and time and time again which is please tidy up your application but if you're

actually moving to the cloud and container izing things this is the perfect opportunity to do it like if you if you've got a rubbish your old application they're still using telnet that's using deprecated protocols stop tidy it up some of these changes are five minutes I seriously seriously in a previous job had some developers come to me with an application they wanted to put into the cloud and it still had telnet on it and they're like is this fine and I'm like no this is not fine and it's like a five minute change it wasn't as if they can the application didn't support something different so actually go and tidy things up everything should be encrypted at rest

everything should be encrypted in transit if you have a really old application that uses some weird crazy deprecated protocol that can't be encrypted really you should be putting it in some kind of VPN now I know that's a little bit more complicated but please try and do it and I know that when we you'll have a limited project scope you'll have limited budgets but some of these are really quick and easy and as we all know with security hygiene is literally the most important thing it's all I'm going to go on about for the next 20 minutes and next thing is isolation this is a quote from Jimmy Page I have absolutely no idea what he's referring to but it

was the only vaguely famous person quote about isolation I could find so when we move to the cloud and when we containerize we isolate things does everybody know zero trust I'm in San Francisco so I expect everyone should like one person sorry if there's a two people thanks for waving a lot three I can count as well so when we move to the cloud we should be using a zero trust model and if you don't know what that means what it means is that instead of having your traditional security castle I'm doing this in the wrong order I'm gonna click through to another slide instead of using the traditional castle security model where we have a moat and

big walls and impenetrable perimeter but then as soon as it's breached it's a free-for-all instead of doing that going back to my other slide we isolate things and that means that every every instance every machine every part in the cloud doesn't inherently trust and we have to explicitly allow it the reason we do this is so that if someone breached an instance a machine in the cloud or one of your containers it would they would they would hopefully hopefully find it difficult to move laterally and coming on to the next bit of moving into the cloud I'm only gonna touch on this really briefly but has anyone heard of shared responsibility model and it doesn't matter what crowd provider you

use yeah yeah thanks for lots of very enthusiastic waving thank you so responsibility model if you're not familiar with this every single cloud provider ever will be talking about this it is who does what in the cloud and which security controls you're responsible for and I swear to God if this one thing if you move to the cloud you need to and it's this because it's whether it's Google ad whereas Microsoft whoever doesn't matter it's which security controls are you responsible for and which ones are they responsible for and it differs depending on the kind of model you're using so look at it in terms of different homo mentorships so SAS software as a service that's like a

hotel room basically almost everything's done for you you pretty much just turn up and use it platform as a service like a furnished apartment so you have to do a bit of cleaning they only come and service it like once or twice a week and I asked infrastructure as a service that's a rental so you're pretty much looking after it you just kind of pay for the the shell I don't maybe not shell you actually just pay for the apartment and then private cloud you run everything cuz it's your house hopefully that makes sense there's loads of different analogies there's also things like pizza as a service that one's cool as well but I'll go with that one

because I could find a nice picture so I'm going to click through castles so caring for your OS and your Orchestrator now if there's one thing literally one thing I know I just said that about something else as well but if there is one thing you really should do with container security it is this which is control the images control the images you're using to build your containers I don't care what OS you're using if it's Alpine if you're just using a straight-up WordPress or I don't care what it is but make sure it comes from a decent source so of course when you're first building your container repository you're going to have to pull something

from the internet but even official repositories can have problems last year there were compromised images on the official docker hub and and it they had a Bitcoin miner or something in it the usual kind of compromised thing but it had been downloaded something like sixty thousand times before they found it so that's at least sixty thousand containers running if not more with a Bitcoin miner end so you shouldn't just be pulling from random things on the Internet pull pull an image from the internet make it your base image secure it run checkmarks across it maybe put your security tools on whatever and then store it in a private image repository now if you're in the cloud all the major

cloud providers have them and they're free so use them they're not always for some of them are free and all the third-party ones if you have a problem with using with using cloud ones because what you don't want is like some kind of surprise attack and it turns out there's yeah I know can't give like when you can't think of anything else surprise a tap tap gift someone probably they found my upsetting a few days ago and then I felt a bit awkward I just thought it was funny but anyway I hope no one finds that upsetting the image but yeah you know if you've downloaded malware or a Bitcoin miner or whatever it is into

your container from the word go I mean it's game over like there's no point doing anything else so please do this next is the fault in default I wanted to put the fault now stars but then I realized that was a movie about like someone who is terminally ill and it really didn't work so I kept it like this so again I'm talking about kubernetes Kevin Eddy's default config aren't really secure no it does depend on the version you're using the most recent version I believe is version 13 which isn't bad but I'm not going to assume that if you're in an enterprise environment and you're already using kubernetes that you're on version 13 I

think you're probably not yet so you need to work for your Orchestrator config to secure them some notable bodies in the older versions of kubernetes are the API server listening on port 8080 and having no authentication whatsoever that's great and secrets management is done a net CD more about that later and when I say it's done in at CD I mean it's encoded in base64 a net CD yeah now um Seth our CIS is the center net for internet security they've recently released a Cuban Eddie's benchmark so if you're not too familiar with Cuban Eddie's you're just introducing it to your organization you're not you know super skilled in it yeah and there's no shame in that because it's still pretty

new technology there's a benchmark where you can literally just go through it and it will give you a tick list of all the things you need to secure to get rid of you know just some gotchas and some basic ones because you know this these kind of defaults are pretty scary first time Aaron did this talk was in Seattle so I found a Fraser reference even though I know that Fraser is not actually film was not in Seattle but I love it and if it was old a fan you'll know it's a secret to everybody I'm a big solder phone don't fake creds and secrets into containers please please please please don't do that put

them in as variables as I said before Cuban Eddie's stores secrets in ED CD encoded in base64 and this meme is a bit old now but yeah I really like that song as we know basically for is not encryption this is not secure please don't do this again all major cloud providers have their own built-in secrets management that you can use if you're in the cloud no matter what cloud you're using I would recommend that you do just use the native cloud secrets management because it's easier and it will be more compatible I know there are specific use cases where that may not be may not work properly I know of some I've seen some

personally where the certificate length and the key length doesn't work properly and if that's the case then there are third-party ones out there and something that doesn't get mentioned enough this is kind of a bit of an add-on at the end when you're in if you're doing things in the cloud please rotate keys regularly this doesn't contain a specific but it seems to be something that I don't think is mentioned enough you should be rotating your keys now get it done automated do it in a function of some sort or a lambda job you know depending on which cloud you're in but you shouldn't have the same keys in the cloud indefinitely encrypting things now

how often you want to do that I'm not going to say it might be a month it might be every six months it might be a year that's kind of a risk based discussion that you can pawn off on risk people but please do it it's really really important and if you're automating it what's the problem so now I'm going to tell you my Horror Story number one I have three of them they are extremely vague and for that I apologize but they are from previous jobs and I'm not allowed to give you lots of detail but a dev needed a slightly different image for his container he pulled it from our public repository anyone want

to guess what happened you can shout yes pick my minor if that is exactly what happened now it was picked up relatively quickly I would throw chocolate at you by the way but I can't see you oh I can't see you just about you want a chocolate I can try I'm gonna do it anyway sorry that went bad okay I'm gonna stop that you can get one later come and get one later so check your priveledge again I'm not saying anything here that is mind blowing li knew I just wanted to give a nice overview here sorry everyone's laughing now because I threw chocolates I don't run his route don't run don't run his route I couldn't find anything

so I did Groot because I'm funny so I know there are some very specific instances where you do need to run as root if the container needs to modify the host system but if that is not the case you should not be running as root but you really shouldn't if you do need to do it then you should be using some kind of runtime security so there's loads of stuff out there SELinux a farmer SATCOM profiles whatever use as many as you want to do I know that some of these particularly SELinux has a bit of a rep for being a complete pain in the ass to configure but if you haven't tried it in the last few years I would

recommend trying it again because it's actually got a lot easier and this runtime security whilst you're putting it on the containers that run is real just put it on the rest of them as well just do it I know everyone cringes a application whitelisting or run time security but it's really it's a good practice and it's a bit of pain but go for it so checking your privilege for orchestrators so I've got a picture Clippy because I literally could not find an interesting picture for this slide and you know I work for Microsoft and Clippy so because you know we all love him I have Clippy stickers at the end if anyone wants them by the way so

kubernetes yeah I know I haven't got that many so like feel free to rush me a few people at the front here you're sorted so as I said before kubernetes has got some terrible defaults depending on which version you're on like I said the newer versions are getting better but in an enterprise environment I doubt everyone's running the latest version yeah the dashboard had full admin privileges prior to version 1.7 and there was literally no our back in it before version 1.8 sure only got our back in version two which is very very new just to throw in a different Orchestrator there now if it sounds to ownerís to go through orchestrators and change these defaults

yourself or if you don't feel confident there's no shame in saying hey let's have a managed cuba Nettie's cluster again all the cloud providers do them they will do a lot of the security for you now it won't be appropriate for every single application that you've got or whatever but you know keep going with it it's there's no shame in saying hey right now we don't have the tools to do this lets you or we don't have the skills let's use something managed no shame promise Horror Story number two kubernetes cluster it was a test one was left exposed to the internet with bad defaults and anybody want to guess what happened I won't throw choclate this

time don't be scared yes I know I know I know I know it was picked up relatively quickly and I don't I know there are other threats to containers out there but Bitcoin miner seems to be the thing de jour so there you go so containing your enthusiasm for shifting left as I said towards the beginning I couldn't think of much for this so don't assume that your own tools your old security tools that you using your traditional application stock are okay because they're probably not they'll at least need an extra plug in or you may need to go from strap scratch and when I talk about security tools I really mean your IDs your heuristics

Vaughn scanning you'll seem I say seem and I know over here everyone says like sim or something I can't say that properly you know what I mean because I've got it written down sorry about my accent run time security and auditing like whatever tools you have make sure they're actually gonna work if your container izing things because there's a good chance you need to tweak them if not replace them same goes for your CI CD pipeline now if you're actually moving into the cloud it may be that you've never had a CI CD pipeline before which makes it really easy because you're going from scratch anyway but bear that in mind you may need to do

some research and talking about research actually please go and do some research they go and benchmark things don't just read the bullet points on the website and just go buy something because I've seen that happen it doesn't work and you should be doing this with the devs and with the security team because how many times have you seen one team pick a tool that another team is going to use but the other team had no say in it and it doesn't work properly or they don't use it properly this is what we want we're all shifting left we're all going to be friends we want action pikachu love you also do love pokemon so this is what we

want we want security in devs working together and I know it sounds cheesy but actually you do get an awful lot of benefit to having both teams agree that this is a tool that's useful for them last but not least Horror Story number three is I saw an organization try to use a traditional vulnerability scanner on the containerized environment they were telling me how it was brilliant they have no vulnerabilities at all yeah because it couldn't see anything this is I had a trump sort of shaking heads this is what I had originally for this slide but then well like a month six weeks ago with the government shutdown I thought my fail picture this was even better so

you know yeah because yeah I'm finally moving on to my little experiment and as it said in my CFP submission for if there's anyone here in here who read it or it may have been in the blurb as well I decided that I would try and see if I could get myself owned and how I did this was spinning up containers Cuban Eddie Cicotte clusters and leaving them open to the internet for a few months and I did it on a unnamed cloud hosting provider it's not my employer but I wasn't brave enough to have it on any infrastructure I own and I wanted one who would support PayPal payments because I didn't want to put a credit

card on it to be frank surveys that I try and try to do the opposite of everything I've done here now my hypothesis was that I could definitely get a container owned in about 10 or 15 minutes because you know hackers buy I was pretty sure like yes 10 minutes someone will definitely own it and anyone want to guess like what containers I thought might be particularly vulnerable that I spun up that would be particularly vulnerable to being owned any guesses sorry WordPress yes and Ubuntu 14.4 14.04 because that one's full of holes um actually I I've done this so many times I've actually spent quite a lot of money on it as much as I was willing to spend anyway so what

actually happened actually not very much I have loads of containers cubed Eddie's classes open for the internet and no one owned them I'm really sad and I realize this is kind of semi giving you the wrong message now when I've just been telling you how important it is to secure your containers I don't know it could be that the car provider I was using they had a lot of tools that I wasn't able to see or configure that was blocking it and protecting it I don't know but a pen tester who's based in Australia called Atticus she has come over to the states before and talked she's a very good public speaker if you ever get the chance to see her she

actually tweeted this a while back in January saying that she couldn't get it took a while for her to get a wordpress container own - and I was like cool if Atticus can't do it I'm good so I did though have to go and reverse engineer a cloud platform as much as I could to actually get my container owned which was interesting and I use a tool called piece art or it's PS ad I don't know how you say it it's an open-source tool that looks at the IP tables of content of Ubuntu and Linux stuff in order to c4 to scan for things that are happening and it's a bit like a very basic IDs the

most interesting thing I found about this is these who is lookups and how people apparently use these for adverts so we had please know that scénic is not an isp black host I can't I have to read this like this specially crafted and optimized for the bun with hungry applications I didn't know who is was now being used as adverts and my favorite one though the good old one from China is security trouble hmm yeah interesting in conclusion tidy up for application before you do cloud migration and all containerization you're probably doing them together but if you're not tidy application Orchestrator defaults a terrible so change them go and use guides if you can't use guides think about using

something managed please please please please please make sure you know where container images come from don't run as root that's obvious unless you really need to keep your secret secret shift left but make sure you've got the right tools to support it or you've got the right resource to go and benchmark things and purposely trying to get containers hacked is apparently harder than one would expect but don't have that as your final takeaway despite the fact I just put that it's nearly my last slide it's very it's it does happen it is real so yeah I don't know what I was doing wrong or apparently anonymous cloud hosting providers have a lot more in bill protection than I realized

here are some useful links some of the stuff I've been talking about today the siskiyou Nettie's benchmark the sis docker security benchmark the nist special publication 800-53

liz rice and michael harrison blasts wrote a kubernetes security book that you can get online as a PDF for free it's pretty up-to-date and it goes step by step to securing a kubernetes cluster so have a look at that obviously Microsoft have got some stuff on Azure containers the container and orchestrated security and it's very generic so it really a most a bit applies to everything not just Azure and PS ad or piece art however you say it that tool is there it's done by a chap called Michael rash and I think I'm pretty much just up for time but we might have time for like a question or so and yeah that's me thank you very

much

any questions I'm putting my glasses on that way so I can actually see people

when you say not to run as route do you mean is there any information you could give us about like container route versus overall route on the orchestrator or anything like that

it depends it's really contact specific like there are instant there are times when you will need to run its route but when you when you say run as route do you mean like obviously you've got the host system and then you have the container do you mean the container running his route because that's what I was referring to really well I know in darker there's the concept of like mapping route to your actual route user on the you know exactly and so I'm wondering in kubernetes are you talking about when you say don't run his route are you talking about don't rut like set up kubernetes to run in a different way that doesn't map the

route user or are you talking a little bit different really kind of talking about containers they're the actual container sitting on the host communities as runs in a slightly different way you've got like namespaces and stuff but there's a big debate around whether that's precisely the same or not we probably don't want to go that any more questions

oh thanks so much for you talk um so one hypothesis I have for why it took so long and no one attacked your container was because you know the leet hackers probably think it's a honeypot given that you set up that more than 10 Boehner villas some of them were ridiculously easy like I had a wordpress that wasn't even stuff and then I did want to admin admin you know like it was a little bit obvious to anyone who maybe wasn't a hacker so I'm wondering kind of based on your experience what are your techniques to determine if something is a honeypot versus if it's actually kind of a real vulnerability that's a good question I mean what you've just said

anything that looks too good to be true attack target why is probably is but the thing is there are also people who are just lazy or they've done like they've spun up a test instance and forgotten about it so I think it's really really difficult to tell sometimes like I always tend to err on the side of if it's too good to be true it probably is but honey pots are getting much more advanced as well like they genuinely look like working proper parts you know in like machines whatever they are whatever they're trying to be so to be honest with you it's I think for me I just kind of guess it's educated guess

because you I you would say in equal parts you could have someone who's just messed up and forgotten about something or made a silly mistake or it's a far too good to be true honeypot I don't think there's a real definitive way if anyone has on tell me because I want to know question [Music] so how did you know that they weren't owned I don't for sure like from what I can tell they were not owned like I was also I didn't put it up on here but that cloud hosting provider had some other stuff on that GUI that you could have a look at in terms of like the different kernel calls and unlike the CPU activity

etc like I'm not saying that I am perfect enough to say they weren't owned but I am reasonably sure they were not but like with everything in security I'm not a hundred percent it's just my my theory also my building go up enough I feel like my bill would have gone up because it not always but a lot of it is Bitcoin mining like that seems to be the easy one that's the most common and I was thinking that was probably what I would get but hey who knows maybe they are so small and so clever that I don't even know they've owned me and now I'm just going around the world doing a presentation about how I didn't get

owned maybe who knows

anyone all right thank you thank you [Applause]