BSides Scotland 2019: May The Setenforce Be With You! - Sanja Bonic

okay hi the morning and so this is made to set and for spirit here there's going to be a lot of references in this one and let's just get on with it who I I I'm old enough to be grumpy at new technology and I hated containers I kind of still dislike them I worked at Red Hat and that means everything's containers and for the last I was a developer in a years ago and then when I switched into database engineering I had to start working in copper and working with docker meant I had to go into what I didn't like ended up working at Red Hat now where everything is container based and we have a desktop edition of this

container if you have Optimas servers container based so I had to eventually get into that what I'd like to for you to get from this talk is to go through this demographic I do not use containers which is 46% on make 85,000 responses on stack overflow developer survey just recently and I'd like you to go into this tin graphic where do you try it for development test in production no just outside of work at least your tried it and have an educated opinion whether you like it or not and why am I here my boyfriend really wanted to go to this conference and then it might be go out of spite I put into the talk and got

accepted this is why I did this is new by the way it's very tight little discount is very nice yes and where am I going with this we're gonna talk about containers and honking vulnerabilities an exploration [Music] anyway container Compton or actually a container is just a sophisticated use of namespaces which is something in the Linux kernel and it did to isolate processes so basically you have a certain set of container kernel teachers and those are isolated they don't talk to each other or shouldn't cam but they are isolated you can isolate them and the container is basically just depth so if you want to after this talk eventually start with containers and you don't want to talk a little compartment

or any other runtimes you want to just start from scratch there are many many tutorials on the internet that you can just contain us from scratch and several really good ones there you can just have a few command line commands and you'll end up having your own containers obviously this is multiplied for production cgroups is an uncapped namespace and allocates resources so and then of course the PI system people make up a container containing content the 15 containers got info or no this is actually important a stop to get to their did site that explaining containers and abilities and and here you have on the top you have see groups and on the bottom there namespaces and

how they work together they're controlled by process speed let's escape the clear container completely inside of it it's not very complicated when you get into it it's very easy if you've been a suicide man it's just a bunch of commands to create an isolated process that can run stuff in it so and then there's container run times so you don't have to create problem scratch those are for example dr. and Padma but you can create them from scratch you can put you to do every time you use if you use the cloud you probably better advise to at some point have done system administration on your local server if then you know what's happening otherwise

if you start doing cloud stuff it's just the same if you suddenly start using each there and have never use JavaScript alone okay let's go streaking scalability who knows about the February vulnerability and containers okay who works with containers actually who has tried them okay and you add one to the means raise their hands and someone just say this talk is it interesting for you because they develop and or do you just want to see because you do want to go to day problems okay good okay so the PVD is e1c contain a breakout yeah it was discovered in February and it allows you to execute malicious code in containers it allows you to escape the container

and I has access to the whole squad system how that happens here's from the Bugzilla comm you can find other sources as well but that is a good security team and obviously a tinted agenda structure and it will have a lot about the vulnerabilities it says a blog was found in the way Ramsey handled system proud scriptures and running containers a malicious container could used to scroll to override content of run of the run see binary secondly indifferently run arbitrary commands in the container whole system how does happen is you have to download a it's basically says that it needs minimal user interaction that minimum user interactions just you download a public container image which

has malicious code in it otherwise you can column and use some X sub from the actual article on depth internet that come and it says why you should run exit in it it's set on by default in the Red Hat manage the admins don't run it to have difficulty maintained and it really is but once you get to once you realize how it's done it's not as hard it's just like a bit like doing a zone file somewhere on your DNS service it's once you learn how to do it you know how to do it's not hard after you after learning and the container vulnerability it blocked through correct use of username spaces I'll get into what that

means in a bit although the docker package when certain people take orders to their the open source where I am which is Moby engine and there's doctor doctor doctor itself it depends on how you set the namespaces or whether you're go level at all and also if you put a seven it's an important mode it's not a problem at all so while they've typed up this vulnerability it wasn't actually one it depends on which which operating system use and which which run time in terms of how the permissions were set but it wasn't just from see you can use REM pod men that were affected at all the lxd and it apache needs us so basically

everything that panel runs containers a check this were before everything that attach due to the right game spaces you executed code and then you can escape this it's hard to do and I have a link to a demo where you can try it yourself at the very end okay so this is what the road looks like a flawless family meeting empty-handed system party scripters decision to see how the actual renewability state reported and write it down this is just a screenshot at the top of the website if you get there done in about the comment it's really interesting so if you are into security these are really valuable thing the trying to figure out how it works it done infrastructure to

allow doing forensics or other in this again investigative types and using more infrastructure distance and it ready to start so in one of those commenter says this memorable is mitigated if SAS is enforcing mode and even person mode is for example a prerequisite will open should contain a platform or for the platforms that use container station then further down you see that atomic host for example is not affected by this does anyone know atomic oh okay so it's an immutable operating system which means most of the operating system is actually immutable which means you cannot write to it it's a read-only operating system we're only in this case only flash home and /var are writable

which means even in ki-joon have malicious code it's likely to not be able to do anything with it it depends on what exactly the malicious code is doing but in most cases the musical opportunity can save you from security coverage I'll explain so the terminology which is just names people made up and you have to learn it by heart and then pretend they're smart but people do just come up with things and then we learning them so none of this is none of this is something you know you just learn the concepts you learned its terms and then suddenly you know thing this is what I was always annoyed at when people start throwing around abbreviations and things

the names and then you know it's it's better to explain thank you sir so run see is just a thing it's a seal to peruse it on the command line it spawns and round OCI container OCI is open container initiative and physically are standard it's done by the lunar foundation it's a standard for creating containers and how image specification build specifications are basically now you can properly to use Padma and they all work with each other you can for example use Codman which is a container runtime without a daemon and you can use it a block requires to create and to manage containers which is really nice so yes thunders are important in this case especially so from this

vulnerability head run see to see your item is pumped in here you have X Ln X as the limit is it's an additional security control it helps you isolate the processes basically deploy containers in the host and you you set labels it's a labeling system more or less and it has policy so you start the label you'll say container underscore t and then this sets a label that's this can be called by cantina underscore runtime underscore t and otherwise container engines you only have access to them so there are policies which define which label has access to what and you have labels that basically what the FC Linux is it's just very complicated because you define it for every process so in

Apache that have HTTP HTTP underscore T and then the the runtime and you use escape so you have the kernel namespace and you have to use a namespace and the problem with a lot of vulnerabilities if things are allowed to get from the using namespace into the kernel memory that's where you get into trouble and so the way to enforce experiments on your system if you use Linux is to do second post one to check whether it is important you'll type get enforce one just getting both and whereas Jesus will show you two states up here and the Linux and we also have our active in its coloring book you can download it from the link below or just search engine it

and it very nice and it explained I said it's in a very good level but Hannity does conferences on push it they don't have anymore but yeah you can you just downloaded some if you have these for container commandos as well so they are three different types of coloring books just to learn container to learn a silliness and yeah okay so terminology wise anti Linux was the security control layer and we had rumty next is atomic host which is just one of the many mutable operating systems out there as you know right head has acquired porous for Isis contain early enough speak of mutable and he also had a comic book and cake immutable now

we're going to have the door for us and we also have relatively close real close so and then there's others there's for example endless OS which are targeting developing countries and they're helping them to give easy to make laptops with their own operating system which is NSS and then they help developing countries by making the laptop available and when you really don't have Wi-Fi for example or just internet in general you have your snapshot of your system and then whenever you get to an internet you can just update easily and the way atomic host worked with it use RPM ice-cream and illiberal h3 is kind of like gifts for operating systems you can rebase to

a new snapshot and redeploy and if something about working in the updates you just fall back so you go back and after you go back you can then when you know the update prepared go back in kit it's just like it's and branches okay so user name spaces I've tried to explain that a little bit of game spaces you have the tournament it's not even amazing and you basically make sure that access is only granted where it's necessary okay you take the waste for the container vulnerability this for security in blonde in general read the manual always read the manual and simply use a cloud if you for example when I used to set up redshift

which is a database data warehouse on AWS we always read to all the venue you just set it up and then given that this is the same where all the vulnerabilities with default passwords come from if you just set up them something called a tutorial and they actually deployed in production so always read the manual always make sure that the security policies are intense and that it is law from the table without them talking and don't forget to update your system regularly it's really important and if you have an immutable system then it is if you will practice something wrong that's that's one bonus if you don't update there was another vulnerability which was in communities

in November 2018 so just recently it well who knows about this one okay oh you can easily be viewed by malicious attacks and you don't even know about them so this this special vulnerability for example you can't believe you forensics on it this is from the actual issue on github link on the bottom right there is no simple way to detect whether it is vulnerability has been used so basically they have access to the API can be an anonymous user because you did guys and you cannot distinguish the authorized and and correct fests from the malicious ones because it's just not it's just not in the logs because it's a valid it's a valid thing so what happened is there

were the privilege escalation so you call the API and it did get to do another API in another API and then you can compromise quads who are running co-located on and it won't host because usually you have a machine and it has several pods several groups of containers running and then if you escape wrong container you have access to everything and that's where you can also explode environmental variables secrets secrets of educators password system it's rotated project or containment of the environments so this was a very bad winner ability and yeah it's fixed now and kubernetes but you know you never know what's not fixed okay exploration rights so we're gonna pit it to continue constant and we've done a

bit of the vulnerabilities at least in terms of understanding it's going on let's go into exploration core platforms which is what to use or what you develop or we have then contains from scratch and their container runtimes and contain an Orchestrator those are the things that are basically use both in infrastructure economic platform wide on that same Stack Overflow survey we had over 67,000 actually yeah 67,000 responses and so people more than half of the people develop professionally for the core Linux after that Windows and then within those two although Phil topper in AWS so those are the four main things people develop in and forth when they use their own laptops this is what

is most like Windows and then kind of evenly distributed between Mac press infinite space so they are not working mostly on the thing they're developing for let's go to the most loved ones so Linux is very loved it is nice and then we have doctor I don't know where that flub but it's there and component is they are good to use they're easy to use and once you it's just that it's a bit of a learning curve that you would start and and then we have the wanted one so people want talker people want AWS and energen divinity people just want to developer that but most of the people are just like if you if you're a JavaScript developer

starting out people will tell you to use jQuery then this is the same thing you have to understand the details and really know what's going on do you really need containers I talk to a lot of people who don't like containers don't think you need them so let's let's take a quick intro into do you actually need containers know if you're working alone so if you're developing on your own you don't need containers it's nice because if you download stuff and you can just download the container image for example than work on that but you don't need it if it seems really small and the machines are all set up for the same development you know you have all the

things tools you also don't here and if you what you go out into the scale to just about in participant booth company that dermatologists internally for their accounting or something you also know you're not likely to need containers and and also you need some time for retraining and proper training which means if you suddenly usually technologies you will need to learn the bottle you can just go one of one of the chimp and spa and learn about it and at the point is no one else knows about how to use it and what we did for example in some companies view all the hats people developing in containers and then pushing that code into a non container

iPad from which all the words if you're comfortable around containers on their own then you can think a cue that and even if your company doesn't use you can change maybe if you're working alone want it on your skills you want to be enjoyable fast you'll just put any container on time on your resume if your team likely to get bigger at all the very useful so and the first team we use vagrant and dr. which was very annoying to set up and stop all this didn't work out nowadays things are really easy and then nowadays we just have a readme on github we explain the exact steps to set things up and you know you'll have the

exact same system as the person within there for five years and then you can just really focus in the code instead of on the setup and if you also can deal with the added complexity and retraining skills then maybe it's for you unless it is Perl native which is another common term that we use a lot right now the cloud native means you have all the people who are already trained or using the cloud and you're using the cloud for long it means you're either in a public cloud like e ie WS or you have your private cloud set up or you have a bit of both so I posed when you really need containers which should

think about getting them is if you work in a large team or across different teams but for example we have desktop team you have so many teams that all work on some specific part and then you will eat containers it makes things so much easier and obviously if you're using cloud is making whole lot of sense because you have more control over what you do they'll be pushing rather than having to focus on what image you're actually using in the cloud if you're in testing it's super useful you can just have your database run up quickly tap against the database and then shut down the container so it's very useful for a from scratch database that you want to

keep using although for a stateful one you can keep to continue running tests against the database and then purge it all or leave it on depending on what you can see it's very useful also if your application needs to scale and you know that you have this one container then you just use your platforms suddenly you'll have twenty hundred thousand machines it easy because you don't have to take care of that you just develop against this one container and push it it's done this is very useful if you want to clean system up to trying out new things so I for example your desktop and you have to install Python 2003 go and run and you really don't want these

to be on your system and because it is low tier then you can also use containers which is nice okay common workflows the way the way a lot of people at red head work who are directly developing for kubernetes or for the immutable hosts and these are direct quotes I got two until my colleagues one worked inside docker containers and says his local environment he just basically uses your local environment the settings and file system pipes turns into a docker container which is easy commands it is to look up how to type those via docker run parameters and then they'll happen your system inside a container you can play around with it but you're not

infecting your own post and some of us are using all effects for example and so yeah you can use coconut anomalous you can still do all the stuff you configured in assistance another interesting workflow is so someone wanted to use hot men inside odd men problem to continue rent and this is the same as a few Joker inside proper the way a doctor and odd men very so I contain a runtime if the wrapper around these big container on pops right it's a wrapper that already developed for you you can do it yourself but it's just going to take a long time and obviously then a one-person show running before pin if you use an established run

term like doctor apartment for example then you will have to you learn those commands like doctor run hot man run some people so even alias so you have alias doctor pub men and then they will use they will just continue using docker run but in the algorithm PubMed behind it the the thing about that is so most tutorials are for doctor opera fee because it's the more comping and but it has a daemon running so it's a single point of failure if you use the doctor Damon that you need to run in order to run docker containers you cannot it's a single point of failure and it's big whereas in part Minh portman does not

need to Damon both can be run root without root now so I'm privileged which is really nice which means if you use root inside the container you are bound against your user link space inside your container so when you use root inside the container in reality you're not using verge on the whole system which means you avoid a lot of the security problems you'd have even if you run malicious code inside you continue so definitely if a tutorial tells you to run anything contain the runtime privilege don't do it and if you have a specific case like you want to run docker into a doctor and you need to write privilege you can test it but

don't use it in production and so this person specifically for example month of use cloud men inside pod men so we have silver blue which is a desktop operating system and it uses a fedora toolbox which is basically at the mutable operating system and it has a fedora tin box which is a pod man container and in that container you have your own operating system that you can develop against if you want to use odd man inside the pod man it's only worth so one workaround is to replace user bin pod men which is what happens when you type admin and just replace that the flat-pack spawn host pseudopod man and then type the argument into it which

then creates type X pawn creates pod man container so it runs the process for admin on the root on the host system so basically you just pipe it onto the host and from there create one or can continue working in your total but it's a little complicated when you've never done it it's bound to be complicated but once you get into it it's easy just like everything else or it is yeah it's easy need not simple one way we call the desktop in workflow usage is pet containers so when you work inside your development you'll say pet container so my pet is or I develop this in my pet where is the server thing is cattle so

it might not be that is very the deuterium to use this these phrases but capital is how you would call containers that run at scale and reggae auto scale whereas your pet is your personal thing that you use on your own load weather obsessed approach server and then develop against t1 types of comments and security you do not use container images from untrusted sources if people tell you to download it it's cool for your own planets good for your own testing but as we've seen there are vulnerabilities hard word to say so there are vulnerabilities and we need to understand that if we just download something from the internet I mean you will probably know but if you

just go to any developer who is not very into security and a lot of them are not and you just tell them you have to create your own container because you can address the source they will usually be very angry with you I charge em yeah so make sure your on your containers I'm privileged so no new taxes do not run them as routes very important and certain first one if you use a different system there's a armor armor for Ubuntu in Libya and I think it's up but yeah at Fedora we use Esther the nips and I think a lot of other disparities in Selenas as well and make things you need to go get possible so if you can use

operating systems that are immutable that's also scary good obviously because if you have malicious code and hand write to anything then it's just plain and definitely set limits and alerts on CPU consumption because there are also it's not just about your own data and about your own system there are also um crypto currency miners that are basically inserted into you contain images and they will just use your hardware to mine from the container on your hardware so and you've won now there's there's it's hard it's very hard to do digital forensics on the omni on this type of stuff especially as we've seen is to put youtuber new similar ability from november so definitely set

the limits and alerts which if you use the cloud you can easily do in just 17 degree that's 50 years and if you have your own long premise other and definitely check your mobs regularly even if some things won't show up because some will know those inside like that you have to check your locks regular if you suspicious activity that you knew that if you're a suburban or have done anything similar to that okay let's go to the future outlook just in terms of what technology is where it's having so we have the immutable on the vinyl purchases and some problematic updates and that's exactly what currents and potato limit states and that's what the time we posted and that's where

endless lies is so a lot of the a lot of the infrastructure developers aren't heading poet immutable which it also means that they can provide security opportunity fast because they know that the system cannot be different than this it's easier to develop a security update you don't have to wait as much you just push the security update and you know it's fine and it's gonna run in thousand easier on sysadmin they know nothing can go wrong crafter can go wrong and inside the containers but that is contained and it's also not affected by the updates for example you can have on your system on your host it in half Pattinson and then and then and and you've got in the

containers you have packing thing and they both don't talk to each other it's fine so when you update it's really easy so get into the immutable operating systems check them out and our guest up we have less individual security issues more like big bangs this rug Gnostic where we have this Latino vulnerability where it just effective form of everything but only to use containers in route and if you didn't have linux enabled there are so many is that you have to do to actually get to this attainable mobility this is also anywhere there's news about some vulnerability we probably know that it's not that I effect a very low percentage and just like Todd before and

in the keynote said that you know if we have the proper procedures the stop first it is very unlikely to happen and if it happens you have to proceed you to resolve it quickly or it is presently not be pretty much on disturber he have one hump it to the mall so that pumpkins are outlook that we're gonna preserve in having in general the infrastructure is for example were currently working on the machine kinetic operator where you just define some basics and there's also ignitions and container limits which we also continue using or you just define all the all the things and then this Auto scale it creates the MS and just Auto scale creates creates creates and

is all with the same configuration you type it out once and you have your pepper on okay and now this is playtime if you have a lot of good view you can test this out already or when you get home this is actually really fun and you can basically test what happens if a container was to escape confinement so this is what this uses Codman and yeah if you just want to test out this vulnerability can happen in February it is a code you can do with so so it's just convincing peculiar and we also have beefed up the pump such containers where you'll find Padma and bilder and some demos and got seen so

they become such containers is Rights Act in that in that regard yeah this is yeah this is my ending side and then are there any question yes so the key areas for a developer who's one familiar with containerization what would be the top key things for them to learn to move into developing in this type of space it depends in that sense that when I started doing c7 I really went deep into it and spends 40 hours I think probably 40 hours to tonight dude and just learned everything on the Linode library but like deep down stop so it depends on what kind of learner they are if you just want to type it's just straight go straight into acting

they will publish use tutorials which are a double-edged sword because they tell you to just download this image and then work on this and then you get into the habit of three map then yeah probably not useful so it's hard if normally you just say use the tutorials that are out there and there are so many it's a pain validity on use of our plugin I'd say Putman just because you know we're working on it and it doesn't have a big Damon so it's it's yeah it's it doesn't have a single indicator in it then but the French this is fine it's just it's containers if you use continue to you have an easier book

though once you learned it I probably just really do to start containers from scratch thing if I if I was to advise someone and learning containers I'd say go to your search engine Google board you know containers from scratch there's a one Eric something and here's a very good post on using communications class there's also lists rice with several videos on it but the Eric closest the best on the continuum scratch and I just done that for example and from other tutorials for containers from scratch it's super easy it's actually really fun and that's how you really learn it but if you first I need to learn limits because containers are limits go they need to learn Linux and if they don't

know limit and getting into all the thing is hard but you have the practical scenario to the no limits okay the Danish instruction and then after that any tutorials with the added knowledge of security this is important and the other questions there was one [Music] okay yeah I mean really just a tutorial on the main page or whatever one you're using or talks there are several talks out there on YouTube the chicken just or personally I just like a good blog post that explains everything to me and then just go ahead and usually found this on that and once you know that it's easier for you to get into it yeah so I know like in a crowd

like this we're all going to be like yes build containers from scratch is great fun but most developers and things are most likely going to be using kind of pre-built containers and things like that is there do you think there's any way to kind of make developers and not do that and just encourage them to be more active in learning but well there are your own registries right if you're if you're talking as your own person or if you're talking as your company if you and your company whoever is promoting the container thing if it's not already a thing need to know about registries in humans registries where you as a company build your own images and then you tell

new developers or existing development you just download that image and you know it's safe so that's that basically the most important image registers there's several out there and that you can also hold on premise so you don't need to go to some websites third party you can just hold everything on premise first yeah yeah so I'm not a developer or anything like that but in the organization that I work for one of the things that concerns we slightly about containers is about managing secrets and I was just wondering what you think which is is the best way of managing secrets between containers is it using something like docker secrets or sort of send yourself up something yourself like

a shock or faults or using something like open shifts what would you recommend a little to you but but you know every everything has their secret management and for whatever you're using there is going to be a best way to do superb management and if you use something like ignition or whatever you will have your content files pointed at to somewhere just have to make sure that which has happened many times on github for example companies poster things on it up and then they also upload their secrets so that have happens and this is bad so obviously your core secret management make sure it's if you host the total code in a private repository but it's out there make sure it's safe

encrypted or something so in the code if you are a developer you will have a complete file of all the threads inside make sure it's encrypted in secret inside or something like that or but that's also that from scratch approach you should actually just use secret management from the plasma means if that openshift would have a way if that's talking there's the opposite so there's just forever platforming you won't use dr. secrets with you know like another runtime and and Python movies and so that doesn't really answer to your question because it depends on what platform even yeah sorry any other questions or but anything containers or a prescription stuff or cloud no okay on thank you and have a

great day