How Secure is your Linux Distro?

okay so welcome to how secure is your linux distribution or if you prefer the Linux Mint saw bash fest of 2016 I am at our old who am i or who the heck am I um I'm a I've been working hacking on linux distributions since I was about 17 or 16 mostly debian but slackware lately so I do mostly instant I used to do mostly instant messaging software and I are seeing various communication stuff like that that integrating a debian that sort of thing and by the way this is my first talk so don't be too cruel to me um so who here uses linux mint you're probably own already or will be soon enough but

um the inspiration for this talk came about in February of this year who here knows what happened to linux mint in febuary okay so anyone want to tell us what happened to linux mint in February

people were corrected for two hours that's almost right they compromised the distribution itself the distribution download servers and replaced but and replace the official Linux Mint I says with a compromised version and by the way since their distribution servers were also their website service they change the md5 sum on the website so but it got me thinking what is the linux distribution anyway well it may seem obvious it's the stuff you insert into your computer to make it go unless you're on windows in which case you're in the wrong room but how a developer defines a linux distribution is the linux kernel obviously otherwise it would be a freebsd distribution which debian does also do you should check it

out it's quite awesome them together with a set of packages a set of software upstream software that makes a complete operating system you can think of the Linux kernel is the CPU of the system that way um it's the same analogy as the same CPU is in the nest on nintendo entertainment system and the Tamagotchi but they're different systems right no one got that analogy so depending on what the distributor decides to do he can control almost anything so the distribution is the OS not linux not the kernel itself the colonel only provides drivers and other such mundane stuff it doesn't even build itself without a bootloader which is also provided by your distribution so

security engineering and the linux distribution context ken is critically important for the whole linux ecosys and apart from the major distributions everyone does it badly and it's not just and and it's not just linux mint it's the firmware that runs inside your router is a linux distribution the the thing that runs your TV is the linux distribution so what this talk is going to cover is how security engineering is done in the major linux distributions so that if you're like an embedded systems designer you can take some lessons from all right so how do so so how does how do the big players define security well while it's not written down anywhere but I've patched together a definition so no

linux distribution will be able to secure all the software all the time there are vulnerabilities in packages coming out daily some are more critical than others but every day the linux distributions or at least the big ones deal with security problems so you won't be able to secure all the software all the time so what a linux distribution or at least the big ones Debian RedHat etc will try to do is not introduce any bugs new bug new security vulnerabilities themselves and the parts of the system they do control are as secure as we can possibly make them usability and security are always a trade-off so we can't go locking down the system and requiring it

to run only signed binaries or it wouldn't sell well as a distribution so so and the other parties react quickly the reason I know that security vulnerabilities are being fixed daily is because they are being actually worked on daily and um security teams know what's go what vulnerabilities are coming up in the next week we'll get and the next month you can't predict vulnerabilities but as soon as one is spotted um it gets picked it gets picked up immediately and worked on otherwise people will hit you with bats or or call you up and swear at you in German I've had this experience myself at two in the morning yeah yeah but look like a joke

how'd they even get them how they even kept my phone number oh I can't so now now that we've dealt with definitions let's deal with specifics there there are multiple levels to securing an entire OS and it's not just responding to vulnerabilities and software it's also making sure they don't get there to begin with so and how do we make sure voter abilities don't happen to begin with we secure the toolchain GCC the noodles the linker in recent years with the advancement of static analysis technology and compilers it has become possible to prevent an entire Club entire classes of vulnerabilities before they even happen at least in most cases hackers are clever and are beginning to get around

it but we do have least most of the obvious stuff covered now I'm not a compiler designer so I can't really tell you how that all works all I know is it's that that it's there it broke stuff we fixed it now integer overflow just stack smashing all other classes of vulnerabilities don't happen or at least are harder to get to work than they used to be it used to be trivially easy to exploit a unix box and this applied everywhere freebsd anything all you had to do was find a buffer overflow in I set a set youíd binary so all you had to do is find a buffer overflow in ping and you were golden now that doesn't happen

our least it's harder so if if you want specific details about what we're doing see me after or what or what the team that's responsible for these things are doing so the next important step to securing the Linux distribution is securing the channel of distribution your apt you're young you're Dan a pure whatever package manager you have it has to be has do not install anything it's not supposed to install and that's turns out to be harder than it harder than it looks um because we can't use traditional TLS security on repositories anyone know why now um because we have to rely on commercial CD ends in that case um basically if you have one

central server you abandon and a distributed system is much more secure than a centralized system every day of the week so what the major distributions have adopted is they sign their repository index files with gpg now what a repository index file is is it lists every binary package every rpm every Deb that's available in that repository together with its secure hash sha-1 md5 whatever it really doesn't matter so after apt is done downloading the first thing it does is check the md5 or whatever hasher using and you should be using cha 256 unless you've royally broke in your apt configuration which is very hard to do these days so the first thing it will do is check against its

database of hashes and it won't install anything us the hash matches but anyone want to know what can be the difference between a boner a piece of software with a remote hole and a piece of software without a removal one bit one bit in a binary um can often mean the difference between a vulnerable piece of software and are not an ounce of software so so basically we use PGP to make sure that no man in the middle attack happens because it's trivially easy to change the md5 or the package the signature of the specific package you're working on and um then flip the bit in the binary if you're using TLS Oh up or so pgp is used to sign the index

files and that's secured and that secures the channel of distribution in most cases um we are come the ftp master team of Debian and there are similar teams in Red Hat are constantly are constantly doing research on as to how to break it so they discover vulnerabilities before the hackers in this particular piece of infrastructure so what what so basically two attacks that have been prevented before they were ever able to happen in the wild where the replay attack where you replay a known unknown vulnerable version of the software as if its current there was no way to detect 2010 now there is it's called valid until it it simply date stamps the release files and the other attack was

not so much an attack but a design flaw in apps and I only know specifically about Debian um so the design flaw an app that was discovered was um basically any key that was registered in apt as a valid signing key would be accepted for all repositories there was no way to detect what repository went with what keys and new versions of Debian Jesse onwards it's no longer possible to do that so so that's repos security for you and I'm being terribly boring but once a repo is compromised the way the Linux Mint one was how long is the cycle time until it gets reset by the correct teams no longer be distributing well lytic

Smith was a media was a compromise of their ISO media not the repository itself now if you compromise a repository say you break into a repository sir if you have access to the repo server it's trivially easy and might be undetectable if you compromise the repo because the way automatic signing works on repo servers is you have to have the passphrase to the gpg key in memory every time you do a update of the repo which no one wants to do everyone wants to get their changes in fast and make it works and no one no one wants to be the guy that's standing by to enter the passphrase 24 hours a day seven days a week so um what they do

what most repos do is have a key that is not password protected at all so once you have that repo server you own the whole distribution and there are two ways of preventing that and both are usually employed by good distributions one is to make the repo server as difficult to hack as you possibly can make it which um back when I ran a repo I'm made it so you had to have PGP encrypted port knocking on it so the firewall wouldn't even let you in unless you or me or have access to my gpg key in any event and the second way to do that to secure against a repo compromise is to have the anyway everyone knows how

PGP works and will understand what i mean by revocation certificate so basically what debian and I think Fedora does this as well is have their revocation certificate and they put it through a secret sharing scheme and give the secrets to geographically distributed persons that are trust so basically there are 12 people who debian who sort of watch for a repo compromise and if it happens they get together and 7 mm and if seven of them agree that the repo has been compromised they're allowed to revoke any key that debian has ever used so once that happens actual staff work and that the time the time it takes varies um I don't know I don't know

who's currently on staff to do that in debian but if the people responsible for doing that we're anything like the people responsible for checking out security vulnerabilities back when I was in actively involved it would be measured in you 48 hours at the latest before you have German people cursing at you into him and then if that doesn't work they have French people cursing and then the Swedes get a bit it's not a fun day when you have a major problem and you're not taking care of it now linux mint as far as I know doesn't have any of these security procedures set up so what amazes most people would know anything about linux distribution

security engineering is not that a linux mint hack happened it's that it didn't happen sooner and you can see thee and you can look at the lwn comment sections and mailing list to see the reaction from the community as this happened i thought i bet they didn't have Clem's phone number they would be cursing at him so the next bit of security management in linux distributions is called patch management or change management see all the repo security in the world is no good if you don't tightly control who has access to change the repo at any given time um and and it's um and so the idea is any change in the distribution you all know who did it

when and hopefully why so you want a public record as possible and on you Bunty you can install Debian dash key ring and run a command to get a list of the thousand or so people who have direct commit access to the repo and their email addresses so as an active distribution contributor you get a lot of spam emails I get like five thousand a day and I haven't been involved in about a year so um because even without direct access the person with direct access better know who made that change when and why otherwise they won't have access for very um and and at least the Debian people and slackware two are very serious about who gets

access to what you have to go through a whole identity validation process where you go to New York City show them your driver's license and on you have to take a test it's like the worst final you've ever had and it's individually and they've been and when you go to apply for commit access they've been watching you for typically a year or more I don't want to face my own committers exam yet because they know exactly where I'm likely to mess up and they will ask you those questions um so and Fedora is similar they have not pgp validating their committers they have ssl certificates x509 infrastructure which is a bit easier for Red Hat Inc to

managed centrally um and that's a bit of security flaw there but um or could be potentially but as long as they're maintaining their correct procedures they are gapping their central signing server that shootin that's not much of a problem but as I said in security decentralized systems beat centralized systems any day of the week hang on that better not be I wouldn't be surprised if someone from the community was watching this would not be surprised in the least um now the next aspect I don't have a slide for this because it's kind of freeform and there's no set process because every security incident is different is what happens when you discover or it is discovered that a security vulnerability

exists in some part of the system and likes a bash or we have the shellshock vulnerability a couple of your Telugu so what happens when vulnerability like that is discovered well you all know what CVEs are right they're basically public reports of security vulnerabilities that exists in a piece of software um and

debian has what is called the security incident tracker basically its data mining and Fedora has a similar thing but it's not I think it i don't i don't know enough about their procedures to tell you what it's called or how it works or anything so basically their data mining the cds for any penny bug that applies to a package in the distribution and when something like that is discovered you get a friendly email and if you don't respond to the friendly email saying you have a security problem in your package or one of your teammates does one for example you typically get an annoyed email from a human and if you don't respond to that

they attempt to find other contact information other than your public email address and that's at the point where you get German people calling you at two in the morning came and basically they say you have a problem fix it or it's going to be bad for you um and

so basically if it's a publicly reported vulnerability it'll get uploaded as soon as it's done and it better be done fast like a week is is the furthest it's acceptable to be at for responding to a security vulnerability before you get annoyed calls and eventually your access to commit on that package is revoked and the security team takes over and it's a very bad day for your reputation anyway um but it is it's a private vulnerability things work a little bit different um because private vulnerabilities are private so the reporter of the vulnerability or whatever organization responsible for assigning the CVE will typically reach out to the major distributions and the week and if it's bad enough the major

distributions will reach out to the minor distributions and so typically the process works such that the patch is already in and fixed as soon as it's public so as soon as the vulnerability goes public you of a patch in the already built ready to update ready to go now for zero-day vulnerabilities slightly different um because your native owner bodis will get it depending on how serious it is we'll get the security team involved they're involved already in all security aspects but it will get the security team will go hands-on they usually like to leave it to the individual committer to fix their own security to fix security vulnerabilities for which they are responsive and people are leaving so am I out of

time where I'm I just boring okay um so um my first talk and I look dead comparable so so that's how CVE response works and um I haven't been involved in about a year in debian work but hang on now that I'll stay there sorry um but um I almost got another annoyed German phone call if my team mate hadn't been able to take care of what was going on because the vulnerability in a package I was responsible for was spotted last week and I didn't know about it and I will be asked when I can go back in May end of end of May to contribute to Debian where was Island the security vulnerability happened and um so

security is taken very seriously in most major distributions like I said in the minor distributions in the router firmware it's not at all with linux mint and a bunch of the smaller distributions they typically only have one server on which they put their website and there is 0 distribution and their build system and everything so compromising that one box through PHP VD or WordPress or whatever is the keys to the kingdom now linux mint was very lucky and this is the only case study in recent years um if you want older case studies of what happened the major distributions before they smartened up they look a lot like the linesman's case study would but linux mint is the only

major case study of a major breach in a distribution we have so what happened there was basically how is basically the download server as i said was compromised and not replaced with a malicious version of the distribution they inserted a package on to the I so they slipped streamed it onto the ISO which is very easy to do if you have the iso building tools right on the system there and we're at the point in the top where I expect questions so just stop me there um so they slip streamed a piece of linux now are called tsunami um which is basically a standard bonnet such as you'd get on a drive-by download for windows or apple but because they had

control of the build servers they could build it specifically for their target distribution not although not all malware for linux works all distributions all the time um because it's so heterogeneous and that's part of what makes Linux more secure than any other platform out there um or freebsd might be able to boast of more secure open open DSD they only get a better security record because they're at plane because they're paranoid and no one uses them mexican um that's exactly but so inserted this malware onto the disk I'll replace the md5 sum on the website and bang they were in business now because linux users are smart and that's the that's the up or most linux users are somewhat smart

that's the other security that's the other a bit of security in this puzzle it's you and I it's all the enthusiasts solving Linux users out there who noticed this stuff who are paying attention this was spotted within hours and they shut down the download server and got everything all squared away eventually but and but the only reason this wasn't as bad as it could be was and the only reason this was able to be noticed was the guy who did it I went and bragged to ZDNet and made his malware obvious nap now if you were going to be really malicious you could do a bit flip in the ssl package and make it vulnerable or or insert a

rootkit into the ext4 or or XFS drivers and then not brag about it um but you would still get noticed so this probably has it these first like major other smaller ones we we don't know and as long as they maintain their security practices correctly um um it shouldn't be a problem mint has a lot of users but is only managed by three developers who are very good at graphical design and making things look pretty they're amazing at that I installed the mint user interface on top of slacker that's how good it is that people from other distributions pull their software and are installing it on their own systems but because they don't care about low level engineering

and because they're a poorly funded team on it and don't really understand the security implications of what they're doing I mean we have some of the Debian RedHat and the other big players in the distribution market have some of the most talented security researchers in the in the world working for um so if you're starting a distribution if you're doing embedded device work if you're doing anything like that you should learn from them and learn from their mistakes um and um take it stand on the shoulders of giants in essence um because Debian fedora and everybody else have already done all of this security work for you and it only and and so it doesn't take a lot to re-implement

or use what other the bigger distributions are using so but linux mint doesn't didn't really care about security at about correct low level engineering before they had I bet they're carrying a lot now um but um so don't put the iso on the same server as the website don't put your build system on the same server as your website okay rule number one of web applications and linux distributions don't put web applications on the same server that runs your downloads don't don't put debbie ins website is statically compile it's all a bunch of HTML files on there are some CGI scripts um but the fool but the Debian user forums which are run by Debian are on a completely separate

server on a completely separate June in a completely separate geographical area on a completely separate subnet of the VPN apparently I don't know this for sure but the fourth but the when when the Debian user forums were hacked a few years ago I think it was 2011 someone nothing happened other than the forums were hacked in that sucked um but it had no distribution wide consequences now if the if a server like the Debian master server were hacked or rise or Alioth which are the central infrastructure of the project is managed by those three servers so if those three servers get hacked it's not going to be a good day and it wasn't a good day in 2005 when

they got hacked or one of them did I forget I forget exactly which one but it doesn't really matter but and it's very hard to hack those servers so um I went off on a tangent there but back to the original question linux mint is small in terms of development team size to what they're trying to accomplish um and that's what matters I mean they could easily like um not distribute iso's and just rely on you bunt to IFOS and a little script to get their stuff working but they don't do that they want to distribute the ISO themselves and that that has security consequences so anyone have any more questions because I'm running out of material and I still

have 10 minutes whoo nothing's off topic I'm I'm the king of tangents so yep

however have you ever um do you program have you ever done a greater than sign when you meant to put a lesson Sonya and if everyone does that all the time and the problem is at least for x86 the opcode the machine instruction that essentially controls what the computer is doing for greater than or less than our one bit apart and depending on where that is in your secure in your program you could easily have a remote hole if and there are other up codes which are one bit apart or or or or inserting an extra bite and it would still work but it would cause a remote hole all depends on knowing the software itself and

knowing how how its assembled everything but you can manipulate it there was there is a classic case and this was discussed at the Chaos Computer Club meeting of 2014 I can look up the exact talk title if you want that conclusively demonstrated that you could have a remote hole by flipping one bit so binary manipulation is a thing now there are no recorded cases of it happening in the wild but who knows like North Korea distributes a linux distribution who knows what they're doing with it um and China has this great firewall that can do deep packet inspection and whatever so who knows what they're doing with that you could easily get an encryption back door if you had to build

on what was earlier talked about if you are at the keynote you could you don't even have to legislate it you can if you control the Internet the pipes you can theoretically control everything so does that answer your question over there and

um you install Debian and um look up mate on debian or cinnamon on debian or you or slackware has even better support for this um you can essentially download cinnamon or mate like if we can get out of this I'm i'm using iterative i'm using mates on um yeah and their cinnamon spins um that are attached to larger distributions you bunt to mate remix is a good one and I believe there's one for cinnamon coming out as well I don't know if that's out yet but basically the software that's a user interface is disconnected from the distribution so you can essentially do it on any distribution distribution if you know how and the distro overlords

being the good good people that we are have provided ways to do that on a more secure distribution so debian fedora anyone will have this sort of capability you just have to look for it and not reflexively download mint because it's easy um and I don't mean to rag on mint actually I do because I um well it may seem like i'm picking on them but it could have easily this could easily apply to any small distribution like solid xk or app to sador apostate or whatever it's called um this could even apply to slackware if the security if the people running it weren't smart and paying attention and everything so that's about all the time I have um I

think and basically i'll answer questions until they kick me out um there you

anyone you could compromise I applied this to the major distributions but it could easily apply to the firmware in your TV everyone's using Linux now um so um it could easily apply to your phone you could easily apply to anything that runs linux um who would want to do it um generally anyone with an interest in hacking those things for any purpose um um yeah monitoring um um there could be trade secret implications like a lot of development companies use linux as part of their development work like um so trade secret theft just sort of anything you would want to hack anybody over a distribution is a way as with any other operating system so does that answer

your question do I still have time yes um any anyone have any more questions so you guys start your own again um and be like man um and now no I'm probably going to do a proof of concept distribution that just some weird things with the colonel but that'll stay in the lab for down um so does anyone have any more questions there can i I over there

ah yes I know they exist but I don't know about their Pacific procedures for securing that image so theoretically a hacker who is inside amazon or inside digital ocean or any sort of cloud provider could slip stream a piece of malware a rootkit or whatever inside those images as well that's another aspect that I didn't cover because it's way too broad to actually cover but theoretically anyone could attack that attack also works so those images better be secure um so do I still have time or okay does anyone have any more questions this is really kind of embarrassing running out of material 10 min for brother

you