Mixtape to Master Key Scenarios: How to block the Dark Army from mayhem using API-driven access control

yeah sure hi everybody thanks for coming my name is Erin I'm here to talk about reducing mixtape to master key scenarios blocking the dark army using API driven access control this time it makes no sense we're about to explain it and talk about why can you hear me better alright thank you so I should speak up I'll shout all right so little about me I'm the founder of box Pass were a new DevOps security company we focus on infrastructure identity and access control and you know we build a product kind of like what I'm showing you today but we think it's so important that we want to show everyone how to build it for themselves as well

previously I had various engineering operations and security roles at Pinterest Bebo doodle danger if of any of those any of those places all right so let's talk about this title defining what we mean by mixtape to master key so first what does the mixtape if anyone remembers write the old cassette tape on the left their mixtape so the theme of b-sides was mr. robot so I tried to pull my title from I might be the only one who actually did this hey it's like thing about it costume party party when you're not you know it's from Legally Blonde right yeah thank you so in mr. robot and in the first season someone correct me from but Cisco who

worked for the dark army was trying to fish someone who does that all safe so he's handing out his his mixtape trying to get him to put it in the CD at the office and succeeded right so might as well be a malware USB key left in an opposite front door same idea somebody got fished that's the mixtape right sadface and master key you know fairly straightforward right master key is a key that can open any lock and in this context that we're talking about today it's if you have root on any host so basically how do we make sure that we don't leak keys that can get you root on any of those I don't make more sense now

awesome all right so how our master key is created oftentimes by default like a company who's gonna have two levels of access you either have no access or you have full access and you know I see a lot of heads nodding actually pretty normal practice see there you're an engineer at this company or you're anybody else it's either yes or no so we've talked a little bit about you know why this is not the best policy but even at companies where they do have you know granular access pretty soon an employee is gonna have collected them all it's like Pokemon like gotta catch them all with every single permissions all right so why are they dangerous pretty

self-explanatory guard against malicious employee sort of the least likely fish stolen credential pretty likely somewhat likely and I was putting the slide together I thought to myself you know one of the big benefits of what we're talking about today is just preventing employee mistakes like if you're not supposed to be on a host at that time then maybe will prevent you from being on that host accidentally so you know lot of examples even in my career of someone thought they were deleting the death database and there goes staging if you're lucky they're prod if you're unlucky all right this is like this happens a lot happened to me my teams in my career so there's a lot about you as

much as guarding against outsiders all right so the alternative to master keys that we're talking about today is just to make sure that if you get a permission it has an innate or there's something that's gonna cause that permission to end other than you leaving the company if you're if you're actress it's not ending when you leave the company that's a very different problem we're more important so you know maybe there should be some end event time-based some activity that happens that we can talk about that that permission bills all right you want these to be short ideally would be you can use this this token to log in once and then you can never log in again

unless you get that permission again there's there's a whole spectrum here but the idea is that they all expire and no as I just said the sooner the better maybe you could measure that in days maybe it could be hours maybe could be minutes or maybe there's some other event that's gonna require this access to go away all right and let's answer this question when should these permissions expire so they're obvious ones leave the company of course if you put an actual time value on that permission this is this expires in an hour this is gone in a day this one's gone at the end of your consulting arrangement with our company but there's some other ones that a little more

dynamic that we can talk about so this is granted until you're on call rotation is done so when you're in pagey duty and you're on call then you're gonna have a permission and when your week is up a lot of this access is going to go away or all right we're back all right there's a JIRA ticket that maybe has to be open you'd have some access it's been granted approved it's in process you have that access you closed the ticket the access is gone or something really really clever you didn't you you didn't use this permission enough and now it's gone away we call this use it or lose it if you're not using something you clearly

don't need it it's gonna go away all right I sold you you want to build this cool let's talk about actually how you go wire this up your environment authentication berthe's authorization you know we have passwords and keys they're really pretty good at telling us who you are and then often if you miss the authorization step so whenever you are but now what can you do and the traditional systems to set this up usually right now is you got chef puppet ansible solve these sorts of things your config management tools we have scripts in there to edit your sshd about allowed users allowed groups you can fix up etsy group about who's in the pseudo who can

log in as a different user x' this works it's pretty pervasive but it's very slow right if you want to make a change to your access you check out the repo from github you you check in your change maybe you get a code reviewed push it out to the github repo then maybe chef server picks it up there maybe you have to run knife right to take a long time we're talking about things that are much more dynamic some people will switch to an LDAP server right a lot of overhead to set these up and let's use a cloud provider like us and there's somewhat inflexible it's harder to set up for fools who can be on what kinds of hosts

as well and then we're gonna talk about today is doing this with SSH keys so conceptually it's pretty easy if you have your config management puts every user on every host already but there are no SSH keys then that user can't log in and you can manipulate the SSH keys to figure out who's allowed to login and as what users alright authorization with SSH keys this should look pretty familiar two examples here of my authorized keys file and routes authorized keys file right so you open this file up add someone's public key remove someone's public key don't make any mistakes you're gonna get locked out of this machine I've been there anybody else yeah be careful look it's actually

easy right add remove keys we decided who can login to this machine so that's really the crux what we're talking about today right so if you do it manually a lot of work what else you got well I'll show you what we got let's make a centralized SSH key server to decide what keys are on which hosts under what users okay go spin up a database my sequel maybe we're gonna put everyone's SSH keys in there so we have a little web UI everyone in your company who's allowed to log in manage their own keys so when they get a new laptop we're not gonna come bug you to put their new SSH keys in github usually just make a little web

front-end using the walk to off against your office 365 your Google Apps whatever you already have upload their public SSH keys all right now I've got our database now we're gonna talk about once the keys are in that database how you going to give onto your linux server so a lot of options here one of the obvious ways is to just put an agent on every machine right agent will preview the database periodically pull those keys down ride them on to the proper authorized keys files on disk you're golden right so this works it's a little fragile if that code breaks if it runs out of memory and you don't load an accusing that machine and you can't load

any keys anymore that could be bad and another problem is it could pull very we want to make changes that are by the minute definitely several times an hour if you have all these machines pulling this one little API or I mean should have to really it's a lot of load when you're only making changes periodically so that we're gonna switch this from a polling model to more of an on demand model and here's how to do it inside of sshd config there's this new command as a six point two I think SSH where you can give it a command that we write to return keys to SSH so now instead of it looking on disk we're gonna have it pull

the keys from the database the one argument by the way is the username that's trying to log in so here's that script this script is gonna get run every time someone tries to log into this machine using SSH and it's fairly straightforward argument you know dollar one is the argument the username then we're gonna get the hostname I'll show you why that's important then if this is an Amazon machine we're gonna get the instance ID which is available through the metadata server so that's that's that first curl which is local to the machine we're gonna take all that data and that's where the second crop comes in gonna take all that data we're gonna send it to the key server sorry to the

the key database and see if there any keys for this user so here's the example requests at the bottom it's gonna make a request to your key server getting some SSH keys users Aaron host name is Prague - DB - one and the instanced ID is whatever your instance ID is on Amazon so with that information we can start building rules so here's here's our new diagram we've got an API key in between the linux server and the SSH key database and here's what that endpoint looks like pretty simple grab those variables we just passed in get the user from the database decide is this user it's a simple question is this user a lot on the host that just made

this request pretty straightforward if so return the keys and it's returning the keys in this format it's just like it was reading the file on disk but it came from that script so that's just returned as a HTTP response this is you know grab this could be this is Django ich yeah to be Flast could be node to be whatever your like to write ap as in it's now improving our diagram a little bit we're gonna apply on the top we have now we need to keep a database of rules who's allowed on the machines when we mix that in with the API the key response and this is where you get to use your

imagination what kind of what kind of rules you want to plug into your system so let's talk about some options you brought up some of these but they'll spell them out so to check if the user is still employed you could use your API on Google Apps G suite whatever they call it this week office 365 maybe you have some other way like an HR system that you can plug into so start using all these API guys that are available to you to start building these rules okay just maybe you should we check number one if you're building this all right another rule SSH key rotation how many here have systems to actually rotate SSH

keys be surprised if I see too many hands yeah so we can argue whether this matters or not but the reality is if you have any sort of auditors for compliance they care it's on their checklist and they if they care that means you might have to care now we have a built in system to cause it if you store a timestamp with every SSH key you upload we're near 90 180 360 days are up he never gets returned built-in key rotation alright pretty basic your time limit has exceeded this you were allowed on that host or those group of hosts for 24 hours up to 24 hours Lucky's not gonna be returned anymore can't log in

this is where it gets a little interesting so all those all this Thea that we passed in username hostname instanceid we can start using for pretty cool features so based on the on the host name you can start creating what we call coasters so a group of all prod machines would be a very horizontal group and that's the first example so you can and you can define these with regular expressions against the hostname so in the first example production machine would be the red X but starts with prod and has a hyphen and than anything that we got so an example of Pradesh dp1 you know that's a prod machine we can look in our rules database and say does this

user have access to production right now and that's how we can grant access or not grant access to this production machine the bottom example we could turn this into what we call more of a vertical rule so is this user it should be able access database machine vertically throughout any of the environment frog staging dev so maybe you'll have a regular expression that says the database if it has the word dashdb - you decided this is the database host base for naming convention to make the rule based on that we can do the same thing like this thing a very Amazon centric view but we have that instance ID we're going to do with it well we can

take that instance I need we can go to the ec2 API we can trade that instance ID to the end for DBC id7 and id8 agz all sorts of cool stuff so now rules can say I'll know it's a production machine if it's in the production vp c or if it's in the production subnet or maybe it has tag on it lots of information could be gleaned from especially the tags API is off so again plugged in this with one of your rules is this user on call go check your page of duty your ops Genie your Victor ops API figure it out this users on call you know this some people think this is

the dumbest idea I've ever heard of some people think this is pretty powerful love to hear your thoughts on it but if you really want to make sure that when you're on call you have all the resources and resources you need when you're off call your Kili keeping is not a big of a risk this might be for you

I use JIRA as an example here to be you know any of them the many ticketing systems that have an API this is great for workflow someone needs a very important patch done on the very important machine to open a ticket have it approved by the proper people that grants access to that machine for a short amount of time the work is done the ticket is closed the access goes away it's a recurring theme here access is granted access goes away as soon as it's not needed again this this user has not made this request for a long time so they used to have this permission we thought it was important but it's nice that these all these permissions just at

your feet over time if they're not being used so that way the set of access that a user has is is very focused around what their work is right now

so the possibilities are sort of endless here these are the stuff that I've come up with love to hear your thoughts on other api's you can plug into this the system and so in summary want to make your access grants very specific as specific as you can it's we've given you a lot of tools here to like focus them vertically horizontally whatever makes sense for that employee you want to make them as temporary as you can so ideally they only need them for the just the amount of time of the they have them for only the amount of time they need them and that way the master key leaks sorry that we have the users key leaks you

know that it's not the master key in this scenario all right that's in thank you very much if you like this system I want to build it yourself

you [Applause]

you