Look Ma, No Hands! Decentralizing Security for Scale

Oh [Music] [Applause]

[Music]

so I am NOT with ENSO and you are a security car requires on a year ago Mexico our new Brandon happy pharrell imagine himself I just want to give you a load of in soda or virus is it is promised starts looking back a lot of really show me we do about 90 million DNS requests per day we do send you on insurance appreciative fraud every day calipari because they're doing a real hard we aren't offers instruction some challenges for us each number required and the significance there is that when we're startup we were you know 10 engineers and the 50 engineers men 100 and we somehow now have got to 300 and the model of having one or two security

engineers for for you know 10 people or 4 100 people it just doesn't scale when you have 300 people and we can't hire that many engineers to scale the team that you all know it's very hard to find good people in security and weird our engineering team is kind of distributed around a few different sites so dev up because I know you guys love all these fancy terms that we like to throw around what it means to us is that we're moving away from where we have already moved away from the model where we have an operation steam over here that's running the servers are running the systems with software they're responsible if the thing goes down and night so they get

paged but then you even write that code it is design it they don't really understand how it works and over here we have the development the developers that actually you're building things and coming up with things and when they're happy with it they throw it over the wall to the operations team and the operations team is supposed to somehow know how it all works so you break down that wall right then you create teams called dev ops teams so if i write some software and it goes down I actually get page that it happened design systems that over DNS that have gone down in the middle of the night and I got page and woken up and I had to

respond to it my team has respond to it and you bet like after that happened you know the next day I come into work and I start focusing on things like high availability for fault tolerance or let me go and look and learn how auto-scaling groups work so we know that it works for these other features why can't i work for security and that's what we decided to start thinking about so i put this up here as like a traditional security monitoring so we build these rooms put a bunch of blue lights in them of like a lot of maps and stuff and i'm not going to lie they're pretty cool like it's kind of like an

urge dream these rooms like I'd like to work in here just for like a few hours a day if you like I feel important doing cool stuff and I'll really know who came up with these things who came up with the first stock with all these blue lights I like to think maybe it's like some former military commander that got into security and said you know back back in the military we had these tactical operations rooms war rooms and it works there so why can't it work for security why can't it work for the cyber you know let's solve the cyber by creating these rooms and hiring some of the best cyber security engineers in

America and put them all in a room and we're going to fix this problem but it really doesn't work you can't just try to aggregate things from all over your organization information and put it all in one place and then try to dissect and reverse-engineer how all these systems work and I kind of feel bad for the people that work in these rooms I know some of you are here today I've worked in that role as well this is tough it's really hard like your job in this room gets harder and harder every single day because you're taking in logs maybe it starts out as just a few thousand a day and then it gets the hundred thousand a

day and then you're in the millions and you're getting events from from all over the place from systems you didn't design systems you don't understand because you didn't write that code and you're tasked with finding anomalies and how do you know what's anomalous if you don't even know what the nominal state of the system is so this really just doesn't work so how can we stop complaining about this and get to some solutions so I'd like to think of there being some walls between these different parts of an organization so development and organization and operations this is kind of what we talked about when we talk about DevOps you know as a community we said hey we don't like that

there's a wall between there let's break that down and create what we call dev ops teams and then we started talking in the security community about development and security engineering so when I want to say security engineering I'm talking about like baking security into the system doing things like static analysis or doing threat modeling so people will talk about things like sex dev ops or whatever other fancy terms that I hate but that's what they title there there talk and I think we've done we've done a good amount of work in that area we're getting developers to think more about security and how do we write secure code but I don't think we've really talked a

lot about this last one which is how can we break down the wall between development and security monitoring so how do we get rid of those those blue rooms or maybe keep the blue rooms but instead of having one giant for the room let's have one for every team if you write some code and you're the one responsible for if it goes down at night and you're responsible for the availability of that then you understand how it works and how to keep it running you also should get those alerts when you're getting attacked hey there's some weird sequel queries coming in well I don't know if those are nama lyst or not but the team that wrote the product they

could tell you right away whether that looks normal or not so why not let them do that work so I put this up here and it's maybe maybe a little bit of a misnomer but um so what I think is most small engineering organizations don't usually have things like performance teens scalability teams and availability teams now I know that there's people in this room that work on teams like this so lying a little bit but in general a lot of companies don't have these these things so if we don't have those things why do we have security teams so let's step back for a second and assume that we do have these movies team so let's take performance engineers for

example so one thing performance engineers don't usually do is like go running around the organization trying to convince everyone that you need a right task code you need to write it now it's really important if you don't write fast code they also don't want write unit tests for code that they didn't write doesn't make any sense why why would you do that and they don't aggregating logs from a thousand different components all into one place and then spend just like tons of resources doing things like data science and ml so that we can try to like find that one needle in the haystack anomaly for all these things that really you could probably go walk up to the teens

that are writing these products and say like panda Singh looks weird is it weird and they might be like no that's totally normal but that other thing over there that looks normal to you that's actually really weird but we do all of these things in security for some reason I don't know why but it doesn't have to be that way so product teams understand their product better than everyone else we said that and if we decentralized security monitoring we get another benefit to we can eliminate this back-and-forth politics and communications that we have in our organizations between groups right how many times you guys work on security teams have you had to try and convince

someone that the security of some fixing security was important you had to escalate it up the chain and get a VP involved this is so important we need to get it fixed it seems like the developers don't care about security that's what it seems like from our perspective but I think when you really start talking back and forth and you have that communication you don't have this barrier you realize that's not really the case people do care about security they just don't really know where to start if you tell someone hey go secure that thing where do I even start if you are someone who works on like a red team or you're a pen tester

and you've attacked things do you really know how to fix those things some people are great at doing bolts but a lot of times I've seen people say hey I broke your thing i found a nexus s and then they say okay how it help me help me fix it and i don't know i haven't use that language before so we need to get those people working get everyone working together more and I think we can get the benefits we saw from DevOps over in this new this new model with security as well so what we saw with DevOps is if teams own the full stack of their products than their more responsible in there they're going to

build things that are more reliable because frankly think they don't want to get paged for this thing going down so if you also say that hey security is also your responsibility then you can you can be pretty sure that people are going to start thinking about security and they're going to come after you to ask a how how do i secure this and I think we saw this in our organization when we started pushing this down and promoting this model we changed from a mode where the security team was going after each dev team and poking at them and saying hey security is important what are you doing there let us scan your stuff we move from that so the

teams are actually coming after us and saying hey we want to do threat modeling hey we want to we want to know how do we fix this particular vulnerability that this scanner says we have what's the best way to fix it and that was that was pretty often that was kind of exciting for me to not have to be the bad guy and instead actually be someone that can help so the security team is not responsible for security and that's kind of like the model where we're trying to adapt at at Open DNS and every team is responsible for their own security so that's nice and easy to say but I'm sure that if you all went back to your

companies or wherever you spend your time tomorrow and you said hey we're not responsible for security chris said your sis knows might not be so happy so so there's two things so one thing is this really does have to get support from the top like the upper level of directors management's and sea levels and all that they need to like be on board with us that engineering teams are responsible for security as a part of what they're building just like availability it's just like performances you know engineers don't usually stay like I don't care if my codes not fast that's the performance teams job but I hear people say that for security like I don't know about security like

the security team will handle it so that's one thing then easily come down from from the from the top and the second thing is that the security team then needs to start shifting focus so I'm not saying you shouldn't have security teams and that we all should just stop showing up to work but what I am saying is that we need a shift what we spend our time doing so instead of doing manual triaging and processing and trying to scan things and find who owns this asset that I scanned that it had a high criticality vulnerability we should leave that to the teams and let them do it and in order to let men do it that

means we need to spend more time with development developing security tools doing doing doing education to an extent but I worry a little bit to say that work is I think we need to make it extremely easy for developers to use things like if you're if you're running Jenkins to do see I for example a lot of development teams will run things like linters right oh my i want to run this winter to make sure that my python code is really pretty and follows a standard so i'll run like Pepe well why not why not just run you know something like bandit in there as well right in your right in your job I think if you see

those things coming in you're going to want to fix them it's kind of like a gamification of sorts okay so enough preaching that idea so I was going to talk about two of the tools that we're hacking on that I think are kind of going in that direction of empowering developers to do security or to take on security the very early stages with this so really really interested to get feedback afterward and find me in the chat but the principles here are that they need to easily fit into the current teams workflows so we can't build some tool that's going to require them to spend a lot of effort to get started or else is just not going to want to do it

so find what they're already using like maybe you use github and inside or maybe you use Jenkins integrated in with those things even if you don't like them very much it doesn't matter like just just integrate it in what what's going to be the easiest solution for the developers no reinventing the wheel we want to like right you know yet another secret management tool or yet another name your name your tool right we want to just use the stuff that's already out there if there's really something not there we can write it and open source it but so good source community is amazing like let's take advantage of that and let's just write just enough code to

integrate it in with our workflow at our companies to make it easy it should be automated we're huge on automation on api's if we have to click a button we get really mad so make it automate and make it easy don't don't like make someone have to manually do something and stupid simple so I the developer shouldn't have to think about like I'm gonna get this SS officer which cypher should I use sure ladies ECC I heard that was kind of a cool thing up and coming like no you shouldn't have to think about any of that just click a button then it'll click upon call an API and you'll get a cert so i'll talk about

two tools so the first tool is it's called starboard again like extremely simple this thing it's not fancy at all what we were looking for was just a way to do very very simple port scanning so as we were moving from the model of rack and stack hardware data center that we had from for years and years and we were moving more into the cloud of course where an AWS of course we're doing docker all these things of course you know that that model of that central fire wall where a security team manages the ACLS just doesn't work anymore it developers themselves or writing security groups around AWS ec2 instances developers and cells are writing in

docker we have we have a docker hosting platform that has its own little firewall thing for each container so the developers are writing that vacant expose things to the internet there's no sign off and so how can they monitor that things look like the way they want them to look and so we wrote this super simple thing basically you can spin up these little and map scanners that you port scans at different perspectives in your network and I made a really simple even more simple example here which was let's say I just have something and I want to know what ports going to access from inside my network reports from outside and then it compares those things to a baseline so

the developers themselves read a little baselining yamel and if anything is different that goes into to nagios because nagios is what we're we're all using for how many decades there's still nothing better but that's what we rely on for monitoring so if it builds into their their their workflow today so anything anyway if something changes that will turn red and nagios and if it's a bad enough thing it'll send out a page of duty so teams are using this and teams are learning things about the things that they've built that didn't even know like oh I didn't even realize that port was open and it was only internal so that's good but still why is

that open yeah that's that so the next the next two lists that we're working on is trying to make ssl certs like what i'm calling dead simple so we for for a long time and still in many cases have this very you know manual process that i'm sure a lot of a lot of people have where you the developers are having to figure out how to create a key and then how do i create the csr thing and then let me google for the commands and copy and paste them and and then i have to sign out upload my csr to this CA and it's very very complicated we started using a tool from netflix called lemur

which is really awesome by the way and that helped us move forward quite a bit so developers instead we're going to that tool and they were requesting search from there and then that thing kind of does the magic for them so we wanted to make it like even easier how can we make it even easier so well we're building is this thing called bins and basically you call this API and say hey I want to start with these domains in this case I just have like two stored US Open DNS and stored on umbrella and what it will do is in the back end it will say okay was this a public we facing cert or or any other doesn't need to be

publicly trusted or not and if it's publicly trusted then it'll go to lessen Krypton it'll get it and it's free what the group is awesome it'll put the key involved and we have these other options there too so there's there's a limit to how many search you can get with less occur for things like 20 per week per org so if we hit that limit than we fell over two digit sir and then we'll pay for it but hopefully that limit will will go up it's less incred people are listening please raise the limit and then if there's something internal we actually haven't implemented this yet but we're we're doing this soon then we'll we'll hit our internal CA and

get an internal sir because we do a lot of things like container to container communication or anything that we want to use client-side search for but this still wasn't easy enough because then the developers now get the cert and it gets the pointer to the key involved but then they have to figure out how to deploy from vault which isn't that hard but why not make it even easier so we have this thing called quadra which is a container platform which is kind of like uber Nettie's so we started working on this thing like before kubu Nettie's was released so that's why we're not using that but we deploy most of our things on de Quadra now so we thought how can we

integrate it with cuadras so they can just have one stop shop so basically now what happens is developers can go right to quadra when they're deploying their containers and there's a special type of container in quadro that we have it's just a simple load balancer I previously you would have had to take the all the cert material and give it to it as as a pointer to our secret storage so it would do it download it but it was still kind of clunky so this way now you could say hey I want a load balancer and I wanted to support ssl and these are the two domains that I'm going to be running on that on that load balancer so

quadrants is okay cool and it goes and to talk to visit in service and it gets the surf for you and it puts it in the load balancer and when it starts getting close to expiration it'll bring up a new one with a new cert move the traffic over and destroy the other thing so basically developers don't have to know anything at all about us and so sir which is really awesome because of how many problems they cause when they expire and that's pretty much it so honestly guys I really want to like kick off like a discussion that's not here right now but when I when I go down if you have questions or if you have ideas

I really want to like know if you guys are thinking around the around like the same path and if you've done some of this stuff already I'd love to learn more about what works and what doesn't work because I know this is for us kind of a new idea and you might say hey that doesn't work Chris I'm starting to tell you we tried it five years ago but find me and Chet thanks on behalf of a Fitbit and besides CSF thank you Chris another round of applause for him