Presentation -- Scaling Security Training at the Speed of Devops

it's red sorry yeah brothers yeah okay everyone we're gonna go to get started your way back from your seat I think we're still waiting on art I think we are speaking with Mike yeah

[Music]

[Music]

all right before we get started just a quick poll how many of you are managing your DevOps people you're not raising your hand and you've got to be weird about the team everybody on the side we're gonna target you first of all a little bit about the scooter though he introduced Hitchcock binaa Cisco for about seven and half years now new mixture of contesting and security goals and I'm Brian imple very similar background the latest I've been in Cisco about six years in testing money Roger sequel principle generated Cisco's security research then give loves period ended friends hope it's big dated so so we're going to hook up to quick little double strings what's happening in the

industry and this is Debbie crowd with this whole discussion about continues to liberté can do testing or lease on a frequent basis right so we're seeing this mark bastard right right so our company

so [Music]

[Music]

[Music]

[Music]

[Music]

[Music]

and some random hackers somewhere in the world plans a new vulnerability or any way to scan something right there in a favorite language they don't adhere to a standard of an output format JSON or XML

[Music] turned into like this scenario everybody something anything everywhere or games yes this is gonna be me let me guess why it wasn't me it's not named Walker anymore no an executive it provide an executive of hamburgers all the time true story so you know an idea this is just like two and a half years ago yeah two years so the industry change to their tools they're doing this now

which speaking to burglars Clearing House now this is happened before it would

[Music]

[Music]

this was a huge one that we wanted not to keep a tool inside of Cisco this

good-looking clicker users so they don't do anything right you realize that your honor hurts developers who have

[Music]

you might some questions so this overall

so the reason doctor is that's every test that gets written it's written into a at the heart here you got so script everything you want and fit into your conc pipeline we've also got another ride which consumes you can get all the stuff through your web browsers down here you have some reading use so as Roger said we had yeah uh but relay you have a private network you've got machine once and they're not publicly accessible you can read that inside your Wow that will hook up to this revenue scheme here and that's how let's go down we run our own duck register that's where we keep our secured tests although we have this recently introduced the idea of running

your own registers so if there's a test that's not included like doctor says you want to be very open and extensible so you can set your own register whitelisted in your setup and you're really controlled test from your databases to everything up here we have console and vault and consoles used for service discovery it's not terribly exciting both we use that for a home of our city search and so but all of our results are encrypted risks as well as all usernames passwords can be grading so things like that and it's all right in my ninja boxing so this is an example demo network something that we actually kind of use on the newer project itself

we were in AWS and we've got our this whole thing is BBC and up here to our public subnet and down here is our predecessor in the public some that we've got an SSH surgery that's enclosed and private that most likely is drivers so some of the demo videos that will show this is the setup so go quick some of the terminology that we use in regard to NORAD we have the idea of organizations and it's kind of like the idea organizations in github it's just a logical way to group things users are allowed to create any one River or positions they want so they organize your assets machine that really just anything that it could be a poem we don't really have

best opponents of moon but who didn't ID and there's chance to test something on services so a service is something that runs on machine it's really just any application elimination that exposes either disappeared and a security test in Quran it is just secured to this wrapped inside of the diver so go through a demonstration here to show how boolean assets into its like I said this is gonna be based on that demo a to us so the first thing we'll do is create an organization this is the demo from our access at 600 so now created a organization will go considered as our default up with the login it takes us straight to it you didn't have to so

when we go there and see that we don't have any machine right now so first thing we're gonna do is we're gonna come in we have the idea that you can actually hold everything from your instructor to service provider all right now these four open stagnated us so we're gonna put in our details or testing that one here and what that's gonna do is it's going to go out reach up in the a to dis API and look at the ec2

and now we go back to our machines we can see that we ran a few machines two of them are the one that I spoke about earlier slide one neccessity wants a Webster the third one is after the relay the pop out later but he was already up there and so just in there

okay so quickly to go over how you can skin insulin or how it the public see in the world like I said everything's API driven so you be either from your work anywhere it why that's it so that huh using your JSON call are your own a back hauls the EDI it will spit up the doctors little grooves up to your public asset and that's the simplest case but obviously not every not every asset here accessible so he received less internationally so you would stand

so what we're gonna do is we're gonna actually enable some test run we used to have Qualcomm my default because it was just our people we're going on the both package on this one which actually would ssh into machine and look at the various packages and reacts all this is you know [Music] this is just showing the described machine so now we're gonna go ahead and add to get see and this has changed a little bit but at least will be either and the team in this one it'll show the current version once you put in you never get back you can do it again this is encrypted at rest so now that it has an ssh key

[Music]

so yes clippings so now we got to started let's get into that machine and pick it off once you go back into machine so that's what we'll show yet skin is little progress to scrape arms and nothing's come back here obviously each tool takes us up in my timings depending on the tools of your own that's actually not ourselves so we see we finished and now we come back and it shows us the actual raw outlet of and in the statements and you can also get more detailed description of left light found and that all has to do with how the test was written what is your most one thing there we got this little bug icon we

have any idea that you are also able to take your results and export them to another third-party tracker so you're using your ear and something pops up but that level it will send it to your Giri

okay so that was all very well for front skin old message right but we have no way to reach this machinery right so even you could never see robbers so as I said we had the idea of a relay which this is a machine that actually runs docker and it runs our relay client inside there it phones back home and it's up to a queue and then as you start saying you want to run scans it pushes messages down on that cue move back down to the relay relay didn't reaches out to wherever whichever registry the screw tests beside them pulls it down runs that against the whole sentence so that device in their tests list them

now and you can see down here we've got 30 box which is our relay and so it's instant message on the relay relay intestines there are requirements but we tried to make them as simple as possible as far as getting to relay into your network and connects back out to us all of these that you can see our egress so you don't have to open your firewall to let things in but you do have to let outbound access on eSports we have a morality book you guys will be able to use it because it's just go but that represents wherever your instance would be running and then also this is our documentation server there's no cure but

other than all of our stuff is thoroughly documented he and gets that as well it's a tan or I get lab that io / Doc's you know and it shows how happy that relating soul yeah Debbie package marketing package things like that and so I like support now we need to try and stand just present ain't turning web server so we visit with that machine

and this one we're gonna do as a test I can't remember exactly what was that pass the test object now we've got tests that that will do that sir

oh that's oh yeah this is going to go through right so the way that we register relays is your organization has its own organizations hope and it's meant to be someone of a shared secret it's when you go into your dashboard record it could be changed it's it's meant to be a piece of information so that when the relay connects back up and said I wanna join this organization you can see right there there's the organization co-curricular that's the only thing that this is useful is registering really so that's what we didn't feel super secret and also hey why we have another city party but even with that or do that [Music] hood it's just not gonna they're not

going to know about each other so you can see hardly that actually runs a sadhaka container its up so that we have to really install and here's where we are able to have one more sake right if someone were you organization open and really look it's not actually going to work until you come in and verify that we do have the ability to do it Auto verify which that means game on the registers up is just not even worse but sadly recommend we just want to have an optional visual tools so now we're going to start a skin and now that seems matching earring from the relay rather than from total clown and you can see I

think one thing I've dealt with mentioned was you have the ability to add tests at the organization level and at the Machine level at the organization level it's one of the test is going to apply to all machines at the machine no violation so that's why we had sold balls up there as a test of the run we added a dealer digitally and so again it raining they've got some doing things so there's a new things have changed since we recorded this again this was at our website 16 we also now have any idea requirements that's what private assessment requirements are so didn't Cisco we have a bunch of products appeared baseline requirements and those

requirements to actually accurately test it might require multiple tests will that in the planning so you have the ability to come in on their own requirements to say you know requirement one you need to run tests one two three to test for long six so this is just a quick slide to talk about enterprise box and in that box as well this is how our deadlocks did running basically we took all the different long center architecture put them into dock containers and as I said the enterprise box really isn't what we've done but the industry is you should be able to spin up here instead of box you know works but you build what I'm not writing

actually maybe against so I'm going to talk about Brian talking about the architecture time and what you can do with the system Roger kind of set the stage for our motivation so I'm going to go over some of the tooling that we have for developing tests or using the API and things like that so to start I think Brian mentioned there's an probably Roger to is kind of on the fundamental pieces of our application is that all of the tests are wrapped up into dr. images so this gives us the ability to share code so for those of you don't know how dr. works you can start with a base image and then any number of derivative

images from that there may be some limitation at some point but but in general you can share like if you want to have an image that has certain tools installed and all you would have to worry about is running running this specific subset of tools or parsing the information in a certain way given the whitening or trying to - the - to do so with that in mind we provide a daily basis image that's got some libraries installed and they easy to communicate back with being more added API as well as to work processes run tests things like that so the kind of fundamental idea here is a little use some existing tool like in map of

course you can also write something that is completely custom so you can kind of start with there in terms of running this security tool it gets we did have a thickening process for the official test mountain which a lot of that stuff is still I guess Cisco in journals so as Brian mentioned you can point at any doctor registry you want so there's the ability to use in a test content that you would like but as far as our process you know we've been able to locate a lot of CI most of which is open source at least see iPod part of things to automate the building of tests the getting of the test back into the

API and making them available also we would do support running from any docker registry so the first piece of instrumentation I'll go over is the NORAD CLI so the what what that is capable of doing right now is helping you to generate test content we also it's open source as a ruby gem so it's there's you go to be a web page there's all kinds of instructions for installation for a sample walkthrough and stuff like that it's as simple as just running to install NORAD CLI on whatever platform you're going to be from and we'll look beyond up once a little bit for the internet seeing all these examples everything they're doing is in Ruby because that's the language

that pilot three of us know best so that's a lot of the instrumentation that we have is is written in the Ruby but since these tests run in a doctor and engineers middle requirement to use Ruby or Python or any other language just something that can run in doctor is all you need to do you can even take advantage of some of the libraries that we have written that are in Ruby by just writing your tool complex in the language of your choice and then the amount of Ruby you would need to parse those results and send them back to your your additive API is very small so you just want to throw that out there because everything if

you're going to see this in to be an investment group don't like Ruby so what's what to insult the NORAD CLI tool Falls pretty standard in terms of being able to every time you type committed or a sub demand you can get help and it's going to print usage information for what you need to do next to continue using the total more complex ways so in order to start creating to test has leak in this example we're starting from the air mount so we're going to assume that we wouldn't have an existing test directory at all so the first thing that it's one can do is set up your repository structures so in this example this example I'm just going to

create a repository the best quality side asheville and you can just see some of the scaffolding that is going to generate it's going to be some default some of those things are going to be biased to awards so spheal because that's when people who are using this tool the most what everything is changeable from there most of what you're going to do would come into this set Testament it's that the command you're going to use to green test whether it's spitting out scaffolding running tests running the tests against the test we do have the concept of running validation tests to just kind of as a sanity check that your security tests are building properly that they

don't crash when they're they run things like that so some of the interesting man tier if you were to run we helped musician on set test you get go from building every image that you've got you can build specific images this execute command actually allows you to bride rather than running the security test in a as a harness you can just run it locally there on your box against a certain target you can see all the output just kinda fell apart that's the heating way you would expect and then you get into some scaffolding men's and the validations up here is what you would run if you are going to write unit tests or integration tests against your

security content the scaffolding tool is probably the one that is in the top five or most value because there are you know kind of some design choices we had to meet in terms of how to structure test content once you get used to it you know it doesn't seem like as much but early on there are only three or four files that you have to set up for every test so the Scotland poll is going to let you you get from the command line you can set the test type you can set what options is one of the things like that so it is in this case we're just going to do a simple beam test and we're going

to treat I just can't reach the post as a security chip so if you run the scalp or beam test you're going to see a doc while reading so some of that goes along with pulling things in through Alex to see on we have documentation server that has pulls from this grid me that shows what the test does how it works etcetera your manifest file which we'll get into more in a moment a wrapper and then your validation test if you are going to use those so a sample dr. Platt mud using NORAD and writing or your test content and dr. which is does not require any sort of major expertise with docker typically it's just a few lines it's

picking what Basin that you want to start Club so that maybe even NORAD basins that we have which is that they can do punches 1604 well that's maybe is fortunate for account remember copying over any files that you've been needing that image so the wrapper script I in this case is best even is going to be the digit 1 port for this dr. Amy and all that means is that whatever container is starting from this image it's going to execute this file so the amount of dock rhinology needed is pretty limited or that's it the vast majority of use cases the manifest file this is one thing that's a little bit flexible this is the structure that we

have and for our CI process some of these things would perhaps not be relevant if you were developing your hope that's not it but in general we need to specify what registry that this test is going to live in we need to know the name of it with what what version it is we cards this is the command line indication that the it guy's going to use when it starts this stuff and if you have any any options that you want to contribute or technique in this case we go over there you have an option we want to set the default config for the number of times that we will be good for so go

back this is just any kind of vanilla scalpel and now we've got this fit stuff to work better with AP tests so we're just going to specify these things or the target and if there's a few other reserved keywords that will show up in the documentation those are things that as an API is starting the test it's going to auto pull those things in for you so the target is just going to be the machine you're going to test this were in authenticated skin there's some default keywords that we have or sshd ethics user and there's a few others so those things are reserved so there's a lot of pieces I would say 95 some of the

tests that we've written don't require any user configuration at all the processes they come in but the person who wants to run the test comes in they add their machines to NORAD they add what services are running on those machines and then the tests are a dancer's to auto configure from that information so we made a conscious effort actually the first week we did try to bias the tool towards penetration testers turned out that wasn't going to be our most common use case so we started any more towards developers who were going through the security process who needed to run this stuff but didn't understand how to configure the testing tools so instead we just have to give us more information

about their environment so rather than configure the tool just describe what you want to test even though it's in the end it's the same information just kind of changing that context cut down a lot of our support requests in terms of how do I actually run this what does this mean and things like that so it's very possible for them the majority of tests that you write so require no configuration at all and then when they do we do have the ability to devise new paltz databases for most people and then this is the documentation this is kind of a format that we fought so that you can kind of treat this as a case study if you were

writing your own documentation of course you do write it however you would like but it again here's how it shows up in our documentation application so each of the tests that we have and I think we've got around seven something now in Cisco are going to show up here a little bit of information so that anyone who's curious about what they're about to run they can go read about it purge that they don't want actually be able to get this source code which is an option to choose but you know this is kind of a happy medium and okay so the wrapper script here that we'll go over is kind of the bulk of where if you're

developing a test this is the bulk of the work this is the script or you know you can write a library depending on how it says it's going to be it's what's going to call out to the security to long I think that animated yes so it's going to start the security tool in this case that's going to be B and it passes in whatever arguments it's gotten from the API as it was started and anything additional menu descriptive like to hard-code some awesome things you don't want the user to know about or have to worry about but it's really kick things off and it's going to wait for the results to come in once they do you know typically the

model that we follow is the security test dumps its results into a file so that could be XML JSON it could be any format at all but this is the part where it's repeating that little Turkey which is parsing those results and interpreting you know if this piece of information is here in the file and we want to call this a fail or call this a pass we also have in this case we just have a passive fill we do have the concept of informative test so some certain security requirements it's really hard to test them in an automated way you need a person to come in and look or you just need to establish a

baseline and so we can run tests that are informative only that come back to the system they allow a security architect to do you those they can still run this at scale and see all the results in one place and we still have confidence that you know it's the same process every single time that time so many jump pretty quick but you know once once everything is done it's just going to come back to the nor I plow and you want to be able to see it in the UI or be a bit API like Brian was demonstrated gar commands in the nor ID CLI to actually build more images we tried to extract out a lot of media even though

the doctor does a great job of having a pretty simple API in terms of what you type in on with command line you know we tried to add a little bit of an attach direction over that just so not everyone who's developing test has to learn doctor I'm going to skip through unit tests so some of the stuff that we've got going in Cisco we have some I think the main third-party one that we use is for leveraging the Koala scanners that are deployed throughout Cisco already we use open source tools like pulse variety showing exact which is a web standard and server specialist need to talk a little bit more about quickly here in just a moment and

Brian was talking about the requirements interface so I would insist that we do have a concept of a Cisco security about the lifecycle lots of companies have things like that and we are able to structure our tests specifically for requirements that cisco has that other people may not need to be in to this so it goes back to making the tool adaptable to whatever you need to work on one open-source tool that we relied heavily on is service net some of you have probably heard of that but from a very high level it's a DSL for checking server configuration is a way to build very customized desk and this example here it's would've run authenticated on a box

and you can describe the file that you want to look at and you can test the things in its content so here we're just seeing if we have a secure sshd configuration and SSH client configuration it's very accessible very powerful it can do a lot of different types of Texas's you know fraction of what it can do you can use it to what run commands you can see it packages are installed and things like that it has a abstraction layer over the operating system so you can write tests with the same file that will run against multiple popular operating systems without having to have a different source code for each one finally I'm going to talk about a

couple of the API clients that we have in terms of being able to instrument all the different features in the API we have a Python SDK region is ready to Python is because we had someone else on our team jump in and wanted users and Cisco needed something written most of what they do this is in Python so we've got a Python SDK that wraps the majority of the VI you've got a API package written in go I've been working on that recently because we are working also working on a terraform provided plugin terraform is I think a pretty popular infrastructure as a service tool a dead loss tool for describing your infrastructure and resources and so we've been working on

leveraging that to let people have in machines their order to test etcetera in tune or a beat of that we do have all the API endpoints documented and this is available on on bit lab or open source at this URL where as I did a lot of i/o there's no ten or fifteen celada packages there we are under the aptitude license so you're free to do with it as you wish yeah so just to wrap up we want to be able to be configurable take all of this different testing paradigms all these different tools package them up in one nice little robot that can do all of our security for us thanks

to deal with my village I know they're taking donations from Africans in sorrow got some awesome prizes like participating see yep okay the registration is upstairs out here Department axis is good for down cheers oh thank you very much killing it as usual you guys coming up too soon after party