Continuous Security Testing in a DevOps World

uh so a bit about me i'm cto continuum security i spend my time doing uh split between security consulting and development work and that was part of the motivation for creating the bdd security project which is a security testing framework um written in languages and concepts that developers understand but for testing security features right but let's start off talking about devops um 10 minute introduction to devops so we're going to start with waterfall and waterfall was characterized by developing software in long batch runs right we did a lot of upfront design a lot of up upfront development before we had something that we could test so it took us a long time to get feedback from

our tests to find out whether what we developed actually works or not and this resulted in a high risk of project failure because things change in a project it takes two years before you have a product that you can even test it can become quite problematic so along came agile and agile said well let's break this big long batch ron run up into smaller batches and we'll write individual features so we'll create a feature we can test that feature and we can pile them up into a release and once we're happy with the features that we have we can release that those set of features and once they're released they then move over to the ops team ops team get to test it

into various testing environments they have to build that environment deploy it into the environment and then run the tests in that environment and finally deploying into production which is the first time that our users get to see this these new features so there's a bottleneck where features become a release and then make the way to the users right what we care about and what devops says is what we want to do is get those features to our users as fast as possible we don't want to wait for a whole batch release then send it through this laborious qa process build those environments configure those servers and get the the product released what we want to do instead is extend the

agile cycle so we can deploy individual features so we can take one feature and put it through the whole process have it qa tested have it acceptance tested and finally get it deployed into the live environment the benefit this gives us is we reduce the feedback the time that the feedback loops take to get from testing and the time it takes our users to give us feedback on those features so we can deploy things really quickly and we can see that the users like it or they don't like it or it doesn't work or it's broken and in order to do this we really have to work better together dev and ops and the testing team really have to be working

together and we have to get rid of this idea of dev creating a software product handing it to ops and then letting ops take all the responsibility for deploying it and for testing it and and so on so in order to do this devops relies a lot on automation testing automation acceptance tests qa tests automated configuration of the environments in devops environments they're building entire testing environments out of software with virtual machines and with cloud infrastructure and automating the process of deployment so devops introduced well continuous delivery introduced this concept of a continuous delivery pipeline where you feed in security or if you feed in features or requirements on one side of the pipeline those features become code

they are tested unit tested they are then passed through deployed into a testing environment the testing environment then runs acceptance tests qa's tests performance tests and finally it's deployed into production in one long continuous process and devops is what allows us to do that devops gives us the ability to operate this continuous delivery pipeline so if you think of ramping up from doing pure agile all the way to doing continuous deployments the amount of devops you're doing naturally increases so if you're doing just agile and continuous integration you're still working within the development environment developers are building code that code works in their environment and that's their end product they then give that code over to ops and

ops can deploy it into the into the testing environments so at this stage you don't need to be doing devops because you're just handing over the software from development to the operations team once you start doing continuous delivery you're now delivering not just code but code and a working environment and a set of automated tests and you're delivering that into a pre-production environment so there's still a manual component here some type of test you want to do manually before deploying into production continuous deployment goes one step further and says well we're going to automate the entire process and we're going to let developers write code have it automatically tested through all the various environments and finally make its way

into production all in one continuous process so if you're doing continuous delivery or continuous deployment you have to rely on devops to allow you to do that so the challenge to security is quite clear right dev and ops are saying we understand what we have to we understand what we have to deliver and we have we understand how we need to deliver it in fact they understand how to deliver it so well that they've automated the whole process that means they can allow their developers to effectively deploy into production and some real world examples amazon deploys every 11 seconds every 11 seconds there's a new feature hitting production etsy does it 25 times a day and even

governor uk does it 30 times a day so the question security is quite clear if we have this capability if we can deploy that fast into production how can we do it securely and i think the typical answer from a traditional security team is more along the lines of a continuous annoyment model rather than a continuous deployment model right there's no way we can do security testing every 11 seconds that's just not possible and there's no way security is going to allow a bit of code to hit production without undergoing adequate security testing i think if we want to answer that question how can we do this securely we need to realize that security is really not that

special i mean security is really a subset of quality control it's a special type of quality control so if we look at how is devops already implementing quality control within the process we can extend those processes to adapt to security and the first way they're doing it is not a technical control at all it's a cultural control it's holding developers responsible for the quality of their code the famous example of this is pagerduty the devops companies when the production application goes down at 3 am in the morning they don't wake up the ops guy they wake up the developer and all of a sudden code quality starts improving once you start holding the people responsible

who write that code and i think that's a natural extension for security we can hold developers responsible for the security of their code and that fits in well with the dev the devops culture that we're all working together to the same goal there isn't an antagonistic relationship between security and dev and test we all want to get code into production as fast as possible and as safely as possible and then that brings me on to the next one the the idea that there is no them in the team and this is something i think security is particularly guilty of uh we're always referring to those crazy developers uh writing that code that really should never have been in

production we need to get out of that mindset and to think that we're all on the same team and we're all working towards the same goals continuous monitoring is a key part of devops and i'm not going to touch it at all because i think it's an entire topic on its own but the bit that i do want to get into is testing and how can we extend existing testing functionality to work for security so we typically think of tests as verification controls that's what a test does it verifies that something is true but developers use tests as specification in other words they write the test before they write the implementation this means that their tests act as their

requirement they don't maintain a separate document of requirements they maintain a set of tests so they have self-verifying requirements and we can do the same with security we can use automated tests as our security requirements if we do it this way we have tests that are code in other words it's treated as code is controlled as code it's uploaded to the source code repository we can control access to that code we can roll back to a previous test if one of our test fails and we have the capability to automate manual tests in other words do the manual test once and then record it and we can then play back that recording as an automated test

automating the scanning process of course we rely a lot on scanning for to detect security issues um scanning itself is automated but the process around it isn't right we need a human to go and choose to perform a scan we need that human to interpret the results to remove false positives and then to decide what to do about those issues and we can build automation around that entire process so let's take a look at the first attempt to doing this we'll use the developers tool a junit testing framework we use selenium webdriver to control the browser so we can programmatically tell a browser to do things for us and we create our specification we want

to change the session id after login so we write this test first before we even write our app so we know what we need to build so first off driver.get the login page of the site go and read the session cookie then log in and login tells us how to log in find the element called username and enter the username find the password into the password click login and the browser is logged in then go and read the session id again and assert that the old session id is not the same as the new session id so this works we've encapsulated a security requirement as a test but there are some problems so firstly the navigation logic is embedded

in the test we've got a mix here of two things we've got our security requirements and we've got the navigation of how to actually log in and which urls to go to which means that if we wanted to apply our same test to a different app we'd have to go and rewrite all our tests right the second problem is a technical issue with selenium it doesn't expose the http api at all the lowest level stuff we can do is redis cookie through selenium we can't even read the http response code from selenium so relying on a pure selenium solution to do this type of testing is not good enough because we need to be able to access the http layer for

security tests and the biggest problem is that it excludes non-developers if we're in a devops team and we're working developers testing and ops all together we all need to understand the specification we all need to be able to understand if it's failing why it's failing this is written in code that only a developer is going to understand so it's going to exclude most of our team solving those issues is the motivation behind the bdd security testing framework so introduction to bdd this is a bdd specification it's written in a given when then format uses j behave and it's written in a natural language so everybody can understand this right given the login page and the value of

the session cookie is noted when the default user logs in with credentials from this text file and the user is logged in then the value of the session cookie issued after authentication should be different from that of the previously issued session id for everybody who understands what a session id is that makes sense and this is executable there's java backing every one of these lines

oh

okay so this is a bdd port scanner which looks like this again this is a specification only the required ports should be open given the target host when tcp ports from 1 to 443 are scanned we can make this 500 so we can change parameters within the natural language rules we can use five threads and a timeout of 300 milliseconds so when the open ports are selected then only the following ports should be open port 80 and 443 so we can run this scenario

and we have test failure we expected a collection containing 80 and 443 instead we got 22 and 80. and the test failed our specification failed so that's quite a rudimentary port scanner but from a devops point of view and from a process point of view it is i think a bit more interesting than something like nmap and a human because we write this first so the ops guy who needs to go and build the server and configure the firewall knows beforehand what he needs to configure this is his spec the requirements understandable by everyone the requirement itself is self verifying and we can fail the build if our requirement is not met so imagine that continuous delivery

pipeline with a new feature going in if our tests are running in that pipeline and one of our tests fails we can fail the build we can block the pipeline say well this is not going into production because our expectation is not met so a bit more about the bdd security framework it's written in selenium and it uses owasp zap proxy to do the http stuff um and using those two things we can mimic anything that a manual penetration tester can do so we can control the browser we can control the proxy programmatically so we can recreate anything that a manual web app tester would do tests are understandable yep it fits into the dev workflow and obviously into

the continuous delivery continuous integration pipeline the logic of the tests is independent from the navigation code so every bdd test that is run needs its own navigation code for that app but that code is in a separate file so we have reusable tests we can take the same test and run it on a number of different apps and we just have to change the code in one place i think it's easiest to understand it through a demonstration right so the app we're gonna test is this one very simple app office login

and you can see the list of tasks you can type searches in you can search for things click on your profile and so on okay so very basic app so in order to configure this for bdd security you would check out the bdd security project each web app gets its own copy of the bdd security tests first thing we need to configure is a config file we need to decide which browsers we want to use to test at the moment support chrome and firefox and chrome is quite a bit faster so i'm going to use chrome over here it uses two browsers one of them configured for zap and one of them without zap so

because zap is a bit slow for some things if we don't need to read the http layer we don't use that browser we use the regular browser we configure the base url so this is just for convenience sake in our tests we can just refer to that part of the url as our web app there's a secure url for https and here is a java class file that we need to create that that contains the navigation code for our app so i'm going to create a new one here we'll call it b sides besides london okay

okay so a new file besides london we need to tell it what the session id is we need to tell who the users are so in this case we would want at least two users from each role so here we've got alice and bob who are normal users in admin who's an administrative user and some other configuration options most importantly we need to tell it where zap is listening so i've already got zap up and running here listening on localhost okay so the next bit of configuration is to create this file

so we extend a base class which is web application for testing web application at the moment the framework at the moment the framework only tests web apps but it will be able to support web services or anything with an http communication layer so right now web apps and we will implement interfaces that describe the functionality of our app so this gives us the independence of our tests so for example there are built-in interfaces for i login for any app that offers login for i log out for any app that offers log out we'll get our ide to complete that for us

okay so we need to provide these methods we need to tell the framework how to open the login page how to log in how to log out and how to determine whether we are logged in or not and this is not that hard with selenium so to open the login page we just need to access that url which is quite easy with selenium get

like that now we're on the login page we need to tell it how to log in and since how to login is a bit complex we can use selenium ide which is a browser plug-in for firefox that will make this easier for us so it will record our actions and save them as code so on the login form i'm bob i log in with password and i click log in so those actions are stored by selenium ide in the native silanese language and you can then set the clipboard to whichever whichever language you want to export to in our case java junit 4 you can also use ruby or whatever you're familiar with so we can then

paste this into the login function import that and we have login almost working so the login function takes a generic credentials object and we want to use a username and password so we just need to convert that to a new user pass credentials we don't want to log in as bob every time we want to log in as our credentials get username get password and click and that's it we've got a login function for our app next is quite an important one is logged in this the framework often calls is logged in to find out whether the user is actually currently logged in or not so we need some way to determine that and this is something that you'll have

to determine for the specific app outside of the time so i know that if we access the url task list then i see the word welcome here and that doesn't happen if i'm not logged in so we can code that up as driver.get

get the task list then if get page source contains the word welcome then return true in all other cases return false and logout is the final one this is going to be just a simple get

user logout right so that's the configuration that we have to do for every app that we want to test but once that configuration is done you can run the pre-written tests against that configuration so let's run the authentication story for example so the just the the j behave lingo here a story is a set of scenarios and each scenario is a requirement so right here i'm going to run a story the authentication story

run goes and fires up the one chrome instance that's not using zap the other instance is and runs through all the tests and we're done and we have a report of our authentication stories one successful five are failed and two excluded so we can see the details of that our first scenario password should be case sensitive obviously so when the default user logs in then the user is logged in when the case of the password is changed and the user logs in from a fresh page then the user is not logged in but in this case the user was test failure our requirement is not met the login form itself is not presented over https

we expected to see an https url instead we saw an http url failure and so on so you can run the authentication test out the box with this type of configuration you can also run the session management tests test that logout works invalidate the session when the user logs out set the secure flag on the session cookie set the http only flag on the session cookie and so on so these are qae type of tests these are functional security things that you'd want your qa's guys to test but we can also wrap things like anessa scan so a lot of the stories have a narrative the narrative describes why we want this what benefit does this story give us so

here our narrative is that in order to identify the security vulnerabilities on the hosts as an operations engineer responsible for the kind of configuration i want to scan the host for known security vulnerabilities and we have one scenario in there which connects to an server it logs in with the given username and password it uses the provided scanning policy it scans the list of hosts here and this is just a text file so you can add whichever hosts you like here as part of the parameters the scanner is run issues are stored and the following false positives are removed so we can provide a text list of false positives you want to remove here and then we accept our expectation on

the next line no severity two or higher issues should be present if they are this test fails and we can fail the build so that's more kind of an opsi type of test test your environment we also do of course scanning uh app scanning so since we're using osap we might as well scan using osprezap as well so it scans using different policies and for each policy it first navigates the app a given story again is a jay behaved thing if we're going to keep referring to the same story in a number of different places we can just wrap it up and say always run this given story before you run the scenario so always

navigate the app before you scan anything we can go and read this navigator app which says given a new scanning session and the page flow described in the method navigate is run through the proxy we don't have a method navigate in our java file yet we only have is logged in and logged out and so on we exclude certain urls that we don't want to spider we then configure the spider we tell the spider to go and spider the site and whatever it has seen through the navigate method and we wait for the spider to complete once we've navigated the app and we've spidered it then we start scanning it and then we start with the sql injection

scanning policy for example so in order to build this or configure this we need to create a navigate method

so we will open the login page we will log in and since we don't care who logs in we'll just use the default credentials from the configuration and now we can use selenium ide again to record our actions so click on tasks type something in click on search click on profile and that's enough so copy those paste them and we have the set of operations we want to perform through the through the proxy so selenium ide can't recognize the search button so we'll have to find that ourselves

oops first have to see it there it is search button has id search

click and whatever we've typed in here this is the code that's going to run through the scanner and the scanner will recognize it spider from it and then scan it and we can run that here we'll run just the sql injection scanning not the whole story so scenario run

and we should see it start up perform the navigation in the browser and then start performing the scanning it's also done the spidering already and now the scanning

yes i'll get to that so it's quite important that you you you need to make sure your navigation is working properly and with any kind of automated test even if your qa guys are doing automated tests the big problem they're going to run into is test maintenance how do you make sure your tests aren't failing all the time the web developer goes and changes the id of a button now all of a sudden all your tests fail so it's a very good idea to stick these types of verification statements in your navigation code am i on the page i expect to be on after i click the search button just to make sure that your tests

aren't failing so we have a test failure here again let's have a look

so the application should not contain sql injection vulnerabilities we expected to find no medium or higher risk but we did find two and it tells you where the url is what the parameter is and the cwe id for that and you can use these three variables in your false positive if you decide these are false positives you just add it to the story and it'll ignore those false positives the next time the story is run

okay so this is a app scanning type of thing again but we can do some more interesting things because since we have selenium and we have the proxy we can even test access control between users

so to test access control the framework needs to know who is logged in what actions they perform and then what sensitive data or what special data should they and only them see when they perform those actions so in this example if bob is logged in and bob clicks on profile then only bob should see his email address there and we can code that up in a test so create an arbitrary method we'll call it view bob's profile oops we will firstly drive driver do a get

for a page say just to make sure that we are on on the front page of the app we'll get that url then we will click on the profile link which we already know how to do like that and that's it so that's our action that's one of the variables we need to know what action bob performs and now using an annotation we tell it about who should perform that so it's users equals bob which users are permitted to perform this action only bob in this case we can't provide a list of users and let's just complete this sensitive data is bob at continuumsecurity.net

oops i have an error here

okay that's better so with this information the framework has enough to perform access control tests it knows who's allowed to perform this method what the actions are in this method and then what is the sensor of data it's meant to to see there so we can go to the authorization story and let's run that

run so it's first going to go and perform that action as bob it's going to record those http requests that are part of that view bob's profile and store them and then it's going to log in as every other user switch in the session ids and replay those methods so it's exactly how you would test this when you're doing a manual web app test except the whole thing is automated so authorization test it consists of the author authorization story is three scenarios the very first scenario sets everything up for us and just makes sure we've entered valid data so it checks can bob calling the view bob's profile method does he actually see this data somewhere

in the http responses it then also goes and stores those http responses and the next scenario is the one that actually does the access control testing so it uses a table that it went and created based on all the other users it knows about so it now logs in as alice logs in as alice and the previously recorded http requests for vue bob's profile are replayed using alice's session id and then alice should not see this data here and if she does then we have an access control floor so we have coded up an access control floor here and we can see that alice does see that data from her session and we can see this is

a very basic access control floor here any user can see any other user's profile but we have an automated way of testing it for any number of methods and any number of users

right ah jenkins okay so we have these tests we have an automated way to run all of these tests next logical step is integrated into our build process and for that i have a pre-recorded session using jenkins and this is this would be the entire process of a developer writing code committing it to source code having the bdd security tests run as part of a jenkins build so our developer goes and makes changes to the web apps that they're building they decide to change the way they do queries they change how they get access to the user's instance and they commit their code to get push it up to github and that's it so from the developer's

point of view that's it they've just committed code all of what happens from this point on is completely automated jenkins is automatically watching github if there are any changes any new commits it goes and pulls down the new version of the web app so it goes and pulls the web app down and starts building it it's compiling the web app it's going to deploy it into a test environment it's deploying it on tomcat here and it automatically kicks off a bdd security build after deploying the web app now bdd security starts running it's running in this case in the virtual machine just so we can see what's going on you can run this in headless mode if you

use xvb on on linux so this is massively speeded up it takes about 10 minutes to run through all the tests for this simple app going along doing functional tests we can see the progress and done yeah essential jenkins plug-in is the bruce schneier plug-in which you must have to give your developers some some idea about security so immediately we see we have test results we have 24 failures in total but all of these are old failures three of them are new failures caused by this new build that we've just committed so we know that there are three new security regressions in our build and we can click on them we can see what the regression is access control problem

cross-site scripting issue another access control issue and because jenkins is integrated with our source control management with github in this case we can navigate to the code that caused this issue so the developer can go and see or hang on a minute those were the changes i made and those were the changes that caused my new security regressions

and the security report the html reports are also embedded within the jenkins ui so while the developers is browsing jenkins he can go and see what the the detailed issue is okay

so there are some limitations there's no email functionality built in yet so if you've gone you want to test self registration you're gonna have to build the email functionality going pop an email off the server to register the user based on an email that the system has sent him account lockout can be an issue so you would either have to create user accounts for the app you're testing that aren't locked out or you'd have to be very careful about how you test those uh those lockouts or have a completely automated log unlocking process that you can activate through the framework at the moment the access control tests are not anti-csrf aware and the access control tests are resending http

requests to the server if there's a csrf token in there they're not going to work because they're sending the old token along this is something i'll change quite quite soon and then as i mentioned earlier test maintenance can be an issue so you should do error checking as part of your tests security testing is a bit different to qa testing or functional testing because in functional testing when a test fails you're pretty sure that something is wrong with your app but with security testing we want to distinguish between a test failure that means something is wrong with our test or a security failure something is wrong with the thing that we're testing so these two

things require two different actions from us one way to do this is to sprinkle your tests with uh error checking like this one so the first one about changing the session id after authentication uh given and when when the default user logs in with the credentials from here so now we know that the user is meant to be logged in but there's an additional step here and the user is logged in this is superfluous from a security testing point of view that's there purely for error checking so that if that fails we know hang on we actually tried to log in and we didn't log in and we were meant to to log in another thing you would do then is as i

described with the navigation method if you perform some action on a page there's a shortcut you can do verify text present some text so you can sprinkle complex selenium code with that verify text present just to make sure that you are on the right page and nothing is being missed by the way if you compare this to pure scanning at least you can do this here with a 100 pure scanner this is often obfuscated you have no idea whether it's actually hitting the pages you believe it's hitting with this approach you can see where it's going and you can ensure that it is on the page that it's meant to be on uh the other thing is captures so i ran

into captures testing a production site and i solved it by using the death by capture service so deathbycapture.com yeah definitelycapture.com is a service it uses humans to solve captures it costs six euros for a thousand solved captures which i think is a bargain i don't want to know how they're doing it's one of those things we just no ethical questions right so with our app we

okay so our app we're going to enable recapture on the login if you fail login it's going to prompt you with a captcha so we'll trigger that manually and now we get a capture so now in order to log in we have to solve the capture so the way to do this is that there's a behavior called the eye capture behavior and we can ask it to implement these methods for us in order to be able to solve captures we need to tell it three things how to get the capture image how to find it on the page how to find the response field on the page and then which solver to use in this

case we only support one which is the depth by capture service so we'll first find the image on the page which is pretty simple because recaptures uses a unique id recapture challenge image is the id of the element we just need to return the selenium in selenium format right which ele web element it is we do the same for the answer where we're going to type the answer in so the framework knows where to put the answer in

and we set the default capture solving service as i said at the moment death by capture is the only one this contains my username and password where i signed up with uh for the service so you can use the same code for your systems and then obviously we need to change the login function because login now needs to check after i've filled in the username and password is a capture prism if it is then solve it and then continue sending on the login form and since we've told it where to find the images and where to find the response field it can do it for us so we can run this test fires up the browser logs in with bob

and his password and now the image has been uploaded to death by capture we're waiting for a response waiting since it's someone typing it in and there we go and that was the capture response so it's a bit slower than doing a purely automated system but at least you can test production apps that have capture enabled

so basically we've moved the way we've we we work with our security requirements in the old way we would have security requirements described in excel sheets in pdfs or in word docs that we all know and ends up at the bottom of someone's drawer and is never referred to again we want our developers to use our requirements to build the app we want our ops guys to use our requirements to build their infrastructure but this is rarely the case we also rely heavily on manual processes and each of these manual processes should be basing their tests on our requirements so this does not fit into a continuous delivery model this method does instead of having static requirements

defined on paper or defined in dead documents we have requirements that are actually self-verifying they're still in a natural language so we can still use them as requirements to communicate with anybody else in the team and they can verify themselves and we can use those to wrap things like scanners to wrap manual tests to wrap qa tests and to run them from in continuous integration servers and so on uh some additional resources on our github page uh bd security uses a few components that we've developed so if you want to use any of those components some of those are available there there's an ssl tester which will test ssl on the site like this support tls 1.2 it will check that

tls-1.2 is supported it will say that you should be patched against heartbleed and it will test that hard bleed is not there right so there are quite a few pre-packaged security stories and of course you're encouraged to modify them for the app under test and ideally you'd have this one set of stories that you could apply to a group of web apps that you're responsible for

questions yes your initial research i know it's always a contentious point sorry yes

yes

it is um it depends also on what type of tests you would run so by wrapping scanners we can do everything that a scanner can do if you wanted to do more kind of exploratory type testing you would want to wrap something like a fuzzer so if you want to find heartbleed proactively you would have to have some kind of super intelligent fuzzer that would go and uh scan for that type of thing and then find those type of anomalies although i don't think we would have i mean we didn't find heartbleed using traditional methods in any case

yes yes from a process point of view we can run security tests in the same place that we run quality control tests this can't replace a manual tester going into an app and finding new vulnerabilities what it does provide is the facility to record those activities and put them into your test suite that you can then repeat on subsequent retests and it costs you nothing to run a retest with this type of thing but yeah and i wouldn't say it's a replacement for doing manual testing now any others yes

yes yes so phantom js is a headless browser um i have read some problems with phantomjs that it doesn't the javascript engine is a webkit and is webkit right it's a webkit engine and it might not interpret some javascript on some sites exactly the same way that chrome might do it so at the moment i'm focusing on chrome and doing headless chrome with x v fb under linux so you could at least run these tests headless you don't have to set up a full x server to run them um but after that i'll definitely be interested in running phantom js it's also significantly faster than a real browser so that would be a nice plus point yep

so you can run it from you can run you can kick off the tests either on the command line or as a java file so if you can run a command line on the under cruise control you can run it under uh ability security good thank you thanks

you