2017 - Security Testing As Part Of The Release Pipeline by David Brownhill and Craig Scott Angell

hi everyone so I'm critical angel currently work as a lead software test engineer I work in the textile industry in Edinburgh my colleague David Brownhill worked with me a company called Skyscanner where we got into security testing which see the natural progression from quality software from functional point of view to cleave the security side of things we're gonna go through a couple of things that we've been doing since then with two examples of how we've automated security so I've worked for a long time in software testing notably the finance defense sector very often companies have worked for being reactive we try to switch out to be more proactive in how we're testing for these things and also

with a move to agile in a lot of companies now both non security testing and the project does not work so you need something with faster feedback manual testing isn't always feasible during the development cycle we typically work in one-week sprints recording two-week sprints in the past to get manual security tested in each of those for everything you're developing it just doesn't work so automating things is the natural progression so there's a couple of things we've done the first thing which is probably more important than the technical aspect is to build a security aware and security conscious culture and then the shift in behavior and thinking it's probably the first step other way you don't get

buy-in and these projects go left on the site we include everyone so as well as the engineers we talk to people higher up in the company we've talked to other departments and we try and get sort of people to be aware what security means what the risks are on the industry and get buy-in that way and then we build it into the methodology of how we work and the key is to automate iterate and automate some more on how that just running continuously to give us feedback so the educational side so usually it starts off with knowledge training sessions talks he's build up to workshops where people can attend try things out and then we use try and roll

up some so online training people don't always like to do stuff forced upon them it gives them a chance to do stuff in their own time but one key thing that we found works is each engineering team if we can train a security champion that pushes the stuff into the team into the methodology a point of contact for the company that we see better uptake in automating of security I'm following secure coding practices so we leverage different tools to do this static analysis time become our student security tools and what we do is we treat everything that comes back from these just like any other functional defect that we find we categorize things by severity high medium slow and take

them into each sprint based on high is first medium second and just work away through that list and start fixing things up so there's a couple of things that we've used in the past and looked at then various trials with there's a framework called BDD security mitten with burp suite integration gauntlet and I will use other dynamic security scanning tools retina scanners open source we've also used paid services as well often in the tech so startup industry we train golden source but paid services are the experts sometimes that's the right thing to do as well so one of my favorite the frame workers be the security we use the heavily where we're at sky scanner this allowed us to

quickly build an automation framework as it comes with a lot of scenarios predefined uses BDD gherkin syntax given when then it's a high level scenario description so people outside the technical areas can also understand what we're testing so it comes with a range of tools or slap is probably the main one it uses to drive for a lot of the testing but also use a selenium driver for the web interface extendable to NASA security scanner the one we used was version one use J behave in Apache ant the current version and implement now my current company is version two using cucumber and Gradle the good thing with this as well as the non-functional security aspects

we get the test the functional say we can test captures limiting login attempts and various other things that we can drive with the selenium driver in the web front-end we find this is beneficial because very often those have to be in separate Suites but we can have everything security focused together we also managed to integrate arrest API for testing arrest client we can test our API in the same manner where these scenarios we can with anything else so very as I said many stories are predefined I'm not gonna read off every one of them there's a lot more these you can go and have a look online yourself but you know there's quite a lot of the

Box it also comes with a sample application just to try things out before you even start implementing anything so snare user said in the stories we are feature files which has a narrative is essentially a collection of like-minded scenarios to test an area of an application and if also given when then so we've got an example here for open ports given some prerequisite when some action happens then some assertion they're pretty straightforward and very easy to to write and these link then in the backend to the automated automation scripts that drive these tests as another example here with two scenarios in cross-origin resource sharing this was one actually with an intern that we fed back into the open source framework

again on the education side we've gone into an involve got education he said he didn't do much security at university in the software course so it was a another progression to try and get stuff in the industry so here's an example of how we did that from a pipeline point of view so our source code is in git repository we use a continuous integration server we use teamcity at the time we're using AWS as a service as well hopefully you can see that clearly there the team said the value ansible was pushing a package up to an s3 bucket we have an instance we we only switch on when we need it to run these tests saves us some money because

we switch you back off at the end target our services run the tests and pull the results back into our integration server as an example there with some failures the good thing we're doing this is our pipelines can then be blocked from release into production and we find that then people jump on things because they want things out so again if you highlights the need for security the value we find with this a lot of the early tests in faster feedback so every week you know we're developing new things as part of that we're writing new tests we're running those tests I will get feedback straight away the thing with that is it also has

a feedback loop as we fix things as we find them people are applying that knowledge and not making the same mistakes as we go forward but ultimately it gives us confidence in our products and it reduces the risk when things get into production we're not going to have some sort of incident happen it doesn't take away from the fact that we also need to have penetration testers come in at some point but rather than pay a lot of money out for them to come back repeatedly because we've constantly going issues we get to a point where we've got a good confidence and the things that come back aren't big lists very often but based on that expertise

they do find some things you don't think of which is good I was why you need them I'm gonna pass over to David Brown he'll who's gonna go through another example with gauntlet and the way he's used that so these use different tools curl and map s lies sequel map etc I will just switch over for David to go through some of that

thank you Scott so nice work with a traded term Skyscanner and this way we initially we've been getting them to applicator secured as part of the continuously with Piper so since then of moisture on government projects did we pay and it's just our X D BSA and mostly working on social service service teams so from that perspective it means that when stories are coming interest sprints and there's also an innovation we're going to be assessing a user story from planning what criteria is required from application security sesser management co-chairs managing the user ID and also performance as well so those two major areas trying to push those into the pipeline and alongside the unusual unit

integration and functional testing so some of the digital service he did he said I don't know like with you sir but we were split across these three areas topic development architecture tech support shopping development of teams creating you know features from out of clients from product owners and the in Sydney we have an architecture and we are obviously saying here performance and security information assurance that's all about did protection was managed separately so we're trying to get these clients into functional features take the support that's obviously prosecuted working as well and it's not functional testing final wars and network access and it we have access to director father's so continues delivering flow looking set up the

factory Plaza trip was was coming out of my features and then we put you from left to right this is a man testing mango gates initially obviously the less mature but apparently Accords look mangled it's so look it's a creators talking early about a lot of secure testing we used to do manually using tools like both routes and Zach s otherwise we're just doing that specifically within the feature teams so you've got it symptoms prior to and we'd actually just use those tools and Magna verify that they were correct and so everything once read we'd have the external penetration testers for me we would hope that they would have spent so much time going to the whole broad range

of security security protocol yes it obviously did a full range but your tribe because of the low-hanging fruit they can sort of get for those quicker and these and opposite more funding niche cases and that way and that's where we were trying to put in a keyword into the pipeline because obviously we did manual chest fruits of the stories was a time-consuming because months were called bombs and the security so spend a lot of time with writing Jamie describes that matters app scripts push with the pipeline it's just a left or right so we this is the odd journey for continuous integration so as you see this the stop started motion always target the function tests extend period from

the bottom approach you began the union's integration converted contract in UI and then version 2 what what look at the gate spot my static analysis it and this moves first started doing our security testing so sonic qube it's a tortoise of fi these functional practice of coding national development teams between also as performant code and secure codes or the Tsukuba curves actually what comes out the boxes minimal that said but gives you foot football which I had added bangles and hook it plug it was from children your choice so easy because psychoanalysis insurance fashion next stage food then 2.2 from arts to do this integration implementation plan is so we analyze our code that is security written and

performance and then offer that these two working satisfactorily in push in on test so we know from some test service we would take people are Peter teams Jaime de scripts and the fat pockets that's one patient review till we were then sort of Victor fit those into the pipeline source of it they weren't just winning as so it was obviously ruined finished aims with what is this Rita over night so some sort of in the true key dates perspective and the neighbor ascending on Pacific Street said we wouldn't do data nice and fast so good feedback and false root overnight as well so we had attempted an empty environment in the ministry to spin up except for publisher there - I'm

short of testing diamonds for for much aggression and applications hence I went to my teaching and code for Superman's sonic qube yes is the words that we had added into our subsonic analysis by the pipeline so Olaf top tip I think they've been published later annually but there's some gently lovely send Jason to be tough by the set and so the easily remote was having plugged into our Sonic you put the code data object perform specific injections a big one now we use gauntlet so we talked about BDD continued security to frameworks like selenium when you use inspect flow or there's a DSL test his language terms like gauge and Concordia which use markdown quite a

cool going to it's similar in a sense that you find it's your tests are with range action certs of just pull this one out [Music] and just ensuring that all the cookies are said the property manager journey and I mean their abilities which can be exploited properties that we this script as part of the firefight ensure that all cookies at magically production code obviously we wanted Dave to write codes securely so as part of [Music] the process of developing this we have code reviews with teams technique so the 51 EV articulated or received developer and they be responsible for reviewing all code that's written so it has to be written in so recommended by engineers

but we'd also have a checklist for code to be performance and secure which we developed in has probably industry standard guidelines these rule six defined that we're trying to automate them into analysis sonic qube and the tech vectors here is having finally also defined the we could set up the attack vectors that would likely to be important to be exploited and trying trying to do there is on sort of bit up workshops where we never have engineers trying to area where users type vectors and in style sites this is sort of tech time exercise yeah it's regression execution server just like a functional regression you're then regression Suites against new functionality sort of previous function hasn't been broken and

we'd also be written as as part of our security so people have made changes to have a sessions being managed that define the previous trading business rooms important and we're talking about the attack vectors would be several tech notes that meet where we'd be to the manually exercising stuff against Arsenal first accept and P reduction bands and we could also but having defined those people keep up we'd also had those and - - such that he would actually want some vectors to do this okay Commodus no she did not function requirement ii environments for the volume and ability tests and so we've got the day producer general description of Deb's writing proxy active status so they'd be right

about the experiment enough i'm seventeen we would interview those into the test suites and the tax law is that i would come out of planning so just like they did with test scenario since criteria and you define acceptance test and pleasure at the requirements for that we'd have a textile very split identify view implant and then we flesh out how we define those as vectors and that's it I think Christians any questions yes I'll be one for you David yes there is the personally know if you're interested decision that came at some of the structure are the engineering team and as a kind I mean I've dinner with they'd obviously they had to approve content on different

options but then I joined sonic qube was the factor and that's what we were working with so I don't know why they chose that one in your experience have you used other options [Music]

yeah I wasn't privy to have they came to them afraid I guess when we worked together a Skyscanner we did go through proof of concepts we tried to view these frameworks we also tried various paid services we tried a couple of comment with name of them off the top of my head but we ended up going with Verico there's a security scanning option for spam egg analysis but we went through you or three months or trials with different different software what are the pros and cons I think we used well we part of our pivot concept was with white have security yeah we tried my hat and they were quite good from the support level for mitigating flaws have

reported yeah [Music] expense I think it always comes down to

something as a test engineer you get all the time particularly like functional tests for instance with selenium as well take a long time to run the way we get around that is those type of sweeter innately if there are failures they still block the release if we want to release the next day until we get a green run through but they're not part of the flow through to our pre-production environment so we've got various we've got like a development environment integration environment pre-production and then out to production so that's how we do that it also wasn't left why we switch to develop specific scenarios yeah so I mean the company I'm at now we use

Arachne scanner we run at weekends because we're on weekly Sprint's finished the sprint run a weekend scan we release on a Monday but again fix the weekend to run sometimes you've got a large application but the specific scenarios allow you to target a bit more so you can instead of run in your scan that will spy your out and find things and test the same things laborious ly you can tag those scenarios so if you've only made changes in one area and you know that you know there's a risk of something you can see by this tag these scenarios and depending what integration the server using if life you're doing stuff per commit or through a pipeline

you can commit with certain parameters to only initiate certain scans so from my point of view we aren't using the framework for the sequel injection we're using Arachne scan of the weekends which goes through an application and there's you can have a whole suite of different attacks that it does for for that it's not something I've looked at specifically you mentioned on your slide about it so maybe you've got a different answer similarly to what you say sudden cube women actually wouldn't said it move says and then some of these that come at the box are our signal injection and the direct object object reference they just come in with a whole load of sweets of

tests so similar to acne you just just wanted overnight you know we wouldn't define specific attack vectors for certain injection because there's so much out there just comes out the box yes probably worth mentioning as well they you still need manual testing to pick some of these things up these tools aren't foolproof they can't false positive and miss things so we're making false positives there all our pipelines which add information functional regression pipelines if they haven't made something release for much aggression pipeline that will introduce some of these security pipelines something Lisa that's

because yeah so what winds going to continues to live you just have the performance and focus all scheduled meet one separately cuz I think we do this guy's gonna to yes any marks so not at the moment but my company is looking at moving to using docker for a lot of our software and particularly for our highways so it's probably something we will look at but not at the moment