5 Keys to Building an App Sec Program in the Age of DevOps

five keys to building a knapsack program in the Age of DevOps Jared thanks a lot bill hanging with us while we worked out some audio stuff guys big hand to the organizers by the way for jumping on that thanks guys very much before I get started so this is this is my Reese's Peanut Butter Cup stalk we are getting app second your DevOps and DevOps in your app SEC and they are two great tastes that taste great together my question before I get started is just you know so that I understand who I'm working with if you are currently you know kind of doing an app sect type job or that's the tribe that you identify

with could you could you show your hands okay we got you know a handful of people in the audience folks who are actively doing software development right now folks who are kind of involved in both a couple hands that's awesome so this talk is for everybody but I'm gonna try to you know kind of pivot by across those perspectives as we go sorry so this is me so it's timely I am probably the only Grammy award-winning product guy that you'll ever meet it's not my fault I was on a recording with the Boston Symphony that got a Grammy about seven years ago which is the one random weird fun fact about me that and my relationship with the Soup

Nazi so more pertinent to this talk I've been at very code since 2008 I came from the software development world and started doing app sec when I got to very code and I've seen a lot of different kind of interesting clashes between those two worlds over the years that have gotten more interesting not less as as DevOps has kind of taken off and started to build momentum and that's what we're really going to start with is to understand the essential you know kind of conflict of worldviews that we're dealing with because fundamentally DevOps is not about you know technology it's not about pipelines it's about getting people aligned behind common goals and then using you know process

and technology and other things to drive forward to the to deliver software that meets the goals of the business so this is kind of where we came from and where I think a lot of organizations still are is this fun culture clash between development on one side and and ops on the other right and I suspect that some of us recognize these worldviews because they're representative of how development and ops have traditionally been incented traditionally development is incentive to push change as quickly as they can so that the business can ultimately make more money right that's what it's all about where Ops is incented to keep things as stable as possible and traditionally change has been the enemy of stability

and what happened with the DevOps movement was a recognition that you have to align those two drivers and get everybody working at you know among the same purpose think of the whole business as a system before you can actually bridge that gap and start making better software faster right well with OPSEC you're kind of back in that culture clash because you may have bridged the gap between development and ops and you're you know deploying in minutes but if you're still looking at security from a perspective of checks and gates and things that have to you know be inserted into the the process and and that perfection has to be attained you know at every step of the way then you're

still in that culture clash and I would argue and we're gonna spend a little time talking about this that the job of security in DevOps is to enable the business to go as fast as it can as safely as it can number one and number two to address the traditional concerns of security because if you're very very safe but you're not shipping software as fast as your competition then ultimately you're not meeting the needs of the business either right so that's what a little bit of what we're going to try to dive into this this morning so you know the culture clash in development security goes deep this is DevOps days Austin this is probably the

most famous image that you'll find it's the number one Google hit for DevOps and security and Google image search and and I think that I've been among apps AK long enough security folks long enough to say that this is a characteristic of our tribe right let's let's engage in some introspection here you know we are all naturally very skeptical of technology movements that promise perfection because we know that technology is imperfect right and so you know when the DevOps unicorn comes along and promises you know rainbow swirls for everyone you know the the security professionals minds you know natural instinct is to think about all the ways it can go wrong so this is you know and

a manifestation of that culture clash right I will be the first to say that I have manifested the culture clash myself right you know I have been very guilty of saying publicly things like you know containerization is great and unless you're doing it with security you're an idiot which is not necessarily the right way to think about it you know you really need to be thinking about what is the business benefit you're trying to get to through adopting these technologies moving faster and how can you help organizations move faster to do it so I think it helps to look not at the way that the market of tools that that security that that even some

industry analysts define DevOps which is really limited to this vision of the CI CD pipeline line right and and that's kind of the perspective that a lot of people go to when they start thinking about DevOps and to go back to first principles go back to what Jean Kim talks about in the Phoenix project and some of the things that he then expands upon in the in the DevOps handbook and talk about what it is that we're trying to do here right and this is you know not intended to be a full introduction to what DevOps is go read those two books if you want that but you know just for the sake of refresher we're talking

about systems thinking where we're not thinking about dev as a silo ops as a silo or now security as a silo either we're thinking about the end-to-end process through which those groups work together to deliver value to the customers that's the first way systems thinking the second way is getting feedback feedback loops so that when something goes wrong and part of the process you get it back to the beginning as quickly as you can so that you can correct it the third way is continuously experimenting and learning so that as you you know continue to go down this path you continually find better ways to improve your process so that you make things better and none of this is new

right these are all insights from manufacturing from assembly line from you know Deming from half a dozen you know what would be considered boring old management topics you know if we were talking about this ten years ago but that all have a lot of relevance to how software is made so from a security perspective what do we you know kind of want to think about with these three principles right number one security is part of the system it's not outside it's something that happens during the process of software development to ensure that you are delivering secure functionality that can't be you know hacked or taken over by an outside actor it's also part of the operations

function because you want to actually understand what's happening to the code that you deploy while you're in production and respond quickly if something security relevant happens you're part of the feedback loop because if something happens from a security perspective along this pipeline you need to very quickly get back to first principles and help the folks who introduced that issue understand it and correct it and security has unique knowledge that helps people do that most developers don't understand the app SEC threat landscape they don't understand secure coding principles so Security's job is to help enable them to respond to these feedback loops and number three engaging in the culture of experimentation and learning so that everybody is getting better at and

getting better and so when you do that you can really start to think of security as part of this assembly line as part of this Factory and so what we're gonna do as we go through the rest of the presentation is talk about some concrete ways that we can take that idea and and pull it backwards but first I want to talk a little bit about talking about security with folks who are doing DevOps and practicing DevOps and help you understand a way that you can frame it so that they understand you know kind of why this is even a concern there's a guy that I work with who's his name is Jimmy Ostrovsky and and I'll spell it

later if if you're interested in looking this up it's worthwhile who spends a lot of his time talking with developers about security issues and he writes that it's a lot like man judging people through the grieving process you get you know kind of complete denial that there's a security problem you get anger at the people that ran the scan you get bargaining to try to push issues off the table you get sadness and throwing up your hands that security is futile and eventually you get to acceptance which is you know you're actually engaging with security in the process and the problem is most developers never get past the first two stages right so I think that part of the

challenge that we have with security and development is that security is a completely different sphere and we don't often talk about it in terms that developers understand and are motivated to think about so I'm gonna do something that I do with you know most developers who aren't working at a security company and talk about this in terms of quality instead because ultimately what we're talking about when we're talking about secure coding standards is building a better application that's more resilient in production but instead of being resilient against things like you know resource consumption and and you know misunderstanding business requirements and the sort of things that are traditionally the domain of quality here we're talking about resilience against

attacks counter to you know what a lot of apps tech professionals told me when I first entered the business I think developers actually do care about security in as much as it's a reflection of the quality of the code that they're writing right and so I think that it's important to make that mental shift and it's very important in the context of DevOps because DevOps has already gone a long ways to helping developers understand how to embed quality checks of other types into the code that they're writing in the the secure the software development life cycles that they're automating and the the reason that the quality is important and the other motivating factor that I bring to

developers is understanding of kind of the impact of you know the the overall issues I like the keynote this morning when Robert talked about the the you know the need to move away from high medium low and talk about you know security issues in terms of actual business impact this visualization is as good an illustration of this as any that I've found if you go to information is beautiful and you look for the world's biggest data breaches visualization you get this cool interactive Explorer that talks about recent breaches that have happened the bubble size is how many records were affected and the color code is you know what's known about how the breach happened what's the actual cause

and you can you can pop down a filter here and you know start to turn off things like you know something was was you know misconfigured something you know had a security bug there was a hack and and oops sorry there we go and if you go and do that and you turn off all of the things that are relevant to app sec you know here we've turned off configuration error hacked and for security and we've left on things like lost your soul and computer lost or stolen media inside job accidental publishing let's look at that again so those are the big bubbles right and they mostly disappear once you turn off the app SEC relevant stuff so this is your

connection from a quality perspective between you know failure to write things securely and the sort of outcomes that you know end up with your company being in headlines and paying fines to regulators and with lawsuits from customers right and you know that just to connect the dots one more time most of the security vulnerabilities that we're talking about that are implicated in these hacks are common coding errors that are present in most applications Berko does a study across hundreds of thousands of applications every year that looks at the prevalence of common vulnerabilities and what we see is 35% of all the applications that were assessed had a hard-coded password in them you know 32% this year and this is

data that was published in October of 2016 state of software security report are vulnerable to sequel injection allowing some sort of data theft you know twenty eight percent have open redirects so that you can chain that together with other types of attacks and a full 50% of all applications that we analyzed inclusive of applique that aren't necessarily subject to web vulnerabilities like applications written in COBOL actually contains some sort of cross-site scripting vulnerability that would allow an attacker to execute code in the in the context of the of the browser so this is why there we want to connect the dots and motivate developers to start thinking about preventing these sort of issues not after an audit has failed not

after somebody has come and run a penetration test on a piece of software that's already in production and told you that you need to you know to make a change but earlier in the process ideally as close to when you introduced the issue as possible and ultimately what we're after is trying to get to a partnership between the apps X side of the house and the development and ops sides of the house so that everybody is working together on the same airframe and getting things flying down the virtual airfield as fast as we possibly can so now let's talk about the practical how to's and I'm gonna boil this down to five arbitrary arbitrarily numbered

principles that we're gonna talk about well the first is around automation you know I said that devops isn't just the CI CD pipeline but a lot of DevOps centers around this innovation which is really you know taking automation and manufacturing principles into the process of building and releasing software and we're going to talk about how AB SEC tooling and processes fit into that we're going to talk about when you do that in the pipeline and our bias is to as early as possible but no earlier we're going to talk about the importance of managing the noise of AB SEC tools in the pipeline and and some of the issues that you can run into with

that we're not going to neglect the fact that I said at the beginning a lot of this is about people in process because we're going to want to talk about the role that AB SEC plays in helping people understand the findings and what do you want to do about the fact that there aren't enough AB SEC people in the world to engage with all the development teams in the more and last but not least we'll talk about the fact that this doesn't end in in development that you know because we want to think about this in a systems model we're gonna try to think about what we do with app sack in terms of production as well so let's start with

automated security as our first perspective so if the push in DevOps is to take that which has been historically a guild based craft based approach to building software you know I think a lot of us that have been in the business for a long time will recognize the thought that you know software has been built you know by artisans traditionally rather than in a predictable and repeatable way right and the same is true with how we've done security as part of that process we've had very skilled people like and I think many of you in this room probably at one point or another engaged in the process of figuring out how somebody can break an application which is awesome

and necessary and doesn't scale right so I'm not saying we stop doing that which is you know manual artisan like and really high quality in terms of breaking applications but I'm saying we find ways to supplement that in the process of automatically building software with as much that you can do in the way of automated testing as possible so that we can catch the low-hanging fruit we can make sure that you know obviously glaring things don't fall through the net and we can ensure that as we continue to build and deploy software more and more rapidly that we are doing what we can without you know contributing to the world's shortage of InfoSec professionals to to actually

address issues that may be inserted during the process by poor coding practices so what am I talking about well there's a whole technology landscape of app sect tools and what you're going to use is going to depend on a couple of things primarily the type of your application the the level of coverage that you can afford and how much you're going to trade-off in terms of speed for that and and and ultimately you know I think the the third piece is what your development practices are so you know from the top static analysis inside out looking at the the code that your developers have written to identify security vulnerabilities pluses you get very good visibility into what the

developer development team is actually written minuses as you miss out on deployment context and historically these tools have been on the noisier side of the security tools that are out there software composition analysis which deals with the threat of including out-of-date components and and never updating them again and therefore inserting known vulnerabilities that that attackers will go after in the into the software that you're using there's a new category called interactive application security testing which basically means your instrumenting the runtime and reporting from within the runtime when you see some sort of security relevant behavior which is cool but which also makes depends on making sure that you have full coverage of all the code paths in the application to

actually report the vulnerability if you're doing interactive testing and you never like go down a path that has a vulnerability and if when you're using the application the tool doesn't report it and then dynamic analysis which I think a lot of folks are familiar with in the context of you know kind of going through and scanning for vulnerabilities in a web application and that's you know a well-known very mature tool set a tools category but because it relies on crawling a web application in real time it's also one of the slowest of the automated technologies so what does that mean from a practical perspective it means we look for ways to do static analysis that are fast and low noise we

look for software composition analysis if we're using open source components in our software which most at least Java developers are doing we look for interactive you know it's an early stage technology so evaluate it and see if it makes sense for you and if you're doing dynamic try to minimize your your reliance on crawling the application and go for a guided scan of where you know vulnerabilities may be and use it more as a full application verification technology and maybe thinking about doing it some in a different way in the pipeline then gating your build on the results of that tool and we'll talk about that in a second it has to be automated not just in the running of the

of the application testing but all so in the way that the tooling gets invoked and the results get consumed which means you need api's and again we're not saying don't do penetration testing we're saying don't gate the release of your software on it you know it becomes you know when your software is changing five times a day and you're trying to do a penetration test against that moving target it makes sense to think from the business perspective on what what frequency actually makes sense to get those business insights about you know where your code is falling down I worked with a team of developers of error code where the the rule is you do a small scale penetration test every

time a new feature is introduced into production and you do a full-scale penetration test with an outside group on a periodic basis to satisfy you know customer requirements so a model like that might work for you let's talk about when you do this again we said our bias is as early as possible in the development lifecycle but no earlier so why is that well what we know from engineering is that if you don't sorry from manufacturing is if you don't catch errors as soon as they're introduced what you end up with is a situation where the errors get caught very late in the cycle they go backwards in the manufacturing process creating enormous disruption to the flow of the the work

product through the the factory and you know basically you know causing an enormous hit on your on your ability to deliver stuff and the same thing happens in the delivery of software issues that are found late in the development process long after the code is written are hugely disruptive to software developers and that's honestly one of the reasons why a lot of software developers really don't like security people because they show up with PDFs with 200 pages of vulnerability findings six to nine months after the code has been written you have to find the developer who wrote it attribution is always fun you have to figure out you know how to go in and make that

correction you have to figure out how little of that you can get away with correcting so that you don't totally blow your schedule right whereas if you were able to check for these issues closer to when the developer wrote the code then it's a much faster correction mechanism and by the way the developer actually starts to learn what it is that they're doing wrong and code for more preventive lis in the first place we talked with a our customers about what technologies to deploy and and how early to deploy them in the process I'd love to be able to give you a one-size-fits-all prescription here but the reality is that dreaded it depends and it depends because there's a couple

of things that are going on different organizations are at different stages of maturity in building out automation and building out this continuous integration pipeline so some folks may have most of the steps automated some folks may not yet have all of their quality unit tests integrated and you know there's a there's a hierarchy of needs with the stuff if you aren't verifying that the software is gonna run in an automated way then doing automated security testing is of limited value right get the get the unit tests working first and then start to work on the security stuff the other thing is that software itself is changing one of the things that we find is there's a push that's

complementary to DevOps to reduce the size of the the code that you're actually pushing through the pipeline so that you minimize the number of disruptive code path exercises you have to go through to figure out you know how to fix things that are broken so that's where the move to micro services comes in so if you have an application that is a four hundred megabyte Java War file I might have a different recommendation for how to do security testing in your pipeline than if you have a fleet of Micra services coming through that are like five megabytes in size or less so what are the options well kind of going from from the middle out the most common

thing you see people recommend is is to automate this testing in the deployment pipeline and the most common integration point is your continuous integration server Jenkins V STS even doing this in maven are always that are legitimate to go about the process and what you do with the results of the test depends in large measure on the characteristics of your application if it's a 400 megabyte war file and you want to scan the whole thing end-to-end you probably don't want to for that end-to-end scan to finish and before you decide whether you're gonna pass or fail the build you probably want to run that asynchronously and then send any findings that are found into the

defect tracking system that the development teams are already using and triage them alongside other issues on the other hand if you're testing small micro services if you're testing smaller increments of code you probably do want to stop the pipeline if you find a security issue so that your developer can fix it as soon as possible moving left one of the things that we're starting to see people do increasingly is arm the developers with the ability to run static analysis even before the code enters the pipeline which is much closer to the point where they introduce the issue and it allows them to learn much more quickly so there what Gartner is starting to call sassed light static

application security testing light tools that run from within the IDE to allow the developer to run a static test in seconds there have been some of these products in the industry for a few years now you know products like secure assist there's a second wave of those tools coming into the market now that if somebody wants to laughter words I'm happy to do that the are are they're really to help developers to test their code right at the time that they're writing it and to learn from any coding errors that they may be committing and then I want to think about pass moving right as well now if I've just said that we want to test as early in the pipeline

as we possibly can why would I be talking about testing in production well for a couple of reasons one reason is that there's sometimes things that happen in the in the context of the security system that you don't see unless you have a fully running application with data with all of the services behind it and you want to actually be able to test that whole system in an integrated way but you also want to be getting code out to customers as soon as possible well DevOps and see ICD have a pattern for this it's called feature flagging it's called Bluegreen deployments and basically what you do is you make a gradual rollout to the user base and

only roll out to the entire user base once you've reached your acceptance criteria such as monitoring for availability such as you know getting UX feedback from a small group of users you've turned out to maybe such as running a full scale dynamic test on a codebase before you push the the load over to that new branch so I think that there's some room for folks to think about what you know when does it make sense to actually do a full system test in production and use the blue-green deployment pattern to actually enable us to be able to do that in a safe and secure way so all of this only makes sense if you've got good data

coming out of the security tools that you're using there is a challenge with false positives historically in in the business and that's because again there's a clash between what security professionals need and what developed first need security professionals are historically terrified that their tool will have a false negative that they'll miss something so they turn the gain way up on the rules that are being run and the problem is typically what happens there is that you get a lot of noise coming along along with those extra true positives that the tool then reports so you know I think that the priority from a development perspective has to be making sure that every result that's reported is as accurate as you possibly

can make it because otherwise you're gonna be throwing up a hand and saying stop the the manufacturing line every time that a finding comes out and if only one and every two of those findings are real and actionable then what happens is that the tools get ripped out of the pipeline because they're not adding value and they're actually taking away from the business's ability to ship code quickly so think about false positives as part of what you want to design for minimizing as you think about how to integrate app second your pipeline this is not all about tools I promise that at the beginning and the reason ultimately is that the folks that I see in the room

and that are down there having a beer at the bar are a very small group of people relative to the need that's out there the data that the LinkedIn folks presented back in back a couple of years ago at at blackhat was that for every three InfoSec position every four InfoSec positions that were currently filled in the job market globally there were an additional additional three open positions that were just not able to be filled because there weren't enough qualified people so the way that you deal with this is that you build champions who can do some of the low-hanging fruit for InfoSec such as doing light weight code reviews such as answering question for developers who

are trying to decide how to do something securely and such as you know looking at something that may be an issue in escalating them to the attention of somebody from a security team and you build those champions out of the development teams that you're already using you designate one or more developers in each team to serve the roles of these champions and then you train them and there are you know mature training methods that are proven to work like computer-based training like just-in-time short training to illustrate you know kind of how to address secure coding issues but there's also practices that we've seen be very effective like capture the flag exercises and other things that are more experienced

learning for these folks but the important thing is train them don't just say you're a champion and then don't give them any knowledge about how to deal with that and what that means and last but not least security and AB SEC does not end once the code is deployed for a couple of reasons number one you want to understand what does happen if somebody is able to successfully breach your application and be able to address that as quickly as possible not just from an incident response perspective although of course that's hugely important but also because if something got through this process that we've built we want to learn from it and we want to plow those learnings back into

the earlier stages of the process so it doesn't happen again so what are you use for monitoring well we go from the most mature solutions like looking at your logs right to you know other solutions that are a little bit more fine control of fine grained in terms of what they can report web application firewalls for all their faults can actually monitor for suspicious issues and you can look for those they are they can be noisy because they're based on packet and there's a new category emerging called runtime application self-protection that sits inside the runtime such just like interactive testing does but it does it at production and you know you can run those and just watch for suspicious

security events again a new technology category but I would recommend you know at least taking a look at it to see if it will help you get the sort of visibility in production one way or the other you have to track what actually happens to that code once it's deployed so that you can get the learnings back earlier in the process as soon as possible so really my answer to where do you secure applications under development in a DevOps process is you think about it at every step of the way from when the developers are working to the build process we had a talk earlier about container security and and I think that you do need to think about the security

of the components that are being packaged in to you know all the way out to operations there are things that you can do in every step of that process that will help address the the issue of security and the beauty of DevOps is because it is this incremental model you don't have to do it all at once pick one get it working iterate pick another control and implement it get it working iterate and you'll be much more successful and if you try to take a Big Bang approach or if you try to concentrate just on one area this pipeline and I think that with that I am going to wrap and see what questions people have in the room

we take one otherwise thank you very much Tim appreciate it thanks very much [Applause]