Every Day Is a Zero Day: Building an In-House Secure SDLC Program

for all you non Harry Potter fans out there so today's talk is a titled everyday a zero day we're gonna be talking about you know your internal secure sdlc and I just want to say feel free to you know spit out any questions raise your hand or whatever if you have anything anything you want to bring up while we're going here I'll be all yours so a little bit about me you know just a quick overview I got into security about a little over two years ago I've really been you know pure security I've been drinking from a firehose ever since before that I did a software development for for 12 years you know remember towards the end of my

time is software development you know we started learning about what the security step was I put in this talking about implementing new functionality like our back that was kind of far into us at the time and it was only a couple years ago so the role based access control right so so you can see I think you know have been on both sides of this this fence over this discussion I have a good perspective on on really what you're dealing with when you're working with development teams for for trying to get remediation successfully so today you know like I said every day is a zero day and when you're you're working with your internal you know software development

teams that's kind of what it is right you're always scanning you're always euros testing testing software to make sure it gets before it gets out the door you find all the bugs so you're dealing with you know a zero-day right that's just a vulnerability that's you know hasn't been disclosed to the vendor yet right so now you're on the internal side of this so we're gonna talk about things like you know kind of how the development process works how you increase some security in there how would you add value to it right we're gonna do a value proposition here working with remediation team right we've talked about that I think he Matt during his keynote talked about this a

little bit right when you disclose this stuff it can go a couple of different ways you know so think about it internally when you disclose vulnerabilities are you met with hostility with your security with your development team or do they kind of accept these things pretty well you know a couple you know ideas on how to measure success here some metrics you know and if we have time we'll poke you to open for the for discussion there this is show hands does anybody have what they were considerest like an inhale secure SDLC program do you have security in in your steal see right now okay you guys do a lot of software development you have a

big development house okay all right all right a couple of building blocks on what you know ideas a couple different areas you can focus on to try to improve for the security here you know first one is really understanding your entire process and and try to determine where you can add you know some aspects of security to to those steps along the way you know from an STL C process perspective I talked about integrating the your remediation into into those processes and more importantly the prevention right let's make sure that the bugs don't get into the programs in the first place where the defect the vulnerabilities there's a couple tools out there or areas where

you can add tools to these processes to help a lot and then how do you add you know either accountability or more transparency about security risks so you know think about when you do find vulnerabilities who owns them where do you kind of store them you know do you just take like your scan reports and put them into a like a password-protected zip file or or spreadsheet or do you give them to the development team for them to kind of triage and plan out and figure out figure out how to how to remediate those things and finally I think you know always awareness is great so how do you increase awareness with teams and then also vice-versa

how do you increase awareness to your security team on how your development process works so all the development processes and every organization would be a little bit different so we're gonna keep this a little high level but I think hopefully you can be able to distill that down to how it would work with your organization's so you know here's the why of why we want to do this you know it cost money to fix fix issues defects in general vulnerabilities you know they can be seen as just another defect in the software program so so you want to you know prevent them or find them and fix them as early as possible right so if you look at look at this chart here

you know it kind of shows you you know if you wait until the end after the you know product goes your your application goes the market and then you find problems it's gonna be very costly to fix them so let's kind of push security kind of up that to the left there so we can prevent and fix the bugs you know before that you come through expensive here from a software development perspective defects just are a quality issue so you know let's try to keep those to a minimum to increase the quality overall right security quality you know it's just one little aspect of the quality of the application so I'm gonna take a quick overview of some

high-level SDLC stuff here and I just have to stop on this chart for a second because I was originally gonna put the more classic view of this thing of how Alice you know a tree swing is made and when I was trying to find a graphic I found this one and I think that this is pretty representative of security testing you know you look at this and from a developer perspective you take one look at this and go why would you set the thing on fire you know and as a good you know red teamer of course you're gonna try to set it on fire you tried everything else to write alternative analysis here so you know I

think that happens a lot when sometimes you give some of these findings to the development team and they say why would you ever do that with this program you know you get results like this so real quick you know development you kind of break down a couple green boxes on here at a high level this is what you do right you have an idea of how an application should work let's let's you know jot some ideas down let's say what does she do put some requires together it shall do that's what she'll do that right give it to somebody else and say hey I've got this good description of an application I want to do you know can

you figure how to make it so you get through to somebody to make a design and say okay well well have it do this and this and this you know then we give it to the software guys and they're gonna build it and implement it you know find out what what type of tools you need with libraries and everything then you're going to test and verify to make sure make sure it works and before you release it and then it gets into a meetin cycle so you know there's green boxes kind of show that the more classical picture this is what we call waterfall right you'd finish one step you go to the next one kind of like

water falling down a you know these steps here and that it's evolved over time you know so those green boxes if you were to just loop back around to the top of them and consider that like one iteration of development some of the new methodologies out there like spiral that's just multiple iterations of development you'll you'll see you know the idea of prototypes coming in you know trying to get to more of a you know working product early on so you can make some determinations of if your design is good you need to course-correct at all you know before you get to the very end and have to fix everything you know nowadays you know agile development with your

DevOps programs is you know much more popular too and really all that is just all those different circles are just more iterations right we get them down to smaller you know chunks of functionality to get a product or functionality out the door a little bit quicker you can also you know fix it or adjust it a little bit quicker you know and the part of this is brought on just by increases in technology in these areas you know continuous integration continuous tests continuous deployment and we'll talk about that a little bit later on you know finals for developers you know you get a lot of testing on on their applications right you know I have put a

couple on here I'm sure there's more I think we all know different different ones on here you know the ones on the bottom are a little bit more newer than than some of the other ones so you know security certainly very important for nowadays from an application depending on how that application is used but you know when you find issues with testing you're competing with all these other you know testing efforts as well so how do you prioritize those things you know how do you make your case for that the security bugs are more important than some of the other bugs they're you know are you had coming up against a brick wall you know do you have an open rapport

with with the development team to try to make that happen you know the other thing too is like when are you doing this testing right you know we talked about different phase you know stages of development and you know the different testing that can happen at those those stages and there's you know different types of testing to do when you know you might just want to scope it down to new functionality because it's new you want to make sure it works but you also want to make sure you didn't break anything in the process so you want a regression test everything goes the same for security but you know it's cutting in practice to do full regression testing every time

you have to put a new release out or new build so you're kind of you know taking a risk-based approach approach to testing here and you know the same thing goes for the security yet you know really you know just want to come back to you know these determinations of the accountability right who makes a determination for for how much testing should be done on the application especially from a security perspective is that going to be done by your development team or is it done by your security team you know who should who should own that you know and then we have acceptance testing I think you know a lot of people can figure this out

right you know we're gonna deliver this product we want to make sure we get a clean security scan on it before it goes out there make sure there's nothing on that maybe that's just an automated scan maybe you're gonna do a pen test as well you know and then finally you know we always talk about in with testing uh you know FreePlay you know you find stuff sometimes when you're not expecting to you know so you don't have really you're following a test script or anything you just something falls out of out of the program you know or you're testing on an ad-hoc basis vulnerability scanning if it's something that you maybe are doing monthly or quarterly or whatever that's

going to fall into that that uh that last category there because you know it's not integrated into the development cycle so when you find something when you're in that you know doing those scans regularly you know we don't exactly have a build ready to you know go ahead and fix it with you know might not have resources available so okay so so how do we add value to your software development lifecycle right you know we need to kind of justify why we want to add this cost to your stakeholders so that when they ask you what am I paying for you know you can kind of answer that question so what I have up here is this

actually diagram I don't know if anybody recognizes this this is actually Microsoft's secure development lifecycle diagram it comes right other documents back when Microsoft had had an STL still very valuable you know documentation there but you'll see the green boxes I made mine green for the same reason but you know the same five boxes for the most part are in here too so what they shows you you know what you can do at each phase along the way from a security perspective to really you know kind of add security right try to identify defects prevent them how do you you know document different things you know reviews you know different quality gates that you can go through to make

sure that you know when you're doing maybe your design or your your architecture you know you have a process that says okay when we're doing that you know we're gonna have a security aspect to it you know and it also has training from an awareness perspective in it and an incident response but a couple things on here that I it's missing that I like to add to I mentioned architecture review I think I'm gonna you know architecture review is very important it's something you know pretty basic you can do four applications every time they're gonna plan a release you know make sure they notify your security team so you can take a look at it so when you

know they're trying to deploy a program with you know out-of-date libraries you can kind of take one look and say hey maybe you guys should chit update these the other thing too is you know in the implementation phase they make sure they're using safe libraries we'll get into you know libraries a little bit later here but but there's are a couple steps to along with all these other ones over here right so adding some of these rates there's a cost of that you're gonna have that people resources time money to do that so when you get out of it right so so there's a benefit to that too and a lot of these are pretty pretty

obvious you know you know want to make sure bad guys can't get in you want to make sure your information can't get out you know maybe you have a compliance reason for doing this sort of stuff you want to increase awareness right when you tell the Velma team hey we're going to follows new process and as security in it well they're gonna probably be aware of a little bit of security you know maybe that'll pique some curiosity and we'll get more more involved with with you know on the security side of things you know and finally cost reduction as I showed you before right the bugs are expensive so let's find them earlier on so we can decrease costs

so so why do you need you know this program you know do we have bugs in our program right so so what I did was I took a look you kind of looked at some public bug databases to try to see what was out there you might heard these programs before but Firefox if you look at it currently 14 million lines of code if you look at going to you know Bugzilla dot moves org look for Firefox product and you look for the security component in there it's got 466 open defects in there so if you look at that currently that gives you a defect rate of one per every 30,000 lines of code so 30,000 lines of

code is really not a lot of code these days right I mean how many programs mature programs you know really that are you know kind of out the door products have less than like 100,000 lines of code in them then I took and these numbers aren't as meaningful but I took a look at the nvd national vulnerability database to see how many vulnerabilities there are out there against that product you know and I compared that to the lines of code so you could argue that you know the current amount of lines of code doesn't have all those vulnerabilities in it you know and I agree right some of them have been closed some of those are old ones but

it's just the idea of you know how do we you know what type of metrics are useful for our applications so so I put that one up there that just kind of as a thought process then I took google chrome did the same thing very similar I was actually surprised I had the double-check the numbers here how similar they were both similar lines of code one different open defect between the two programs from a security perspective so the defect rate is is in line thirty two thousand lines of code you know and a similar number of CDs so then I took G Lib C everybody who doesn't know what that is that's the library you need to run C++ programs in

Linux you know I took a look at data at that library so it's much smaller right 1.25 million lines of code 914 open defects with security flag in there so that gives you a defect rate you know of one every thirteen hundred seventy lines of code versus thirty thousand lines of code for you know web browsers you know but much less fewer CVS so I don't want to make too many us you know presumptions about what this tells you you could argue maybe there's still a lot of untapped vulnerability potential in that library or you know the other thing is the attack surface is so big you're only going to use a small portion of it so so might not be is much of a

concern for you but libraries like G Lib C you know your programs gonna use its gonna inherit all the vulnerabilities that your application you know that library has that your application now has as well so even though you might write a small program it's it's C++ you're gonna run on Linux you're gonna be running G Lipsy you know it's hard to tell exactly how many lines in there you're gonna run until you really do an attack surface evaluation there but but you need to keep in mind and this what I talked about with safe libraries is knowing what is out there from a vulnerability perspective that you're gonna inherit in right so these just go right on top of

anything you wrote in your custom code as well you know and then how do you fix these right how does your application fix these when these came out you know it's not like there's a patch available right you're secure your development team has to actually you know fix that they have to write the code they have to recompile it they have to rebuild it right and then what is that process you know as you a secure provider do you know how your your your company handles that process right you know would this warrant an emergency patch you know some state of out of out of scheduled release for for an application probably you know but but who makes that determination you

know you know do you follow the same process you would if from a functional perspective something like this would happen you deploy an application and it breaks you know what are you doing that case so for these you know know the get a tour info it doesn't have a logo but it's out of G Lib C so I put it up there had a CBS s score of 8.1 with an exploit ability of ten so so so how do you work with your remediation team so you know in case you're wondering what who your remediation team is for this it's gonna be your developers you know so so do you have a rapport with them do you have a

relationship you know that goes well you can kind of reach out and say hey you know ask them a question right try to be in front of vulnerabilities do you guys use this library when you see reports come in or ask them a question about you know you know do you have this problem things like that so I have a couple tips for how to kind of work with that relationship you know we're filled up that relationship I think you want to you know figure out how to have that dialogue with your with your security teams you know the first first suggestion I have is just like any pen test right you do your recon so you want

to know what type of architectures are running you don't want to give them you know suggestions for how to remediate something for the wrong architecture right don't give them examples of WordPress vulnerabilities if you're really running you know in a patch or for desktop guys if you're you know if it's dotnet run time versus you know like an eclipse Java run time make sure you know what to do there understand their process so that you know when you have vulnerabilities you need to disclose that team you have a good idea of when they're gonna get fixed right if they don't have a build planned for maybe six months or a year maybe something that has an annual release and

you find a you know critical vulnerability what are you gonna do about that you know understanding how that process is going to work on their side you know how do they take defects you know did have a change control or change review process that happens where they triage and prioritize all their different defects to figure out what the fix you know try to get into that process you know and then I mean these are just pretty simple recommendations here don't add more work with false positive reduction I think everybody knows knows what to do there because that's just gonna come back you're the last thing you want to do is for anybody when you can somebody report with false

positives and it you know it's going to be more work for both you and it's gonna kind of you know weak in that relationship so you know especially when you're doing it internally you may think well it's internal I don't have to worry about putting on such a good spin on this thing you know I'm just gonna give them the whole raw report well you get to you know development team and they might say oh we need to fix these vulnerabilities and start looking at them and they go you know we can't reproduce this right and that's something that most developers don't like to see when they're given a problem report that isn't real it's it's just a

waste of time you know and then make sure that the rel you know recommendations are relevant I'll get some examples of this but that goes back the architecture a little bit you know understand the context of your application how that works you know and you know if you don't know then ask the development team say hey how do you go is you know do authentication right you know we found this bug but we don't know if it's legitimate or not do you use LDAP right maybe my LDAP injection is a false positive because you guys don't even use how about something like that so here's one example you know don't be wrong I love burp as a tool but

sometimes the reports out of their their scanner can leave a little to be desired because of confidence so you know if you were to just give somebody a you know automated report and you have a bunch of tentative confidence in there you know you're kind of you know rolling the dice on almost gonna happen out of that so I put this example up here just a little backstory on it so what happened while the scanner was running was it runs the first request right and it comes back and gives a response it runs the second request it came back with a with a 408 response so that's a that the thing timed out so in its you

know logic it says these are different I'm gonna flying this potential SQL injection response so but burp is at least nice enough to tell you you know and I put the green box around it that these results are unreliable so they need manual and verification you know so make sure you do that second example is as I talked about make sure your recommendations are relevant so this is right out of a wasp's web site on preventing SQL injection so just show hands does anybody ever try it option number two on here you have okay you know why you had to go with that with the stored procedures or in our do you use towards procedures all right okay

all right that so typically you know if you're already using stored procedures that make sense but you know I haven't really come across too many instances where that's the case and it's more work so usually option ones just makes sense so so make sure these these recommendations are relevant before you know before you just put them in in your report the last thing is I kind of looted this a little bit but but when you get all these vulnerabilities and you get a big report what do you do with them right you know I talked about the transparency so you know the best things you can do is get those defects into the right systems so

whatever system that your developers use the dev guys whatever system that your operations guys use your office your office guys make sure you're using the same system right come up with a severity mapping between what their system does and what your tools do from a criticality perspective you know and then and then enter the these defects are these bugs and into their systems so what's gonna happen there is that when you know an application team or development team's gonna look at their application and see what you know kind of their their defect you know totals look like the security stuff is gonna be in there you know I know I've come across a couple instances

where it where this doesn't happen you know you kind of kind of keep your SEC you know separate set of books on vulnerabilities and it's tough to kind of see the big picture here from up from a risk score perspective or you know enterprise attack surface because you're not all in the same system so get them into the same system whatever whatever that is you know and keep tracking in there right so the security teams if you want to look at the the sdlc risk you know run your reports out of their systems it makes makes a lot of reporting easier and it also like I said the transparencies there you know and it helps you build that case towards

accountability right the dev team can't ignore these findings if they're there in their defect systems yeah so that when there's some considerations here is that when they're trying to plan their next build or release you know from a budgeting or or or a schedule or a resource perspective you know those those bugs are already in their in their backlog all right so the other thing is you can empower your developers right let's give them some security tools that they can use here as well so you know typically development teams will have you know an integrated development environment ID you know a couple popular ones are like eclipse or Visual Studio those come with with tools Visual Studio

has you know a security rule set you can turn on and we're just going to do is kind of going to check the code in real time or at compile time kind of flag some some security issues before those can really leave the developer and get into your code base so you know something you want to consider you know turning those on for your developers you know and then you need to support them too so so after you turn them on you know you're gonna have to work with your development teams to you know make sure that the the ruleset that you have is good you want to have the right you know all the different of the

profile set up properly so that they don't have the false positives in there so it can be effective for them you also don't want to turn off too much stuff you know lower the sensitivity so you're not catching everything either and then you know we're talking about I took a little bit about agile and and some DevOps stuff those really come around like I said with with automation so you know there's a couple tools out there Jenkins is a popular one chefs another one you know there are tools I mean most of the scanners won't really scanners you can integrate in to to your development processes right you know we have a kind of a build pipeline up here

you know you build something you want to kick it off to it to a bill first thing it does is tests it right well run your static code analysis there make sure you get a clean static scan on the thing and if you don't you know maybe you want to have a rule of saying a process you know all critical and high vulnerabilities need to be remediated right and if you find any you know break the bill right there's bill breaker plug-ins for a lot of security tools that allow you to do that you know so that the developer is gonna see a failure in their bill pipeline they're gonna have to fix it before they can

really promote it up to to another level after you build the thing you deploy it to a test environment run your vulnerability scans on it you know run the dynamic stuff catch it they're the same thing let's make sure we get clean scans whatever that however you determined that to be for your organization okay so so you're trying to measure your success your project a little bit here you know a picture of Grace Hopper she's credited finding the first plug you know I think if we followed this process for security bugs we'd probably run out of tape pretty quick but we have a couple other tools we can do so one things I did is I don't

know how well you can see this but I took all those bugs and plotted them for Firefox over time so with this chart showing is at the bottom there in the orange line at the bottom every month that's how many bugs defects security defects were opened against Firefox the green line is how many were closed so you know every time that the orange line goes above the green line you have a residual number of bugs that are in your tool right that's the blue line so over time you know your your security risk of your application will grow if you're not fixing everything you're finding so if you look at something like Firefox you know those

numbers are you know 30 40 50 so this is kind of what I meant when I saying like every day is a zero day right you know the security guys are going to be dealing with an application like this and they're gonna either be fixing a bug you know getting that green line up or they're gonna be dealing with a new new bug from a security perspective here you know this is a great chart to start charting showing across your organization take your vulnerability scans and show over time how they increase you know to try to get backing to say hey we need to do something about this yeah so if you look at the end of

20 the end of 2010 here was you know for some reason they took at the end of January they they fixed a hundred bugs in in Firefox you know it seems like something changed their processes change their security and got more involved something happened to really I think you know kind of corral the security issues in this product right so when you see there's you see something similar to you know more of a plateau there you know and I say plateaus it flattens out because you know you got to think about it the code base continues to increase so so the you know the ratio of bugs per lines of code it probably stayed flat or

it could potentially you know decrease with this chart so I'm gonna dive into this a little bit closer here I'm gonna break those out this there's open defects by severity so so if you see there you know the top line there's it's categorized as normal right so I think Firefox as an application decided we're going to accept the risk of normal bugs you know we don't need to do anything about them they're no cause for alarm you're going to take some of these other higher criticality ones you know that the major is the miners and we're gonna you know make sure we keep our arms around those and we don't let them get out of control so so they filter out a

couple of those and we just look at the major minor and trivial x' you know you can see at the end of 2010 Firefox you know they really you know got their hands around security you know they made a concerted effort to decrease you know a lot of the more critical bugs in their in their tool and they kept them you know they kept improving up there they're doing a good job and I don't know if that's because their developers got smarter so they're doing more prevention they just got more active on remediation I doubt it's because they're doing less testing you know but you know taking a chart like this and and kind of

you know showing that for your organization you know it can really tell story here you know at what point do you get your hands around the security of of your applications right you know and this doesn't need to just be one application right let's put all your applications on one chart show your enterprise risk score right you know and then you can start pointing fingers right so you can say you know we have this problem application over here it needs to get fixed it's really driving up our risk score you know but now you have some data behind it so this date is really easy to get you know at a bugzilla you just run a CSV report so

you can show it same thing for for you know most the other defect tracking tools out there the other way to kind of measure success is to determine the the maturity of your program so a couple ways to do that I there's a lot of ways to do that but plenty of frameworks and everything I just have two that I called out here the one on the right for anybody recognizes that that's a that's CMMI you know it comes with five levels typically I think you got to figure out where where your sweet spot is you know from a process perspective you know where are you now where do you want to be you know take a

look at it on there one thing to note about CMMI is you're probably not going to be able to get us you know Athena scored higher than your development team Falls just because if you're trying to integrate into the SDLC processes you're kind of stuck with whatever maturity that they have as well the other one on here is is open Sam the software assurance maturity model it provides 12 different categories you give yourself a score of zero one two or three you know and you can kind of you know score your current processes then you can take a look and say where do we want to be right and you can kind of you know

put that roadmap together of how to you know increase your security you know do you need to add more processes need to add more steps along the way the SQL see you know and and the things you need to look at when you're when you're trying to determine these things you know look at your artifacts you know you know are you even doing these artifacts you know and then look at your processes you know where where do you have security in the in your program do you just use you know all flight scans at the end you know do you make sure you you you know you have some sort of pre-production deployment step or are you really fully integrated

in all the way up in the design and you know the requirements phases here okay so I'm just gonna wrap it up here you know I just a couple thoughts here you know when I say every days is your day when you're working with your you know securest you'll see is working with your develop teams it really is you know you want to make sure you know it's no different than you know any other zero days out there you know they probably not as high-profile but you know for your organization you know for your brand reputation these things to be very very important so with that being said um you know right now to take any

questions on anything all right all right that's all thank you