BSidesSF 2018 - The SecDevOpronomicon (Clint Gibler)

[Music]

great thank you very much actually before we get started if you all would be willing to indulge me for a moment I would like for us to get in this shenanigans together so something I've been doing the last couple of talks I've given is start with a total non sequitur and then go right into the talk so that when they edit the video and post it online it sounds like I was just giving a very detailed intricate story which in reality never happened it just sounds like because you only catch the tail end of it so if you won't mind indulging me I would like to do that now it'll be our in joke together

and the people who are organizing this they can see this part of the recording and can cut the non sequitur if they don't want it okay and that's how I broke the curse where I could only speak in rhyming iambic pentameter all right let's get started thank you all so much yeah but let's get into it so my name is Clint Gibbler and I'm a security consultant and research director at NCC group we do penetration tests for web apps Network mobile hardware crypto pretty much anything related to security I'm originally from the Midwest and I realized upon moving to San Francisco to get anyone to listen to me I had to dress like this before

that I was a security engineer and before that I was a PhD student at the universe University of california-davis where I dressed like this okay so what is this talk it's basically a collection of specific tools processes and automation Pro tips for how your organization can scale their security efforts basically the background of this is I talked with a number of friends colleagues clients at a number of different organizations and asked hey you have just a couple of app set people and many many engineers how do you deal with that what sort of processes have you put in place that you found successful and which ones have you tried but weren't as successful so basically

this is a collection of pro tips and lessons learned for things that companies tried and found worked as well as some things they tried and what was effective so another reason I'm giving this talk is because I'm hoping that afterward a number of you will come up to me and say hey my company does this you didn't talk about it like this would be a cool thing to do as well I would love to hear about it please say hi and a number of the things in this talk are actually from earlier version earlier versions of this talk where people came up and said oh hey wait have you thought about this and I hadn't and

now I'm sharing it with more people so that we can all get better together okay so I'm gonna talk about a number of specific things but at the end of the day every organization is unique some of these may not make sense at all so adopt what's useful and feel free to discard the rest and at the end of the day what's important is the mindset and methodology of this more than a specific tool or process so I've been talking with a number of individual people to hear what they thought and I've also been doing some literature survey in terms of you know companies writing blog posts about what they do as well as white papers and other conference talks

so basically I'm trying to take everything that people are talking about as well as some things that aren't discussed publicly and put them together into one thing that we can all learn from together okay so first a little bit of background let's talk about agile and DevOps so as all of you know I'm sure there's this gigantic pushed towards agile development processes and the idea of DevOps so releasing say daily instead of quarterly and having infrastructure as code there's a lot of benefits to this you can empower developers and Ops to move more rapidly to release features and be responsive user feedback while at the same time this also brings some challenges to the security team so in

pretty much every organization I've ever spoken with the security team is outnumbered say 50 100 200 to one and they can't possibly review all the code that's getting pushed out very quickly while at the same time security cam P blockers write product in most organizations needs to go forward we need to hit certain timelines and security can't stop this except in the face of very serious issues so one resource that I think points this out really nicely is Tom Daniels of Square wrote a blog post which says essentially you know hey we looked at it and as the security engine as the engineering team is growing rapidly we can't manually test a monitor every new service that's released so therefore

we need to take additional steps ideally automation that is going to scale our security efforts as the engineering team grows so if you were to condense this whole talk down into one bite-sized statement how do you scale security and we're going to talk about this in more detail but to give you a mental framework for the rest of the talk I'm gonna focus on two core ways so one is creating libraries and systems that make it very easy to do things securely and then adding automated that is static and dynamic analysis checks at key points in the SDLC so before we get started and no matter if you have the best ideas in the world at the end of the day you need

buy-in from the parties involved and while this deserves one or more talks in itself I thought I would at least mention it briefly so if you are like hey this seems like a great idea I think we should do it in my org here's some tips and there's probably other things that are even better to do but here are some examples of getting a couple different groups on board so for security management and other executives you can pitch it as hey let's this has nice cost it means benefits you can do more with the same headcount it can scale our security team as the dev team groves and we want to maintain the security baseline across current as well

as future services for security engineers you can pitch it as hey let's automate the boring things and spend more time on higher leverage better ROI things and ideally also protect work/life balance and developers are also very important to get on board because ultimately they're gonna be the main people that deal with these new security tools and processes you're rolling out one thing that a number of companies have found very useful is as they're releasing these new processes they need to have low - additional friction - how developers already working or else I won't be adopted also if you're trying to send alerts or tool output directly to developers these need to have approximately 0% false positive

rate or else they're gonna be mad at you and if you're say making a secure rapper library for something or you're trying to change how they do things you to make it as easy adapter as possible so if you have say a secure XML parsing library have make it have the same method signature as the one they're already using so they just like change the class name boom they're done another way is if you build in something developers already one you can say hey not only if you do this are you more secure but also you have this other thing that you want a sort of orthogonal for example telemetry so it's like hey this is secure but also you now

have some performance metrics or you have a better idea of how people are actually using this in production which they may find a nice driver and also while most developers you know want to do the secure thing you can also pitch them as hey wouldn't you like that more robust or better code quality which they may find a driver as well okay so we've talked about who I am some background some appropriate memes and the focus of this talk the majority of it is going to be talking about the how so quick disclaimer I'm gonna reference some specific tools and products this is not an endorsement I'm going to describe some specific processes this isn't the

best and only way again what matters is how your org works and this is just one specific condensed talk so I can't possibly talk about everything just because I don't talk about it doesn't mean it's not important okay so at a high level these are the core parts we're going to talk about specifically we're gonna talk about with regards to security engineering some libraries that can be built on developer machines as well as in code hosting such as you know github bitbucket or jenkins or other CI CD pipeline type stuff doing static analysis and in the test or QA environment doing dynamic analysis and or fuzzing and then monitoring in production so this is actually a 50

minute talk tried to boil down to 30 minutes so we're probably not gonna be able to cover all of these but rather than just delete the slides I've kept that content in there and I link to the slides at the end so that you can read through them later okay so in my opinion one of the most high ROI things you can do is building libraries and tools that are secure by default for their development if we look back a couple of years we can see that this has had big wins already specifically web frameworks now by default do a bunch of things securely specifically in earlier version of rails whenever you are outputting to the view

you had to include this each method that output encoded that and when you have to do something to be secured rather than it be the default you mess up right there's like thousands of places throughout the front end where you're outputting you outputting user input and getting every single one right is hard for anyone so since these frameworks have gotten better we can see in practice coming from a security consulting point of view we see a number of classes of bugs just categorically going down over time because of better libraries and better frameworks so if you're looking at your organization and thinking okay where should we spend some security engineering time and effort there's many areas but I would encourage

you to look at some of the following anything regarding managing secrets crypto authentication authorization sequel file system access anything that shell exact etc those are good places to start one interesting example is one company banned of the use of md5 you just can't use that standard library function at all and instead they wrap it with this function called non cryptographically secure md5 which developers can use so the point here is yes developers can still do that but they have to explicitly acknowledge you know I'm doing this and I don't necessarily expect to have cryptographic randomness so here's some specific examples one company has a test case that will automatically check for missing access control I think this is

pretty neat so it programmatically enumerates all routes and then int respects on the code in each route and then warns and fails the test case if no authorization check happens so they're using a given library that the authorization check has this function name so they can basically programmatically say oh you aren't checking authorization at all fail and they did find in practice that this found a couple of times where developer added a new endpoint and then didn't do this again this isn't perfect because it's not necessarily ensuring this is checking the appropriate things but at least you're checking if it happens at all other companies made a battle-tested view library that largely eliminated XSS or a data model wrapper

library that everything whenever you talk to you the database essentially that eliminated sequel injection for the past three or four years they said so the core takeaway here is if something is hard to do securely if you have to be aware of this and that threat in order to not shoot yourself in the foot just build is secure by default implementation and enforce everyone use it okay so moving on before we get into talking about some of the lightweight static analysis I wanted to do a brief diversion so at a high level just to make sure we're on the same page static analysis is raising about code based on looking at it that is not running it and then dynamic

analysis is running code and observing how it behaves so regardless of which tool you're using there are fundamental like theoretical computer science limitations to both of these in that static analysis has high coverage and that you can see all of the code but it tends to be imprecise that is there are false positives in dynamic analysis code coverage is a challenge so you may not necessarily execute all functionality within that program but it tends to be more precise in that the issues it finds our true positives again these are sort of hand wavy high-level statements but they tend to be true and there's also a range in complexity for both static and dynamic analysis tools you have more

simple tools and they have more heavyweight advanced ones so when I talked with security engineers at a bunch of companies and said hey D do static analysis what do you think this was almost without exception what they said and this makes me sad because I think static analysis is awesome but I think what happens is people buy this expensive tool and they get really excited and and they think oh I'm gonna run this on our code base we're gonna find all the bugs and it'll be awesome but then in practice what happens is they run it and then this happens so you you get this maybe twelve thousand page report and you don't know which of these

are true where do I spend my time and with a couple of apps like people you don't necessarily want to spend years triaging everything but I would like to point out that it's not just you know you're doing static analysis or you're not there's really a range many shades of complexity if you will ranging from very simplistic or grep you're just looking at like reg X's to something in the middle where you have some understanding of code constructs like you can say you know find any assignment in a conditional or something that has some knowledge of the structure and syntax of the language and then at the far end you have commercial sass which has complex control and data flow

so the point here is there are different types of static and dynamic analysis and depending on your situation and how you're using it one can be very useful or less useful depending and we'll talk about this in a little bit okay so let's talk about some things we can do on developers machines so contextually what you need to be thinking about when you're doing this is okay this needs to be fast have load and all those positives or else developers gonna be mad to implement it you can use git hooks such as pre committed or pre push or things like that or as a part of the test suite as we talked about a little bit ago so some

examples from real companies I've spoken with are some companies are say rejecting all database accesses that don't use an approved API maybe you ban any sequel that looks like it's been can concatenate extremes rejecting known dangerous functions things that are shell executor I know one company in particular has as a part of their test suite es limp runs and it looks for any use of eval or new function or things that in a node.js application could lead to RC e and they essentially ban these so that no one can actually even commit these into their codebase or you can reject commits that seem to be adding secrets like AWS Keys API keys credentials etc so one thing that I'd

like to point out here is all of these are essentially greps right they're very simple very fast but in practice they yield very nice security wins ok so let's talk about in code hosting so this is where the developer has pushed code into github get lab Jenkins or anything in the CI CD pipeline you could implement this in on receive hooks or new edge creation things like that there are many different things you can do I'm just gonna talk about for example once ok so similar to what we were just talking about in developer machines you can check for or say banned or dangerous functions you also probably want to check here in addition to on developers

machines because maybe they haven't set up their machine properly and those hooks aren't in place or maybe they got really frustrated and wanted to push it in anyway so it's nice as a fallback defense to do this here as well an additional application of this that I think is really interesting is you can improve code quality over time by blocking new editions of anti-patterns while allowing existing ones so say so this is some colleagues at a company wanted to migrate to a strict CSP but they have an inline JavaScript everywhere already right so they could try to go through all of those and manually try to patch them but again there are a handful of them and hundreds

and hundreds of developers so probably new ones are gonna be added in faster than they can fix them right so instead what they could do with this idea is they can block new inline JavaScript while gradually refactoring what's there already so basically like keep things how they are so they're not getting worse and then gradually we can make things better over time so you don't always necessarily want to block outright perhaps instead you just want to be alerted about things that you probably care about as a security professional so for example you can have some checks that will send an automatic slack message say in the security Channel let's say hey someone is doing something

interesting this person in this code base on this commit or this branch is doing something you might want to take a look at so what's interesting maybe crypto hashing encryption file system operations or any API call or things that are unique to your codebase that you probably want to know about so what's nice about this is it gives you some context about how applications are evolving where are some hotspots that I probably want to spend my time but again you don't have to look through every commit you just have something essentially watching for you and then gives you a ping when something is probably interesting this also gives you the opportunity to correct some common

error errors early such as someone's using a hashing function which has been approved by the security team but maybe the developer actually wanted to increase instead so again we're not necessarily banning outright but we're more trying to understand the lay of the land and what people are doing similarly you can alert on sensitive file changes so in every codebase there's probably some sensitive or rarely modified parts that generally you want to know about if they ever change so anything related for example authorization authentication session management if you've written some wrappers for encryption to make it easier for developers there are some things that probably shouldn't change that often if ever so what you can do

here is maybe hash the files and then alert whenever these change so essentially if anything potentially very critical to your application you just immediately get notified and if you wanted to you could sort of push this forward and say only the security team or specific senior developers that we trust can modify these critical files and just block or alert if anyone else does if you want to take this even further one thing conceptually you could do is you could build models of which developers tend to edit which files are which code bases and say oh you know Joe or Alice tend to always work on these code bases but I see them touch something totally else probably they

don't have as much context perhaps they're more likely to introduce a bug if they're working on code they tend never to work on and alert on abnormalities where people are working on things they don't normally work on the blog post I referenced earlier from square they do leverage code turn and other code quality metrics for where to focus their limited security engineering time so that can be something to take into account as well and this one isn't too surprising it's useful and best practice to keep track of your out-of-date dependencies and ideally you have a policy based on how out-of-date some thing is and how severe the known vulnerabilities are you patch it in a

certain amount of time one company does something pretty neat where whenever a new library's added to one of their code bases they sort of programmatically do a series of checks to determine should I trust this and if so how much so specifically they look at you know how many downloads has this had in the past say month or quarter how many downloads has this had over all time does it use dangerous functions like does it shell exact things and stuff like that and they build sort of a composite risk score of the new library that has been added to see if the security team should manually look into whether this is something we trust or not so there are

many many tools to do this a number of open source ones for specific languages wasps dependency check supports a number of language and there's countless commercial ones as well so we've talked about a couple of different cases in which you might want to say oh hey if someone pushes new code I wish I could run something in response to that to check it so this is a good idea and a number of tools already exist that do this countless that I'm probably not referencing here but some are scum blur from Netflix and Providence by Salesforce that allow you to do this quite easily it has a plug-in architecture and some related tools that are not necessarily checking every new

code commit but are wrapping other tests that you can use as part of your CI CD pipeline or gauntlet and the robot framework so check them out they may be useful to you okay so one thing that people ask me all the time is should I buy a commercial SAS tool and the answer is it depends again so we talked about things running very quickly on every commit these tools generally you can only run daily or weekly because they take hours or perhaps days to run and really this depends on your company so what's your tech stack your development processes your culture and things like that so specifically some things I would consider if you're not

using it already and are thinking about it or do you have large legacy code bases that haven't been thoroughly vetted perhaps there's some nice low-hanging fruit there that you don't have time to manually look through but you can run a tool and sort of clear a bunch of things out is a lot of your code in Java or another statically typed language so tools tend to be better on those types of languages and as an aside one thing I would note is that they don't tend to talk about this but in practice tools tend to be better on specific languages and even on specific code bases so I've heard from a number of companies where they have the same

tool and they run it on one of their code bases and say Java and they think it's great and then they also have a bunch of say PHP or Python and they get very little to no actionable things so just keep that in mind and inherently dynamic languages are harder for static analysis so things like Python Ruby JavaScript those are just hard not necessarily because the tools are bad but fundamentally from like a theoretical computer science point of view it's just hard another thing is how mature is your security program so if you have a lot of developer training and a bunch of tools and processes already in place then perhaps there is less low-hanging fruit for the tools to find

if you use some custom web frameworks or have non-standard control flow through the application know that you're gonna have to teach the tool how your code bases work or also won't be able to find things of interest and also you probably are gonna need to be willing to invest months of security engineer time in the rollout I've talked with engine security engineers at a number of companies and I don't know any case where they've spent less than say weeks two months of security engineer time rolling these out so again I'm not saying this is bad but just keep this in mind so specifically I would encourage you to calculate your ROI right so how much security engineer

time is required to comb through results versus how many bugs of what severity do we find so if it takes say weeks and weeks of going through bugs triaging the tools output to find a single bug while a security engineer just manually looking through your code can find a bug in like two hours probably you should just do that also is this going to be an initial upfront time investment with low recurring time cost or is it just going to take one security engineer full-time always right because it's not just the cost of the tool it's also the cost of the person time and one thing you may find is that you find some initial good

bugs but then as you improve over time you find less and less good bugs and maybe it's less useful after three or five years and I found that a number of companies have started rolling their own tools that is or writing custom rules for these existing ones okay so let's move on I think that I'm not gonna have time to go into a lot of this in detail so I'm going to hurry through it so I have some time for questions so in general commercial dynamic analysis tools have trouble on modern apps they're very say one-page apps are very JavaScript heavy apps and if you want to know some specific challenges that tools have this epic USA talk I think

two-thirds of the way through goes through some specific things but here's a couple of things that can be useful in practice so maybe you're not doing deep testing but you're just testing for high-level things like do you have a TLS with good ciphers are you using strong security headers etc if you find specific bugs from pen tests bug bounty or internal testing write a unit test to say has this ever regressed for whatever reason this is also called out by Zayn lackeys talk at blackhat USA last year which in my opinion is probably one of the best talks or resources in general about these types of ideas I've ever seen period so I would highly encourage

you to check out this talk if this type of thing is interesting to you and here was a case study basically a company abroad NCC group in to integrate fuzzing into their es DLC because they had a bunch of SI services reading things directly from the internet I'm not gonna talk about it in detail but there's some challenges as you would expect ok so everyone wants to know how do we roll our own we want to build some of our own custom tools he makes it out don't worry so one thing that I would encourage you to think about is rather than trying to have one tool that finds everything instead try to have say one tool that

finds one thing but really really well that is when it finds it it's almost always right and basically you can try to wipe out a bug class with few to no false positives and then report directly to developers and then choose another bug class and repeat and I've seen this at a couple of companies that they've done that successfully ok so let's talk about some monitoring and production things again using config management tools is great you have this chain of custody of how things are changing over time there are some can secret services to manage secrets there's some tools that are interesting so get robbed can automatically crawl all of your different repos and recursively all the

developers on your organization's on github to see if they're leaking anything on an engagement I once found a stand up URL in a developer patch Elisa's so essentially I could join their stand up meeting which would have been awesome if it was a red team but it was just sort of a shorting age mints so I couldn't use that to full effect and Netflix has some really cool tools I will point out as well okay so detecting attacks you can have Canaries on servers or laptops that look juicy you can also have Canaries and say login pages or company logos so if someone copies directly the HTML and CSS and JavaScript from your page it

actually silently beacons out back to you to say if it's hosted somewhere that's not you a couple of companies are overriding the runtime environment to detect code injection type attacks so for example if your company never uses the alert JavaScript function anywhere in the front-end you can actually overwrite that so that it sends some telemetry back to your server and then does the alert as normal so one company actually found someone pentesting them that they weren't informed of and they were like oh we have xs/s here and they actually patched it before they got a bug report and conceptually you can also do this in dynamic languages like Ruby or Python maybe a service never should

shell exec or write to the file system you can just override those api's programmatically at runtime such that those can happen again building secure libraries is useful and you can also detect security tools based on their user agent most of them have a obvious user agent unless you're smart enough to change it cool so I'm building a sort of best practices Pro tips playbook containing these and other pro tips from various companies I've talked with so if you want to share how your company does things I'm more than happy to chat a couple of people have reached out already and it's been awesome and also if you want me to email you when I release a blog posts or like a white

paper on this I would be happy to do that as well so if you click on the first link at the bottom its the slides that I'm hosting right now if you click on the second one it's the post on peer list where you can ask questions and different things like that or feel free to email me directly but yeah thank you very much for your time I think I have a minute or two for questions or you can go to lunch but thank you for your time [Applause]