BSidesSF 2020 - Lessons from DevSecOps Trenches (Clint G • Zane L • Astha S • Justine O • Doug D)

hey everyone thank you so much for coming today I'm excited to be joined by several esteemed friends and colleagues and we're going to be talking about various things that have been successful at various types of companies in terms of scaling their security efforts so we're going to talk about some things that we're great some things that were less so and we're going to spend a lot of time in the middle of perhaps 20 to 30 minutes or so answering questions because we want to make sure that we're speaking about the topics most immediately pressing to you so if you go to SLI do you should be able to pose some questions which we'll be able to

see and vote on together okay so before we get into it let's just introduce ourselves a little bit Zain do you wanna go first sure hey I'm Sam Lackey I'm a co-founder and C so at signal Sciences and previously - thank you and part of that as the CEO at Etsy built and ran the security program they are kind of at the start of the DevOps shift hi everybody I'm Justine Osborne I am a manager of an offensive security team at Apple I've been a breaker of things for most of my career a security consultant for most of my career at NCC and AIESEC partners and yeah now doing Apple I'm Doug Perry I'm currently the

director of defense at data dawg it's relatively recent change prior to that I was director of product security also my backgrounds mostly in offensive security at Yahoo as well as NCC and I second hi everyone I'm aster you probably saw me running around all throughout the conference and if you're a speaker you probably got lots of spam from me so thank everyone for coming my day job is I lead application security at Netflix that includes sort of engineering facing security partnerships as well as there like engineering related things that we do to scale ourselves as a function and outside of that I helped organize this amazing conference and thank you all for coming

right and my name is Clint Gibbler I'm a technical director and research director at NCC group so we do security consulting for all the things mostly penetration testing but we're also doing more as strategic advising on security automation and DevOps and stuff like that another thing I wanted to mention before we get into it is all of us are good friends but I we're gonna purposely try to find some things to disagree about because we find panels where everyone's a green very boring so and also when people disagree I think that shows you the nice like edges around where like a specific saying company culture or background may make something a great idea one place and not as much

in another so you know we disagree strongly we're not going to fight afterwards at least not about that so okay so one thing that I think is interesting is many companies are giving talks about hey here's this cool thing we did and we hear lots of success stories but I think the failure stories is also very interesting you know because mistakes are made things don't go as planned so I'm curious if you could share share some things that you put a non-trivial amount of time into that didn't go very well or at least as expected yeah and this is open for anyone well I can start with most here this was actually not at Apple but at a

different company I worked at square and I attempted as my first project to create a dynamic analyzer and that I put a lot of time and effort into and ran it first time and managed to crash pager duty well toss them essentially and so yeah it was a it was a it was a huge learning experience about how your script that you run on your laptop is it's gonna behave quite differently when you're running it in a real production environment and you really have to think about a lot of things and a lot of consequences to the things that you're building and I think also as a security team you don't really want that to be

the first introduction to a tool that you wrote it doesn't give people a lot of confidence in using it in the future so a real lesson learned there to really think about the actual operation of the tools that you're building and delivering to the organization yeah back in NC days we spent my apps like team and I spent close to a year trying to really bring in both SAST and DAST static analysis and dynamic scanning tool him that was a year of my life I will never get back to put it mildly but I'm sure whisky companies were very happily they're very happy about that it was a near complete disaster on both fronts of really trying to have trying

to adapt sass join in - coin that was written for very much the waterfall era of applications not being that feature-rich being very server-side heavy so not really any client-side functionality or anything like that we really said like look you know these have been common tools of any app sect program is always you know talk about app sec you talk about sass and DAST something we had to learn very painfully and now I've seen a merge across virtually all of the forward-leaning organizations is that those using those sort of controls has either changed radically or gone almost entirely out the window as a result as you've really gone into a environment where your development teams are moving faster and

faster and faster and faster so we can deep dive into what that means later on but I'll do that as a short high-level spectacular year long train wreck yeah plus one on that investment in static analysis where we did spend a bunch of time on so we're a polyglot environment so we did spend a bunch of time getting the tooling built up and getting some of these tools running and then the ROI was just not there for us I think we hit a new record for speed of which we started to trash static analysis yeah so one thing I think this was maybe two wraps at Kelly's ago Scott Scott Behrens on the Netflix team one of

the things they mentioned that I thought was very interesting is they spent a lot of time thinking about or building this thing out without clear sort of success criteria and metrics and one of the lessons learned he said was that after say five or six months of working on something they then said like oh like are we done we're not getting the results we want but do we keep pushing forward or do we stop and so now he said because of that lesson they now spend more time thinking upfront like how do we know when this is a success and when we should pull the plug and I thought that the comment was very insightful

definitely allows us to fail faster yeah so another topic that I think is interesting I hear various people talk about and blog posts and talks is asset inventory sort of this idea of understanding your environment whether it's what's in code how code goes from to production what your cloud environment looks like your employee devices things like that it sort of means different things to different people but I think overall asset inventory means sort of like what do you have and how are they talking or how are they connected so one thing that I'm curious about is I guess if you were to start embarking on the path relatively from scratch trying to get an idea of

your environment in a very you know constantly changing environment in many cases like what would you do and in your work doing this has there been anything surprising or interesting any gotchas maybe to look out for yeah I think one of the things that we realized that as we started doing sort of solving this problem of acid inventory was you don't need just security engineers to build you an acid even if they are sort of software focus security engineers you actually need data engineering expertise so you are building those systems to properly scale if you do have a pretty big environment where you're gonna end up dating that so for us the first version of our asset

inventory which was built by the apps act team basically got to a point where we weren't able to scale it anymore and the work were now doing with our data engineering partners is what is kind of setting us up for not just solving abstract problems with the asset inventory but other use cases like Incident Response and you know even outside of security related to running migration campaigns and things like that for our productivity engineering partners yeah let's say one other thing we learned on the asset inventory side so not on the technical side but on the organizational side how you can get I can start to get some asset inventory set up is sending security whatever you

want to call it business partners or champions or anything like that to meetings with the product teams and the meetings with the kind of build infrastructure teams and just don't have them object to anything they're not there for a security review they're not there for any of that their job is just to sit in the back and listen and find what are new products that are coming out that you didn't what just weren't on your radar at all because you you win two for one with that you find out first of all what's a new you know feature or product or anything like that that's coming out that now you can actually put through a security review but then also

you can find out where did something break down in your pipeline in essence of not finding out about this and adjust where you want to plug in whether that's on the technical sense or organizational sense to find out about hey there's a whole new initiative going on in a second cloud provider that we knew nothing about okay what could we have done to discover this earlier rather than at the product meeting that's discussing the press release going live next week like that which i think is happened to all of us problem yeah so something someone submitted which was actually already on my list of things I wanted to ask is how do you think about the build versus buy

decision sort of what are your criteria and when when would you do one or the other I'll kick this one off there's a lot of variables there including I mean like cost and budget of course you know that's kind of the obvious one but it's also you know the amount of resources you have to devote to that sort of thing you know how long it might take them to build that you know whether you're working with you know full-time developers or security folks who are good at writing Python or something along those lines right I think a big one for me too is that at least in my experience the security teams that I've been on and very much the work has been

very much interrupt-driven and so even though you have a you know it looks like you've got a full quarter to play with it actually ends up with being like two weeks that you can devote to spending time to write this thing and depending on what that thing is that's not nearly enough time and then that's gonna be like this project that you just like dragged along quarter after quarter and you never really complete to your satisfaction and so you have to be really I think aware of what your inbound interrupts are going to be and obviously like by you know by definition you don't always know that but I would just say to be as realistic as possible

with that one thing that we try to leverage whenever we're making a bit versus by decision is something called decision logs that we do internally that helps with that where you're actually kind of being very explicit about all the different great here you consider the different options and why you decided to go in a particular direction so that in the future if you ever have to revisit that decision you know you can you you can look at like what has potentially changed and the other thing is also sometimes based on the Wender you're working with sometimes it's also possible for you to say okay if you can only do 50% of what I need is your

product going in the same direction where I want to go so getting that 50% from you today is going to be better than me riding from zero so I think keeping those things in account and making that a part of the decision process is very important yeah on that I give up complete 180 that I've done in my career on this one of starting out coming from Penta side to building a security team scaling that we're I think when you come from the Penta side it's very easy to think about how oh every product you've ever you've ever looked at you've always found some edge case that you know you've found a vulnerability in it or

something like that nothing's perfect why would I deploy any of that and then you have this magical career experience which is actually going to defense where you learn that hey for most of these areas we have 0% coverage on that and if I can get a tool that I can drop in that gives me 50% coverage or 80% coverage or something like that and then my team is gonna have to build on top to get that remaining 20% I will do that every time now right by my goal in scaling a security organization is getting to that 80% coverage across as many areas as fast as possible so that when you do want to build or if you even have the resources

to build which many teams do not it's for that incremental bit rather than for the foundational piece of all these but had to learn that one the hard way yeah so just to make that a little more concrete can you just list maybe two or three things that you and not like vendor but just like category like thing you bought and thing you decided to build just to make it more concrete because I think the decision criteria is useful too but just like having some anecdotal examples try do that without mentioning my own company now because that's what we did a few areas in the endpoint side of things there's a good example like building what then became a

West query but building on that side on the the endpoint side of things where honestly we probably should have bought is a little bit tougher because this was 10 years ago and no one had go to OS 10 support for anything but that I think that was a good example of I see a lot of security teams today kind of trying to invent their own sim or invent their own end point things or anything like that and thinking hey how can we build on top of some technology that's there to customize for what we want on the app SEC side of things there is a lot that that we really did they're building on both static analysis side because that

wasn't working and then really the laughs and protection side which then became signal sciences but yeah there's certainly no like two-factor off ones or anything like that which I've certainly seen security teams try to build their own sort of authentication services or anything there let's let's not try to maintain that ourselves and build that ourselves I was gonna say account takeover detection was the thing that we were looking at and and we looked at a few different vendors and a little we're having this conversation we're trying to figure out you know it's just some JavaScript and the backend thing like probably do that ourselves and then we went to a local meetup and I think it

was someone from Etsy I forget I forget their name but talked about how they like built this thing over the course of a year and a half I mean this ISA look to each other they were like well I guess we made the right decision to go to buy that product right like it's you know in and there was still like not as many features as we would have liked you know but it was great work don't get me wrong but we're just not gonna invest that much time in that one we could just buy it yeah so in general if it's like not core business function or not very unique to your company would you say

that you would consider buying if the price was reasonable yeah I mean I would say if it if it fit like nothing's gonna be one size fits all but for the particular constraints that you're operating under at your company at the current time if this thing really fits view then buy it and then something that we've done like almost pretty consistently over the although the vendor products that we purchased is we then write custom scripts plug-ins use it in our own unique way and so we end up with kind of like a hybrid thing where we buy and then we build on top of it we keep in touch with a company and see if what they you know what we want

them to do is on their roadmap or not it's always in the next sprint it's never the next sprint I just wanna make that clear but you know like so we kind of end up with this you know but a Frankenstein thing that works well for us one thing I do want to call out there is like keeping an openness to say that if a year now a vendor product will do the thing that I needed to do being appointed deprecating the thing you wrote yourself internally if that's going to reduce the pain of operations and maintenance and things like that on your side I think the maintenance of these things is the thing that no one thinks about right

yeah oh yeah we can do this thing and then but are you gonna maintain it until you leave and then read all the documentation so that someone else who's on call can fix it if it breaks and yeah yeah my answer was gonna be at dynamic analysis I learned the hard way that if you're not prepared to operate that at any sort of scale it's gonna be a massive failure and so sometimes it's better to just to just buy in those situations yeah so I guess along those lines there's been a couple of questions from the audience because there was some earlier comments about challenges with static analysis and dynamic analysis tools so there's a sort of a series of

related questions you know like for example like what sort of scanning do you put into apps like pipelines you know if not sassed then would or just related things like that and I have some thoughts on this but I figured I the offer it up first yeah so I guess okay sure I'll start so an action I'm going to be talking about this a little bit in like not the next slot but the slot after that in this room so I I will talk about this in more detail but I think one thing that I found very interesting from a number of different companies is there's a couple ways to think about it right so you can think like are we

trying to find all the bugs or are we trying to find are these security controls and secure by default frameworks we've built being used like have you not disabled the protections that we built for you so in terms of the big like systemic scalable systemic wins I've seen I'm like learning from Austin and then a lot of this I think comes from Zane's blackhat talk a couple of years ago I'll just shamelessly borrow from it is that basically if you can build this these nice guardrails this knife nice paved road where developers can be better faster and more secure and everything's handled for them they don't have to think of all these risks and you

just then grep for like disable security or dangerously set in HTML and react for example so basically like you you've handled all of these cases so that rather than trying to find very complex like interprocedural data flow analysis type bugs you're instead saying oh we built all these nice security controls for over you you can handle secrets you can do mutual TLS between micro-services like all these nice things we'd built for you and then basically your checks are now for not complicated vulnerabilities but just are you not using the nice things we built for you so that's one trend I've seen across a number of companies that I think is basically the other side of the

coin but I think is very powerful because ultimately you you need to spend time like triaging tool results and I think there are cases where you do want a deep like enter procedural data flow if you have like legacy C C++ code bases and things like that but that's one interesting shift I've noticed one of the things we realized doing some internal analysis about a year and a half ago was for a lot of the critical vulnerabilities that we had found in the previous year we realized that oh if some of our paved road controls were adopted in those systems we would be able to reduce the severity of greater than 30% of those vulnerabilities so at

the end of the day that's why we're here is to reduce a risk to the company right so if we can reduce that risk by not altogether preventing the vulnerability but reducing the impact of it that's definitely a worthwhile trade-off to make I want to disagree with you a little bit thank you weave in agree no max no um so like and I've I'm gonna disagree and also agree so like I I agree that we that that security should provide like tools and like abstractions to make security easier I mean I've said that for a while I completely agree with that approach but like the other thing that we that security people have said for a while is also like you

know educate empower you know like work with your develop make sure they understand security make sure they understand the impact and I feel like we abstract too much it could be bad in the long run right because there may not always be a tool or a framework or a thing that they can use and then so you should completely forget about security the whole time you know like it's just something I kind of like I've struggled with personally yeah that's very you you've sort of helped and coddled people for so long that they don't realize it's like a dangerous world out there so that they maybe don't realize when they're not using sort of a secure by default

framework that there's like 50 things they need to care about that they've never had to care about before right yeah and it's like it's I mean obviously I think the answer there is its balance right but you know make it as easy it may be if you make the secure thing to do the easy thing to do you'll get a lot more like uptake within your within your organization don't you know don't just drop the workflows don't slow things down all that sort of thing but you know I think it's just fine line so yeah that's definitely agree with you that that can be an issue and that is something that we do an offensive

security is try to bring some of that back and make sure that when we talk about the engagements that we do that we share that with as many people as possible in the organization and that gives them some perspective as to okay that's that sort of minor vulnerability that came up in my static analysis that I didn't think was important now I see how it's used in a chain of vulnerabilities to reach something really important and it can really articulate it better for folks yeah sort of along those lines Justine I think sort of read teams as well as like dedsec op teams they're sort of the most hyped new teams recently so I guess for

you in your experience what is if a company is starting or building a red team what are some like key things to make sure to do right or how do you make sure that a red team isn't just finding bugs that are you know maybe or maybe not getting fixed but how do you make sure the red team is like measurably improving a company security posture and like reducing risk just sort of finding one off bugs yeah I mean I think it's a hard problem that I'll probably be working on for the rest of my career but you know just measuring security risk in general and and where you're making an impact but I do think

that offensive security has a real benefit there I think that in general one of my colleagues always says security testing is boils down to sixth grade science it's you have a hypothesis you do an experiment and then you analyze the results and that's really what I think most of security testing should be and I think where we sometimes get lost is what's our hypothesis and I think our hypothesis is often the part where we're not quite sure what we're testing or what we're trying to get at and I think being successful as a red team is really looking at okay what are our hypotheses and do these actually make sense for organization do we really

think that this type of threat is going to affect us do we really think that this you know we're worried about this thing but it's very hard to exploit do we really think that we should focus on that and so we have a hypothesis that we think these are the things than a bad guys good or bad actor is going to do and and so if then we run our experiment and the cool thing about red teams is that they're run on the real running live systems in production and that is something that I think is really fits in well with our new world of DevOps and agile and all of this because it's impossible to do the old method of let's

test everything before it goes out and so red teams give you a chance to kind of see the bigger picture and so you're doing all these little things throughout your development lifecycle to make sure that you're doing you know secure coding and you've got kind of the frameworks in use that you need and everything but there's things that we there's always loopholes there's always things that you didn't think about and I think that red team's also really bring in the operational side of security so we often if we're focused mostly on apps X sometimes it's easy to forget about the things that are corporate infrastructure that's actually running the things that we that we're developing is

sometimes where it all falls down so um not sure if I fully answered the question but yeah cool yeah thank you another thing I heard along this space this was from someone on Google's red team where she said that not just the impact of a red team is not just the number of bugs found but it's also like what additional sort of attacker tactics and indicators of compromise did you test that the blue team was not effectively finding and sort of like the the additional protection mechanisms put in place as a result of the testing not just the bugs is sort of like an important value add which I thought was cool yeah definitely I also would say

that coming back to asset inventory something that the red team can often help with is identifying the things that you have out there we're really good at finding those little weird things that nobody knows about so I've got a spectacular failure case of that actually which would be of the how those two things actually came together which was we ran a kind of no scope attack simulation where our attackers could do we handed them a laptop and said it's as if you fished an employee go just do whatever you want and one of the one of the controls that we expected to detect them earlier early on was a network enumeration right if they they popped

the laptop they're gonna kind of enumerate the environment and kind of figure out what's there before they move on to specific targets and it got two days into the assessment and like are these consultants on another project well they look like they haven't they haven't tripped any of these alarms that we've done in here and this is ridiculous you're not doing anything and later that day they called us up and they're like here's the production database oh okay yeah you have been busy and when we went through the debrief we're like we were really curious why none of the controls we've done around network enumeration and asset enumeration none of those tripped and like oh yeah yeah yeah that's really

cool that you've cleverly instrumented all these parts of your boats corporate network and production network yeah you geniuses didn't put any authentication on your internal wiki so you just have a network diagram right there's we just knew exactly what hosts to go to we'd have to enumerate anything and I think was the perfect example of asset enumeration via your red team as a way to feedback into your blue team controls yeah yeah I would also say that in general back to the question of you know how to how to make your red team successful is the other side to that experiment is getting the results of your experiment out to all the right people and being that feedback loop into

all the other parts of the organization and making sure that you stick around and actually follow through on the things that you've found and don't call it done until those things are no longer possible I think one thing I do want to call out related to just red team or offensive security assessments in general is yes it can be a great tool to increase like organizational awareness around security and things like that but you also want to balance that with making sure that you're not damaging trust with your customers if it can feel like you're trying to trick them or you're trying to make them look stupid or make them look like they don't know

what they're doing with the systems they're responsible for so it's really really important to be aware of that trade-off we'd kind of the organizational awareness yeah yeah that's a good point so another question from the audience is what are your thoughts on threat modeling and security reviews how important is this work to you and how do you scale this effectively as your organization grows so one of the things that we have tried to do in this space is so we handled sort of threat modeling and design reviews through our application security teams on call and what we have tried to do is we have tried to find cases where we are track modeling similar things or doing design

reviews for similar scenarios and then figure out where how can we how can we make most of these variables go away meaning using the pave route to say oh there is a solution for you know the authentication proxy you can use that has a bunch of these features and they're already and then how can we standardize guidance for things that we solve for all the time so that your security guidance does not depend on how cynical your security person is feeling that day and you're getting consistent guidance and it actually becomes easy to do something like that through on-call and that doesn't become sort of the majority of your time you're spending on just doing those individual threat

models and design reviews yeah I think I'm pretty good likings example of this as well is so recently you have made some changes in the security organization and I inherited a team that was doing attack detection and an alerting type of work and they've done a lot of good work over you know a few months before you know I inherited the team but they but but there was like there wasn't any real framework to the detection strategies that they were creating and so I wanted to find a way to put like to focus that right and so we talked about threat modeling just in general and then the more we talk about what are we gonna threat model AWS

threat model every service individually and then GCP as well and then Lenny and so you know we kind of threw out that idea but what we ended up doing was looking at you know existing frameworks and so we settled on miters attack framework and they had someone recently I think come out with cloud-based one and so we essentially did like a gap analysis right with like what our current detection strategies and where where do they line up on this on this board and then you know where are our gaps you know if we have like two or three detection detections for this particular strategy or this particular attack type well then like yeah we want

to fill in all of those blocks and fill in that entire matrix but like we've already got 2 out of 5 that's pretty good over here we've got five columns that have zero colored boxes in them and so maybe we should kind of focus on that first as we like you know kind of just go along and try to essentially fill up this this graff and the other reason I like that like strategy is is that it didn't require us to really dig in and do a whole bunch of extra work right someone already did that work for us we took that framework so if that matrix has started filling it in with what we've

got and it gives us a really good visual representation of how we are progressing over each particular quarter each particular month whatever has the graph slowly like as those boxes slowly get colored in we can make an animated gif and put it somewhere and show our boss and then it but like that gives you like some concrete metrics to say like now look matter tech firm may not be perfect but it's something and you know if we can fill in all those boxes we can feel really good about the amount of attacks and the different strategies that we can detect yeah I think threat modeling is definitely something I've heard again and again from different companies in

terms of like how do you scale it and so like development is happening very quickly there's a handful of apps tech engineers and maybe thousands of developers you can throw out model every story so I've seen sort of three primary approaches so the first is having developers fill out a sort of self service security questionnaire where it's okay you're building a new feature or like a new sort of epic or story level we're gonna give you this questionnaire that says okay what technologies are you using what sensitive data are you touching are you going to be like parsing XML or URLs or what language are you using basically what are things that a security person

would probably ask you and then at the output of that it says oh here's some framework specific guidance here's some things you should make sure not to do and overall just computes sort of a risk score for how risky is this new thing you're building and then the security team rather than having the threat model everything they can then meet with the teams who are building things that are most likely to be the riskiest right because you have to prioritize another thing is basically integrating threat modeling into the development lifecycle itself like hey we're gonna talk about what it needs to do what's the acceptance criteria and then also like hey how could this feature be

abused and then what can we do to prevent that so you just add on like one or two threat model II type questions I think Adams Shostak has like a card game as well that can be played during discussions and then lastly there's this concept of like threat model is code where you're basically making integration tests but they're sort of security tests where you're assuming certain abuse cases and then actually codifying that in code so a security questionnaire is integrating threat modeling into the SDLC and then throughout modulus code is three approaches I've seen some companies do okay cool so there's a number of questions about sort of abstract pipeline like continuous security scanning and I guess what would be some

useful metrics of success or what sort of scanning continuous scanning in terms of static or whatever you have the highest ROI for you I think really targeted scanning similar to sort of what you were talking about before instead of trying to do everything well do pick one thing and do that really well and find kill that class of bugs everywhere rather than trying to sort of focus on killing everything everywhere yeah yeah I mean we basically to be successful in kind of continuous canny plugging in to see ICD SAS - whatever you want to call it now you kind of invert the model that we've always done here exactly as you saying right if SAST

and DAST was always a very heavyweight top-down control where we said we're gonna turn this on across our codebase spend 18 months tuning the thing and then figure it out and it'll eventually work we promise and really to be effective in this world that either your organization is already in or rapidly going into the only way you keep up with the scale and the pace is to start bottoms up and say we're gonna pick this bone class then this vaughn class then this bone class on up and just start bottoms up with those let's pick a few things in each one work our way up from there and what I mean specifically if work your way up is getting to the point

where it can be self served by the development teams or DevOps teams themselves all right it's something that spits out a huge report that the security team has to parse and then send on to another team it doesn't scale you drowning in that in no time you have to be able to have something that can like I mean as simple and as dumb as great here's the four things we grep for for this vulnerability class or the four things we grep for that say you've disabled key paved road security checks and it auto Flags a ticket in Giro or your bug tracking ticket directly to the development teams get that for as many things as you can before you start to go

up in complex and that's in this kind of area of continuous scan II and CI CD that's the only way you scale grep works pretty good yeah actually one of my favorite quotes about that this was like two apps at Callie's ago we had a similar panel so DIF Dada a very senior security person at Dropbox PhD in computer science did a number of papers on like symbolic execution like crazy crazy smart and he was like grep is awesome man like I mean I said it to be wiseass but but seriously like if you can find if there's a number I mean it's not a silver bullet right like it's it's not a it's not a solved problem period even if

you had all the time in the world and if you're again you're trying to keep up with the pace of development at your in your company you're never gonna have enough time it's like stack analysis I think is one of those things that if you get to like 50 percent effectiveness like you want to be honest like it's not a solved problem but like grep and just like looking at dangerous patterns that are specific to the frameworks the apps the the code that you use in your in your company will get you a large portion of the way there right like take old bugs and you know kind of like search for those patterns and you can do

that pretty easily in in you know in your CI and I think like having a very tight feedback loop there and and and telling the developer or the engineer like what what happened and why and how they can fix it and how they can reach out to if they need help like as soon as or as close to as close to it as possible I'm never gonna get that sentence up but like as soon as they've written that bug is is the bet the better you can do I'm great to like I dated all like every developer use their own I like there's like five or six different kind of like we can't can't maintain

plugins for all that stuff but next best step for us is github right and so that's where we've done a bunch of MRSA secondarily like get lab and like the the deployment pipelines right gonna run a bunch of tests there other devs and engineers are running a whole bunch of other linters and all these other sorts of things like we can throw a security test in there too it's not that that's like a little bit farther away than I would prefer but in some cases that's the if that's the best we can do for a particular application or service then I'll take that any day of the week rather than waiting for Justine and her

crew to come and tell me that it's a problem and then I've to file the bug and we have to follow up and then mole management takes over and you know six months later maybe that medium gets fixed one of the things related to this that has worked well for us from an ROI standpoint has been some of the work we've done with finding vulnerable dependencies in our ecosystem in very close partnership with our build and change teams so they have all of this tooling related to wanting to basically manage change cycles for software and turns out that we are trying to do the same thing for security reasons so we have worked really closely with those

teams to figure out okay how do we integrate vulnerable dependency information well with some of their tooling to flag the highest criticality things that we care about in our ecosystem by language and then the way we've approached it is okay this language is most used in our ecosystem let's go after that first let's draw a line related to what criticality will look at what are internal things we know that impact criticality so meaning things like is this internet-facing is this a central tool that is used by a bunch of people things like that so taking that criteria into our triaging process as the apps act team and then really working closely with the central engineering teams to solve that class of

problem for a certain criticality level for a certain language before we move on to the next and that has worked fairly well for us nice thank you I like this one a lot what's the most effective security engagement and aware initiative you've done with your engineering teams and I know Zayn in particular I think has a story he might like to share I I'll start with that one one of the things that led to some of the best security relationships and bug Disko just important security discoveries all euphemistically say would be just finding ways to have the team have engineering teams engage with security and it's really dumb stuff that actually makes a successful it is

literally looking at a lot of times floor plans and having the security team sit in central locations so that other teams walk past your security team because we lost track of the number of times that a major security review started because somebody walked past and like made eye contact and kind of waved low and then did a double-take and was like oh actually you know since I'm here I've got a question for you about something we're thinking about and it was this oh you're designing remote code execution as a service okay got it like those sort of things happen off of that off of the thing Doug was laughing about which is how we leveled up that

technique of putting security centrally in different engineering offices we literally put out bowls of candy on the thing and the amount of folks that flocked to the security area suddenly went through the roof the other security budget that we used was literally some folks from our team say hey who wants to wants to go out to at happy hour tonight great this engineering team is going to a happy hour go to that put the last two rounds of drinks on the security team just buy a round of drinks I guarantee you someone is going to ask you about some product that they're working on or some feature that they're working on and it was the

single best use of security budget between bowls of candy and buying a few drinks that we got that I wish I could go into the details of the things that that uncovered right and it's it's things like that that actually become really effective in driving traffic into the security team because if you start to get the closing thought on that I'd say is for me is you have to think about your security team as a business inside the business and this is your marketing budget right be thinking about how are you driving folks to come engage with your product which is security expertise one of the things that we've done recently is actually have the engineering team that

we're targeting on a red team work with us on exploiting something so we're like okay we found something we're gonna exploit it will you know a lot of times red team's traditionally are very secret no one knows about it but instead we kind of do the opposite and say hey we're gonna exploit this thing real in production join us and see it in action and learn about how your thing works in a different way than you realized and that's been really interesting and fun and engaging and really helpful for the red team to to understand more about how products work and things we hadn't thought about before so we don't do a ton in the awareness engagement space

but I think one thing that consistently works well is when somebody comes to talk to you create a really good experience for them so if they came to you for a security review and then you end and ended up kind of like pushing their timelines by a bunch and not really being helpful and just throwing requirements over the wall to them they're not gonna come back so create a good experience when folks do come and engage with you and they will definitely return yeah I mean 100% I think that's really really important is to just yeah be as realistic as possible right and don't even even if you're your first you know the first thing you think of is to

say no no way or dear god no you know remote execution as a service seen a few of those as well right like that's it's terrifying but again like they're human right and the other thing I would add on to that is that we try very hard to be as responsive as possible even if we don't have the answer right they're like if you leave a message in the in the security slack channel it's not gonna hang out there for two days right like it's I mean listen we're not hard percent on that but you know just try to be as responsive as possible like your time is valuable my time is valuable let's all we're all in this together you

know that's the message I do okay cool so uh we're gonna start wrapping it up so just final thing if you were to join a new organization or just in general looks like the biggest highest impact thing you would do like maybe the thing you would do first or if a company isn't doing this you would make sure that that happened I would say visibility into yeah I mean the core foundations I think of all of this is knowing what you have and knowing who owns it knowing what it is and being able to have visibility of what's happening on your systems I got nothing now actually one thing I was thinking of was um you know if I was

in charge and had the resources I think I would try to protect uh either a very small team or a few individuals I try to protect them from the interrupt-driven work so they could kind of like take the longer view on certain types of problems I again it's just a big thing with us like prioritizing and dealing with interrupts is really painful to get actually like work done again we're there to handle the interrupts we're there to support the rest of the business I totally get that and I totally accept it but something you know how do you level up if you don't have time how do you where do you find the time to create these frameworks and the

paved road for folks if you can't you know get your head straight so I think that was one that is one thing I would do given you know the amount of resources other than that visibility I would say make sure that you have an understanding of what is the risk you're trying to reduce so just an understanding of the enterprises risk what is most important and you know like what's going to be the existential worst-case scenario for the company so you can start prioritizing starting from that because you're not gonna solve all their problems and day one I'd say when you're when you're going into a new security leadership role in a new organization or anything like that

bringing it up one level here I would say the number one thing to focus on is getting that buy-in from the other teams and go to them and stating explicitly look if you're going to any Enterprise tech company it really doesn't matter the size of the company someone there that you're going to need their buy-in has had a negative experience with security at some point in their career I can guarantee right and so if you want any of your projects to ultimately be successful the route that you actually have to start at on there is going to those teams and saying hey security is here to help enable you do this you to do the stuff that you want to do we're

not always gonna get it right but I want your feedback from when we're not getting it right so that we can adapt because that's the only way that you're not gonna have teams trying to route around you and not tell the security team what's going on or anything like that you've got to get that trust and you got to start it right in the beginning well and something I would say is vulnerability management it doesn't necessarily sound very sexy but you need to know where you are and when you're doing various initiatives you want to be able to say oh this actually worked not just it was a good idea right okay cool well thank you so much for your time I'm

going to be talking about related stuff like this in about like an hour or an hour and a half yeah come up we would love to say hi and I have some stickers if you want some of those thanks for your time [Applause]