When DevSecOps Fails

our talk today our talk for this session is when devsecops fails with tonya janka before we get started i'd like to thank sponsors especially the inner circle sponsors critical stack and valley mail our stellar sponsors such as amazon the national security agency silence and others a reminder everybody cell phones this is a live stream event so please double check triple check your cell phones feel free to check them right now make sure they're on silent and at the end if there is uh if you have any questions at the end there is a microphone in the center of the room so please use microphone to ask the questions so with that um welcome tanya jenka hi everyone hi

thank you thank you um so i uh i have a weird career and i do a lot of consulting with different really big companies like companies that you know the names of and then i also my company was bought earlier this year by the nice folks at bright security so i work for them but as a result of all my weird consulting i've seen a lot of companies do stuff and not all of them went well and so over the past couple years since i've been doing devops and dev tech up sorts of things there's like a trend of crap that goes wrong and so i have noticed quite a bit of this um so we're like we're gonna do

devsecops and then it feels like this for the dev team um stuff like the security team's like cool we bought a sas and a desk and an sca and we got a secret scanner and then we bought this like weird startup thing we did that and then we put it all in the pipeline and then the pipeline ran 10 hours instead of 10 minutes and no one wants to hang out with us anymore and so i at first was rang this talk and then i i was like when devsecops fails and so i want to talk about how we can avoid those common mistakes and the thing i wanted to tell you at the beginning of the talk to try to help you

pay slightly more attention is that i brought what i like to call the maple secrets from canada so in canada we have these things called maple trees and they leak candy i kid you not their syrup is sweet and delicious and delightful and we make candy out of it and i brought candies so if you ask a question after i'm gonna give you a candy all the way from canada and everyone gets stickers because that's how i am okay so i uh so uh who so what is devsecops so this is what this talk is about so if you're like devops software development that's the worst you might not want to be here so what the heck is this it's like what is

tony going to talk about so we used to make software a zillion years ago doing waterfowl and i mispronounce it because sometimes i say water fail but basically waterfall at first was good because it was way better than no methodology for building software but we realized quickly about 70 of projects failed i don't know if you know that um so i went i went to college in the 90s and like yeah it didn't go that well um so then agile like really smart people came up with this agile saying and it's like instead of waiting two years to ask if people liked what we made what if we asked regularly what if we met every day to like see how

things are going on and so that improved a lot but then devops happened and the idea of devops i'll get into it more in a bit but devops is not only a methodology for building software it's a complete culture change and so if you're like we got a ci cd and we run it sometimes you're not doing all of devops you're doing part of devops and that's still cool and that's way better than doing none but there's so much more you could have and so lots and lots of developers have been very excited about this for a while the moment i heard of it i was like this is the best um and then eventually security folks

are like why are the devops people breaking stuff why don't they stop for my nice gates that i made for them they want to release eight times a day and i only have time to review code once a year and so we had to as security folks kind of like get get the net we had to like kind of catch up and see what they're doing and so then i start getting a lot of consulting calls about like how can we do this better how can we not piss everyone off constantly but still get things to be more secure and so i i just started having clients say just tell us what goes wrong so that we can like work on that stuff

from the beginning like we don't because a lot of teams that they misstep at the beginning and they lose a lot of trust in the process so i'm hoping we could go over these and then after we could have questions and then maybe at 9 00 p.m tonight some of you want to sing karaoke with me but you don't have to you can just watch okay so who am i i am a giant nerd this is the mandatory about me slide um so i work at bright security we make a dynamic scanner and i am in charge of developer relations which means i get to talk at conferences all the time and they pay it's the best

i founded my own non-profit or my own it is for it was for profit just to be clear if you don't make profits you're non-nonprofit i don't know if you know that but anyway i founded this online um community and thanks to bray everything's free now so if you're like oh this was awesome i learned a lot i have like a zillion courses i made and videos and tutorials and all of it's free so go check that out i'm like i have purple hair and i wrote a book and i do stuff a lot but basically i'm a nerd at large on the internet that's like the main focus you should get away from this and when i

brush my hair i look pretty good okay so yay devops so i love devops i'm really excited and so security folks we've been like trying for quite a while to like get people on board the security train especially software developers and so we did we did not have good luck at this at first so like i don't know about you but when i was a software developer when the security team started talking to me i'm like why are they bothering me and getting in the way of my deadlines that was my opinion as a dev um and so devsecops became a cool new look so who's seen this slide this is a very famous slide by a dude named pete

schlezlock who's very smart and he was like this is how some security people see devops uh they're like you know they're magical and amazing and they move fast and they break stuff and us security folks are left cleaning up their as my friend nancy calls it rainbow poo um but yay marketing let's call it devsecops instead and then it'll be cooler and then people will do it and so this is this is the way that i view so they're still breaking things sometimes they're still moving quite fast and sometimes it doesn't work out but we're trying to train them we're trying to give them tools we're trying to help them we're trying to like keep

them on the track so they get there successfully and we've learned that if devs do their works in sprints we should try to do our work in sprints if they're using a ci cd some of my stuff should be in that ci cd i should not just close my eyes and pretend it's not happening because i'm not going to get a good result so marketing basically got involved and we're like yay um but so then vendors got involved and like we're at a community event that we would not be able to afford to be at if it weren't for nice vendors that sponsored it and so i work for a vendor now i'm vendor scum i think i'm making a t-shirt that

said that but anyway i digress um but so devsecops made a lot of promises so like promises that life would be like this we would all feel like this all the time but really it felt like like what i saw with some products because i've played with a lot of products i saw this so this is the system development lifecycle so requirements would happen no security folks complete silence design security folks know where to be found code security people totally ignoring them doing their own thing and then they're like oh it's test let's run five tools on full blast in the pipeline and see what happens um let's have tons of false positives and break the build all the time let's

break every build no uh and then security being like you guys suck uh and then the devs being like i don't like the security team anymore that i've seen this cycle a lot unfortunately and i know it seems pessimistic but i wants to avoid this so vendors promise life would be perfect but they left out a few details and like this kitty's like i thought this was business class um so i want to talk about how we can avoid failure how we can like look like we're really smart and we've done this for a zillion years even if we're new and if you've been doing this for a zillion years maybe you're gonna learn something that you could do even better

so that is the point of this talk and i'm just checking my time because i talk a lot i'm so good okay so the first thing i see a lot is failing builds for the wrong reasons i talked to so many security folks they're like i can't wait to start breaking builds that's a little bit like saying i can't wait to break their dreams like when you're a dev so who here has written software before in their life okay how many of you have pushed code through a ci cd and you're full of optimism and happiness and you're like i'm gonna make it to prod it's gonna be that day yeah finding out the security folks can't

wait to fail your builds feels bad like when you push to the ci cd you're like i'm gonna run a bunch of tests but you're not like this is crappy and i'm gonna send it you've like done a little testing yourself you're like i think this is pretty good let's run some checks okay today like i think i'm gonna get there so if we're planning to fail the builds just from the start like this is not good so i want to go over good reasons to build to fail a build so yes false positives really suck um okay so the first one if you need gates in order to get security done because a lot of people are like i need dev gates

it needs to be part of a formal approval process it's not security professionals who are using breaking builds to get your way like it's not like i'm gonna break their bill to teach them a lesson i've heard that like i've literally heard that from adults i i walked in on these two guys once in the hallway where i had joined a team to do absec and then i quickly left and worked somewhere else and they were making faces at each other like and then they would laugh and the other one make a face and they laugh and i was like what's going on and they're like we're making faces we're practicing our your stupid face and i'm like

what is this for and they're like it's so when devs ask questions in meetings we can make them know how stupid they are and i was like i can't work on this team i do not belong here this is wrong and so we don't want to break fields we do it if we have to okay so the next one is the pipeline should not be the first time we look at security like that shouldn't be our first thing that we ever do like if they're if they're spending time gathering requirements about what they're going to build i want to be a part of those requirements i i want to have a requirement that's like we're going to do this type of scam

ps here's your license if you want to run it first so you always pass like here's how i want you to build it securely etc in the design phase threat modeling like this shouldn't be the first time and we're going to talk a bit more about that in like a few minutes in this talk so the next one is test the accuracy of your tools for a long time before you break builds so i put a tool in the pipeline if i just have it set to give me data and alert i don't have it so it breaks the build immediately because who here has ever gotten a false positive ever on a product okay all of

you are lying or your arms are very heavy because like the first couple of tools i put in i was just like no but it told me there's this wrong and then it's it's not wrong i don't know what to do and so it i think it's important to put it in make sure you've got it going as fast as it can go you're like doing all of the things and it seems to be going well they've fixed the backlog of security problems that were already there then i turn on breaking builds and it's for high things and we see if they can start getting to prod with that in place cool then maybe we move on to mediums but most places i

go to work they have like thousands of critical bugs that no one's fixing and running around breaking builds isn't going to fix that right and so if i can alert them as to the problem and then eventually they start fixing things they start getting better then eventually i can start breaking builds so another one is you can't if you can't get accurate results with your tool that's okay do the testing outside the pipeline so i don't know maybe some of you have bought a sass and not all sas are like this and people that make sass will be upset i'm going to say this but some of them give a ton of false positives like 90

like you said it to be very sensitive and it tells you everything that could possibly be a vulnerability and you're like this does not belong on the pipeline it just doesn't we can't break a build all the time with those positives because we're breaking everyone's trust in us so that doesn't mean you don't use your tool so let's say you bought a tool and it and it's from a super awesome company and you bought it for three years so you have to keep using this tool no one's going to give you another couple hundred k per year to buy another tool so you're like i'm stuck with this tool that's fine but do the test outside the

pipeline or only do a few tests on the pipeline and then run like your big huge long 18 hour saskian outside the pipeline you could still have friends this is important okay so the next one is use out of band tests um so this means like sorry i i like saw the next slide and was like what what am i talking about um so use out of band tests so what you can do is you can run a test overnight take those results um so my friend that works at a big company that makes really awesome athletic shoes so you've heard of them they took all the stuff out of their pipeline and what they did was is they ran it all

outside the pipeline every day and then they send pass or fail based on their policy to an api and so every time the build is run it just checks the security api and it's like pass or fail yo and if you pass you go to prawn if you fail you already received an email telling you how bad you are you already knew um and so that worked for them and they went from an hour and a half long pipeline to a seven minute pipeline and everyone was like yes okay so another thing is consider if you need your security to be perfect or just good most of us just need good a lot of places i consult they have like i said

thousands of criticals thousands of highs an absolutely insane number of mediums and then they'll get a tool and they're like well it's got to be perfect and find every single possible low i'm like really wise you could just ignore them and never you're never going to fix the lows let's like you're you're begging to have one critical fixed a week so it doesn't make sense for us to spend 10 times longer scanning to try to find those low vulnerabilities when instead we could have a scan that runs in like one tenth of the time and yeah it's gonna miss some stuff but it's gonna be good and so a lot of companies i'm like do

you need to be perfect like did you make the covid vaccine are you doing anti-terrorism activities that are top secret are you amazon marketplace well then you probably could just be good most of us just need to be good we don't need to be perfect and so if you can be realistic with that you could choose a completely different tool set and save tons of time oh sorry i'll put that back for you okay um so then the last one i want to say is celebrate success when people get through the pipeline for the first time and they had to fix 400 things to do it tell them how awesome they are this is a thing a security team forgets

we forget to say good job where usually they're like you made a mistake you're terrible let me tell you how terrible what if we were like hey this team made it through the pipeline the first time let's give them a shout out it sounds crazy but i've done it okay so this is the picture slide so if you wanted to take a picture of all the points i just made you would take a picture of this slide with your phone you don't have to but you can if you want and there's going to be a few more of these and i do these because i hate taking notes but i really want to have notes

so it's like it's a heart it's a conflict right okay so i think most of the people's cameras are down i'm just going to give you one more second but you got to be like faster for the next slide just kidding i don't really have anywhere else to be um okay next so then also i have the unicorn i spoke at grim con this year and that's their logo and i just love it okay so slow tools so this is a thing i see a lot of so so i work at a startup and then we made a really fast tool but some of the original tools are like not fast i would call them not fast and

so i want to talk about how we can go faster so the first time i had someone run a static application security testing tool on the app i was leading the team to build it took 18 hours to run and that's not fast at all that actually really sucks um so so i'm just gonna show you fancy cars i don't know if you like cars but they're very fancy um so you can perform as much testing as possible before you go to the ci cd so like as soon as someone checks in code you can run automated tests while the dev is deving i know it's not word um you can actually give them an ide plugin so they can test

things themselves they can push things to their dev server and if you are an open-minded security team you could give them a tool they could be like pew pew pew pew and they could find things wrong we don't have to do stuff only in the ci cd so if you empower your software developers they could do really cool stuff so in the ide rate when you check code in then you can test as like a final last just is everything fine okay and then go to prod and this can save a lot of time another thing is you can perform just a subset of tests in the ci cd so there's usually things where i'm like that will

give me nightmares if we have that in prod and so i will select those tests and then i'll run other tests against the dev server i'll run other tests when they check in their code but then in the ci cd i'm like i'm gonna do these like six tests like if there's injection i really need to know but i don't care if the security headers are messed up and i love security headers but i'm not gonna break a build over it so this is another way we could go faster um okay so use a har file to ensure your das acts like a laser so let me explain this so a har file h-a-r also known as html archive file is a

thing that you can record of you doing stuff in a browser so qa people use these constantly so who here has done qa before in their life okay so doing the same manual test for the 10th time it's not awesome um i do not like it at all and so they're like oh let's just automate it let's record myself doing it and then as the devs continue to push stuff out i'll just play the recording and it's like it passed it failed so you can take a copy of those and shove it into a dynamic scanner and when you do that it only tests that thing so let's say you're working somewhere in your agile

and you're using a ci cd and you're like oh in the sprint we made these two features the qa person's like awesome i'm going to record a thing to to do that and then they flip the har file to me i shove it into the desk instead of the dust having to crawl and look through things and then brute force everything and figure out what's going on it's just laser focused you can make it take 15 to 20 percent as long as it usually takes by doing this it only really works though if someone else is going to record those har files for you because if you are an apsec team of one if you are a uniforce like i have

been and many of you probably are you can't sit there and like record hard files all day you'll die but if you have an awesome qa team that does automation you can work together they can be your best friends and you can like just be really really fast okay so next i really like blue this car is nice um so only test critical or high issues and skip everything else you can do this like you go into like a dynamic assass whatever tool and you're like skip that skip that just do these things i'm scared about these things my team's not ready for that yet and then you can just test the things you want to you can also is this the

next one oh no that's that's different um but basically skip the tests that don't matter to you right so some i keep i still find injection improv all the time despite owasp's best efforts of alerting the entire world how bad it is and so like if i go somewhere and i do some tests and i find it i'm like well that's going to be the one of the things i'm scanning for but if i work at a place where they have really good apsec education and they've been doing this a long time their devs are doing a really good job i'm like i'll need to test for that i know they're not doing that anymore

we've graduated off of that okay so the next one there are newer tools that work faster so specifically with sas i have noticed a huge trend in our industry so there's the original sas of whom i won't name where if i'm a security consultant and i have three weeks to analyze the security of some code i will use their product and i will make it perfect and i will find everything that might be a vulnerability and i will dive down recursion i will do all the stuff and that's awesome but usually that's not what we're doing usually it's like we need to secure these 500 apps and we only have two people to do it and

one of them's busy and so i'm like oh i'm going to buy a newer next generation style sass and just to be clear i like sassed i'm not trying to say anything bad about it but i'll use one that's newer where instead of doing symbolic analysis what it's doing is just matching known anti patterns it's like i know if you have this pattern you suck it's so lightning fast it's super super fast and i won't name some of them while we're filming because i try not to say good or bad things about vendors publicly i accept one but i'm not gonna say that i'm gonna hold i can do this shut up um [Music] but basically

what you can do is like run one of these newer tools it's not perfect and you've accepted you're only going to be good and that's okay for where you are right now and maybe in two years from then you do want to be perfect but for now you're just going to clean up the huge mess that the previous people left and then you're going to do that later and next generation sas tools can do an awesome job of that i'm sure they wouldn't tell you that okay so teach secure coding i realized that you're like tanya's really biased because she sells secure coding however my company got bought a few months ago and all my online training is free now

so i'm now not selling you that but i think it's really really so first of all take the free courses from wehack purple if you have devs that need to learn secure coding it's free why not but if you can avoid making a mistake in the first place that's the absolute best thing in the world if you can make it so that dev finds the mistake before it's checked into the code that's the second best if you can like the earlier you find something that's cheaper and the easier it is to fix it you're going to save money you're going to save time and sometimes when you find a bug way later you have to use like duct tape and

masking tape to try to fix it because like your design has changed so much and it's such a huge change so if you can find things earlier it's way better and if you can avoid them that's the absolute number one best and then um implement manual peer review of code i know that that might sound silly but like if you teach secure coding and then you have people check someone else's pull request before it goes in you can spot so many things like you can avoid things getting checked in so that's like the second best place that you can find stuff okay and then i think this is the last one practice practice practice practice

practice before you start breaking builds so i like to run the pipeline i run it again i run it again i run it again and then finally i'm like i'm ready to break builds because the team is passing and if we find a bug it's probably going to be real it's probably going to be very bad and we really don't want it to get into prod it will be a mistake it somehow got past everything else and we will break it and we will know we should have broken it because it's a true emergency when we break a build it is similar to pulling the and on cord in an assembly line when you stop an assembly line at a

big company it costs thousands of dollars a second every single person in the factory stopped and you stop because someone's arm is in a meat grinder you stop because the one thing's breaking the next thing breaking the next thing or the pieces are coming out wrong for some reason because a piece of equipment is is broken you when you stop it's serious and we need to treat it like that so we can go fast and so this is the picture slide for this the summary of that we still have three more topics to go and as far as i can tell i am doing okay for time yes that's awesome i also have like a slightly silly meme

after this that you probably don't need to take your things out for but i really like to show it when i meet with csos um how are we doing okay i think we're good uh so i finally checked all the boxes that means we're secure finally ah i wish i had his checklist be so much easier um okay so the next thing is an insecure system development life cycle so if we only do checks in the ci cd we don't give them feedback we don't give them training we don't give them guidance like we're setting them up for failure most software developers don't get taught security when they go to school they get taught nothing i wrote alice

and bob learn absec and i approached university in college so many of them so many of them are like cool we'll teach the cyber students and like teach the devs i want to stop the problem from happening in the first place not teach someone else how to mop it up and all of them are like that's stupid why would we teach devs absec and i was like so now i'm writing another book about secure coding in hopes that maybe one day i'll be accepted by academia but i digress so if we have no security activities during our system development life cycle this is bad and if we only have one in the ci cd we're just doing in the ci cd it's late

it's more expensive this way so i want to talk about very briefly because i will go on and on like literally 250 pages according to my author or my publisher i really like secure system development life cycles so here's some things you could do other than just shoving tools into a ci cd you could give project security requirements what if we told them what we wanted we might even get it we could do a whiteboarding session when they're doing their design and just go over and ask a few questions and offer security advice we could do security unit tests so if they're writing lots of unit tests we could turn some of them into security

unit tests we could do threat modeling we could do secure code review or do static analysis of our code we could look at our dependencies and make sure the dependencies are not really scary and known to be vulnerable to use a tool for this it's called software composition analysis we could do secure coding training we could do dynamic analysis so sometimes called das where you know we're like pew pew pew pew and then we try to figure out what's wrong with the running application we could do penetration testing so penetration testing is the number one thing i see and it's at the end of the sdlc and it is the absolute most expensive way to do

appsec actually no it's the second most the first most is when you have a giant data breach and that's your first security activity sorry correction yeah and i was a pen tester so just to be clear i love i think pen testing is important but i think it should be the last final like when you put the cherry on top of the cake of your security so security regression testing so run those unit tests every time they check in the code check your security again guidance for all technologies so let's say they're making a serverless app what if you gave them a list of requirements of what you wanted what if you're like you're building an api

awesome this is the api gateway we use here's how to use our service match we would like to you to do this this and this please most security people are just they don't say anything they don't give any guidance and they're like you're so wrong why didn't you tell me when you knew i was building this why are you telling me i suck after tell me how to logging monitoring and alerting because i say this a lot no one listens but basically every phase of the system development life cycle i want you to know i'm there for you and that i can help you make a more secure app that's what i want as an

appsec person um so you don't have to do all of this but if you just did one more of these than you're currently doing you will see a huge improvement okay did everyone get a picture that wanted a picture like wave around frantically and awkwardly if you still need time you're not waving awkwardly but i'll let you take the picture anyway okay i made this meme i'm so proud i was like this is what they meant this is what actually happened so i want to talk about people that don't fix bugs so unfixed bugs um don't worry it's in the backlog tanya we're totally going to fix it in 2025. um so i see this a lot i see oh like so

most clients that i talk to the biggest thing they focus on is like tanya they won't fix they won't fix our bugs like we send them stuff and they won't fix the bugs and sometimes the aptek people are like we want 100 of bugs fixed in one day that's our service level agreement that's the sla we want i'm like well that that's not happening um and then devs are like we'll get to it later like we'll do it i did the security for one of the elections in canada and i remember like a dev was like we'll do it after the i was like no i do not want bug fixes after election day thank you very much

um yeah so we have to find a balance of never an instantaneous bug fixes so we want when i say sla i mean service level agreement so i say don't carve your slas in stone so be realistic so let's set realistic expectations for both sides so that you actually get what you want or what you need like the rolling stones so decide your organization's risk tolerance so like i said most of us don't need to be perfect i used to do counterterrorism activities for the canadian government which obviously i can't tell you more than that but we had to be perfect oh my god so good to not have to be perfect now it's way less work that

last five percent is really hard so then set our service level agreements based on what we actually want and then make sure they're realistic and fair like i've worked at places where like their security posture just needs to be not pathetic and they'll be fine there are companies like that there's lots of companies where they're they're not target for any reason none of their data is sensitive everyone's pretty chill and it's like you don't you don't need to be you don't even need to be great you could just be like okay and you'll be fine so let's set realistic goals get management support get management to agree with you communicate why you chose those service

level agreements and then start small and go like this so if you don't have management support if so if you talk to the head of devs and they're like i don't got time for you this is not important tanya go away um i have a friend that's dealing with this right now and he's about to quit he's like they pay me 350 000 a year and he's like i'm going to quit because no one wants to do any apsec and every day i'm a failure and it hurts so much so if you get management support life will be a lot better and then being fair and compromising sometimes okay so support teams and reaching their slas so

last year i did a three-month stint at a company they were a long time client they lost their main absence person and i was moving on to doing another thing and i was like let's do just three straight months and swap out your entire tool set and modernize your stuff it was super awesome and we i had two two different teams where it was a tool that had been acquired and one of them was 38 years old so almost as old as me and i was just like we need to re-architect this let me get you more staff and they're like what really like yeah you're not going to be able to like upgrade off those dependencies

you have vlog4j version one um so let's work together and let me get you some money and or persons to help you to like modernize your app it'll be easier to support it'll be more secure life will be good we know we have to do it eventually and so i supported them in reaching their slas and i gave them time to address their technical debt and i gave them rules to triage so if you see this you've got to fix it but if you see these things we can talk about it and then because they're really worried to meet me that i would be like you suck and i was like you're in a bad place

let's put you in a good place so sometimes you have to help them reach their service level agreements oh yes so with triaging i put like a few rules here one more that's it okay so like what do they have to fix how do they validate things if if your tool gives false positives sometimes so i like to train them show them how um but basically if you can make a grid like if tool x finds this this this or this you gotta fix it right away if it finds any of these things could you fix it in the next 90 days everything else fine then they can just look at it and be like i'm cool

it sounds weird and takes time to set up but once you have it you have way less emails in your inbox okay so let's um set realistic expectations so a thing i really like to do is figure out how a bug happened and how we can prevent it in the future so if i see injection okay so let's give a lunch and learn on injection let's show everyone how to find injection let's show them how to review code for it um and then i like to search all the repos and destroy it because if you see one problem that dev probably worked on a lot of things and i kind of like searching code repos and finding

patterns and destroying them so that's my idea of a good time um this is the picture slide because there's a lot of things that i said there there are a lot and then we're gonna get onto the bruce willis themed part of this presentation because that's important [Music] okay so i started a training company not because that's what i necessarily wanted to do but because i asked everyone what i should do and all of them said i wish you would show me how you do the thing you do and person after person was like you should teach us how you do apsec you should i would pay money for that and it turned out people gave me money

and so then i did it more and more and then i had a company but i'm going to talk about training new tech dev security um so when i worked in the government of canada for 13 and a half years performing public service i did security for half of it and they gave me no training our budget was two thousand dollars a year canadian so it's like four american cents and and i really really really really really want to go to sands really really badly and if you're watching sans i would love to go um i would so take any of their courses if i could for free but but a sans course in canadian dollars is around ten

thousand dollars and so that would be five years of my training budget i'm like can i bank my budget they're like no i'm like can i and they're like you cannot take all your teammates budget and so i never got to go so i started speaking at conferences i kid you not for the sole reason i could get him free and i could learn and so this causes problems so there's a lack of training and knowledge so training and knowledge are not the same thing knowledge comes from experience and learning sometimes you go on training and unfortunately don't learn that much so bruce willis what if we train them and they leave i remember my boss saying

this i'm like what if you don't train me and i stay this is a problem especially in public sector where the budgets are very small if you like when i worked at microsoft it was awesome gave me lots of training sent me on all these things paid for stuff i was like i'm a genius now but before that i had to figure out a way and so i would like to just briefly now that i do not sell training um perks of training your team so this is stuff i've seen so employees that receive training are more likely to keep working for you because it is a big bonus for them it's awesome to be invested in

they're usually happier than ones that don't know what the hell they're doing they're more productive they make fewer technical errors and that can be very expensive if you are lucky like really lucky they will share what they have learned with everyone else so they come back with a book and then they loan it to everyone they're like hey let me show you the thing i built um lastly they are potentially a security champion and i really really like that i'm a big fan of security champions i talk about it a lot i wrote a ten point blog article about it because i just i really love it and that is how i have scaled most of my security programs

because i realized there's like hundreds of them and one of me and as much as i i think i'm great i'm not that great i'm not able to like talk to 400 people and get to know all of them and review everyone's code so i need to scale and with training you can identify the really golden ones that might become a champion for you um join opsec they said it would be easy they said oh poor bruce so i want to do a conclusion um and like oh i'm totally awesome for time yes so i'm gonna give you a bunch of resources at the end so we're not quite done devsecops only works if you do the work

you can't just throw a tool in there and have it work perfectly and that's okay you are still awesome we must avoid false positives in the ci cd whenever possible if you break the build and it's a false alarm there's egg all over your face they don't forget tooling must be fast and if it can't be fast or it can't have mostly true positives we do it outside the pipeline we're still awesome we can still automate it it can still be great but it doesn't belong in the pipeline the rest of the sdlc needs to have security steps we can't we can't just can't only look then we're really we're losing an opportunity if that's when we start

um i negotiate with devs to fix bugs i never come in i'm like you're doing this i am not their boss i have no possible way to control them i am open to bribing them with cookies or training or whatever will work but i cannot make them fix bugs so i negotiate work together and be realistic and sometimes that means we don't get the thing we want or we don't get it for a few more weeks and we're stressed but we'll live providing training and share knowledge every chance you can get because it will value you so if you learn a cool new abstech thing maybe you want to show all your security champions or talk about it

at the all staff die hard is a christmas movie and i will die on this hill and and you can do this you really can do this i hope that this made it sound more doable instead of less doable you can do this for real i'm not kidding i want to encourage all of you like even if you just get one tool scanning in a pipeline you're way ahead of where you previously were you really are and so with that i would like to give you a few free resources i like giving resources um so i have a podcast and um so we just we're just recorded season three like the first one of season three and it's

just about appsec now so it used to be about how to get into cyber then it was like mini cyber security bite-sized lessons and now it's just going to be oh apsack all the time so i had my professional mentor sharif kusa on who's the founder of software secured and reshift and he is the person that got me to join oaus but made me a chapter leader and like just encouraged me so much in my journey and he's so smart and i just got to pick his brain for an hour it's awesome um i really like books i really like these books me and my mom both agree that alice and bob learn is the best

book um but any of the top four books in yellow are fantastic devops resources i own multiple copies of each of them like i have the ebook and then i have the physical book and then i have the audio book because i like them so much they're really really really really good okay so the next one is the community so like i said my company got acquired earlier this year and now all of our courses are free all of our events are free the community is free we don't sell anything now we just teach and train and network for free and so all of you are invited um every monday on twitter since 2018 i

have done this thing called cyber mentoring monday so i realized that me providing finding a professional mentor just brought me to the next level like it propelled my career in a way that i never knew was possible before and so i try to help people find professional mentors every single monday even when it's christmas it's pretty slow on christmas in case you're wondering um but basically i use this hashtag to try to help people join together and find each other and it has resulted in lots and lots of jobs tons of friendships people like forming companies together it's been really magical to see so many senior members of the information security community take time for free to help newbies join our

community and excel i've been astounded by the generosity of our industry repeatedly um bright so i work at bright and they have a highly technical blog so if you um are using a tool and you find a vulnerability even if it's not ours um we have how to fix it for like basically everything ever um we also like sometimes they let me post stuff like i said i'm obsessed with log4j and i wrote like a postmortem and they let me publish it there so there's also like not just remediation advice um me so i'm the last resource i'm gonna list me i'm a nerd on the internet and i um tweet and make videos and i have a

newsletter that i occasionally actually release um and i really like making content and i really like hearing from you and so i'm doing this ask me anything right now where people can just send me questions and then i make a video for everyone about it and so i released nine last week and that was fun and so with that i want to thank all of you very much for coming to my presentation because there's a lot of really cool presentations at this time um and so thank you thank you thank you

if people are brave enough to go the microphone you and ask a question you can't just stand there awkwardly i mean you can but you won't get a candy yes sir you told us that we can have a candy for a question we made so here's my question okay can i have a candy for any type of questions i do um i was hoping yeah you win hacker hacker social engineer okay next um i'm in qa do you have a resource video on we hack purple about har files because those sound amazing okay so we actually have a whole course oh my god no really yeah and i'm actually teaching it tomorrow morning here at 8 30. so tomorrow at 8 30 me and

my friend akira brand also colleague um we're going to teach how to make a github actions ci cd and then how to put um like basically color product from it and we briefly cover heart files but in so i actually turned an entire online course out of it and then we show you how to do the heart file thing with it and it's me and the founder and so we're friends and we're ridiculous and there's a lot of cats we just thought i'm gonna cry please join the community we'd so love to have you yes please take your candy and also you can have stickers if you desire awesome hi question for you um i'm a big fan of

security champions i think they're a great way to scale yes uh on your blog or on your your your char for security champions a lot of the stuff was set up one-on-ones and you know building connections that starts falling apart as you get towards larger security champions teams such as a company i work for has 1500 security champions any suggestions for very large orgs like that and how to scale past your do you have 1500 security champions or 1500 devs security champions oh my god so how many teams do you have 1500 teams then i did close yeah okay so how large or very large order how many apps do you how many sorry how many apsec people do you

have uh i'm not sure that number uh that's okay i'm about 20. that's okay but not like we never have as many as we want right and so basically sometimes what i do with my bosses is i will explain to them like i need more staff and here's my return on investment that i can show you from that and here's how i can show you how i can't complete my job unless blah and so it like let's say you have like six other people so that's seven people and you're like we have this many hours and this many days i can meet with like three security champions per day for 30 minutes and then i have to do the rest of my job and

address the things they told me and said and so calculating this i need this many more people and if you're not going to do that then i can't do the job as well as i want to and so sometimes what i'll do is i'll pick so i'll have every security champion attend the trainings but then i'll only meet one-on-one like once a month or less or i'll just pick out the most sensitive high high priority ones and sometimes it's the big famous product sometimes it's the one that's on fire that's trash that's dying and so i usually make a nightmare list and a wonderful awesome list if that makes sense and i like try really

because like what a lot of teams do is they just focus on the cool new apps and they leave all the legacy behind and as a malicious actor or pen tester the first thing i do is i was like last update in 2008 i'm starting here and so um explain to them like just based on the number of hours i work how this is impossible and if they say well you can meet with 16 people per day because you have eight hours explain to them how that's full because some of them will say stupid things does that help yeah you deserve two candies um next please so you mentioned that your podcast and obviously there's other resources for

like can you speak more into the microphone so you mentioned your podcast and obviously there's other resources that how you can get um like training for but like what would you say is a good resource to see for people who are looking f maybe if they want to get into like what's a good resource to see like okay this is what your typical day-to-day cause you know obviously movies are bad books are bad like you know they don't really show you what it's like to do that job okay for those people who are like trying to figure out do they want to change careers etc what would you recommend okay so i have three resources for you

so one is a blog post that i wrote called jobs in infosec and if you google jobs in infosec and then like she hacks purple you'll find it immediately it gets thousands of reads per month and i wrote it like two or three years ago and i describe every single job then there's this lady named alyssa miller and she's speaking here right now at the exact same time as me which really sucks because she i wanted to see that um anyway she just published a book about all the different types of jobs with interviews of people who did those jobs and then the third one is season one of we hack purple podcast where i

interviewed about 50 different infosec professionals and asked them all about their job and what every day was like and how do you get this job and like what personality traits make you better and stuff at the job and then they give stuff on how to go get into that job and so between those three things you should be good and if you don't remember what i said tweet at me and i'll tweet the answer back at you in writing so that's helpful awesome thank you you deserve candy everyone who wants to come up and get stickers now because it turns out i'm getting booted in like four minutes yes so so this is a real case that i had

um and i'm an app developer and the problem was so fundamental to the code i'll tell you the end the problem is so fundamental to the code that they end up throwing away the entire chunk of code and replacing with the purchase project okay so that was that and that took years and years to figure out but fundamentally the prop the code was so broken that it had built into it an account takeover bug oh so how do you manage an abstract situation like that is so fundamental to a fundamental architectural failure of the code how do you manage that sort of conversation um do you have they said like no we're not fixing it well no i was the app

person who found it and i tried to fix it and other people it was basically years of we give up we're going to throw away the entire code base oh wow so i wasn't in security i just found it right so i was wondering how would you manage that as an abstract person because this is not making up case real so sometimes a rewrite is necessary like sometimes that makes it so much better especially like because of the performance and other things of the app and how it's easier to maintain yeah but in this case it was i looked at it was it was like once it was just like how do you manage it where you can't fix

it fix is not is off the table and everybody says we'd love to but we can't okay so then i usually put a band-aid on it also known as a waff or a rasp and so oaf is a web app firewall and a rasp is a runtime application security protection tool a rasp is more modern it's going to cost you more money but it's going to do a better job all the waft people just be quiet um basically but it's a binary and you have to stick it inside your application it takes a while to set up but then it will block malicious input a waff is more like a bunch of regular expression that lives on your web server

and it's like that looks bad no that looks bad no but advanced attackers can often get past that well a rasp is uh has better results and tends to have less latencies but will cost more money and take significantly more time to set up but less time to maintain okay does that help i don't know but i want to let someone else go okay i'm being kicked out now i have to go but if you two want to go outside the door i would love to take questions from you and you still get a candy because you're still awesome