How to Build an Application Security Program

thank you guys for coming out we're gonna quickly go through a talk on how to build an application security program thanks to will for inviting me out to give this talk just a little bit about me my name is Jerry gamblin I do something at Kenna security I'm not exactly sure what when I started ed Billis told me I get to pick what my title was I'm really thinking I'm gonna go with the shrug emoji that's probably where I'll end up just going to do a little bit of time on my journey to application security sitting around and thinking about it I've spent 20 years in security now nearly right out of high school right into security my very very

first job was with the government I spent 10 years there it was a very very interesting 10 years I learned so much I had all the time in the world I could do any amount of research I'm on it I could learn to code I could hack on things I had a budget of zero dollars I remember when I got an SS license going through procurement on that and getting that was one of the hardest things I've ever done that was $2,000 but I've learned so much I would never trade those 10 years it gave me a really solid base and what the technology was and what I was doing from there I took the lead security role at

CARFAX this was almost 180 degree flip from where I was at with the government for those who don't know Carfax is part of the world's largest data company we have about 1,500 individual data points on every car in the United States from you know when the last time your rule was changed if you got into a fender bender so we were doing this at a massive scale I think when we left we had just about 20,000 servers so I'd gone from a place where I had no money to spend to I had basically all the money I could spend and it really opened my eyes because I quickly learned that people matter way more than tools you

could give me a hundred thousand dollars today I go out and buy the latest sass store - tool and it would make no difference right you have to have somebody to sit in front of that tool no matter how awesome it is and to run it and I learned automation is the way to the future but it's not there yet when you start thinking about how to build your application security program you're gonna have people come in and give you these sales pitches about oh we could automate 95% of your you know your DevOps workflow ask them to show you a company that's been there right because the best companies I've seen are about 65 to 70 percent automation and their

depth DevOps workflow I then ended up on the security research team at Kenneth's and this has been a really really interesting journey for me this is my first startup we were mid stage startups so we're pretty pretty solid but it's also really different I have a good budget management is great but I'm not working with people who wrote this code from the ground up so they have really really invested ownership that I haven't seen before right so when you start talking about code you're not talking about oh some function that somebody else wrote and then transferred three departments over you're normally talking to the person who wrote the code two years ago so it's really really opened my eyes to what

personal equity and personal pride and in a in an environment licks like I'll start with you to drink the water real quick so I'm just gonna jump right into some lessons learned we only have 30 minutes today and I want to make sure that I get through these there is no perfect tool for app sec my boss at the government said that if application security was easy the security team would just do it right they would just buy whatever the other necess is for app sec throw it in and run it application security is a totally different beast here's an eye chart that we use when we talk about application security tool you know there are five different

variants in application security here's all the steps here are the 40 tools that you can use if you need application security you know from your static analysis to your IDE all the way up to your bug bounty and penetration test and for everybody who is taking pictures I will just post these on online right after I get off stage so you don't have to do that what I would put them on internal dot dev so this is a Johnson cran isn't in the building today yet but this is a slide that he makes and talks to it talks about the overlay between your average static application security tool and your dynamic application security tool ten percent is about what you get

so if you're starting an application security program you have to go out and buy the best static analysis tool that you can find for your language if it's Ruby you know you might pick a brakeman if it's you know if it's go you might start a company and write one because there isn't one and then you have to find what you're going to use for your dynamic application security testing there are a ton of good good offers out there that you can find a ton of great tools automation is hard period we nice talk to people about how to think about automation I try to the first step I do is I try to sit down and

have them draw out what their workflow looks like from if you start today and you need to rebuild your stack from the server to code deployed what does that look like and 95% of the time I cannot find one person who can in a company who can sit down and start from oh here's how we provision a server here's how we set the DNS records here's how we get the TSL cert here's how we install everything here's how we push the code here's how we make it live in our production cluster that is where you have to start with automation it's from the basics getting in a room with the white board and figuring out

what you your flow actually looks like and then you can automate it these are my top five goals for automation I always want to be able to move application and security closer to the developer those decision point they quote-unquote own the code they need to see what I see as fast as I see it and then of course the all configuration should be static and you know we should just be able to run out and everything should be the same even the best organizations fail on this I can't tell you how many organizations I've talked to you said oh yeah we just build everything out of puppet and then they get an external penetration test

done and they have 20 or 25 percent drift it's like you said that you build all your servers with the same script how do you have any drift in your environment it's just natural and once you realize that even with the best automation program you can have drift you can really then work on shortening that up and then testing at every stage that's harder than you think it's it's easy to test at your static stage when you put code in and at your dynamic stage but it's those middle middle tests right like on your IDE are you guys all using the same linter right I cannot tell you how many fights I've had with my developers about winter rules I don't

write the winter rules we picked a linter and we're gonna use it and if it says you have to have four spaces here you have to have four spaces here I like those are the types of tools that you have to come to an agreement with now kind of into the soft part of this talk let's talk about management management so I have two takes here the first takes is talking to your management is like having a cat right they don't want to see you as a security team unless you have something for them or they need something from you that might be harsh but just showing up and saying hey I have this problem can I just tell you my

problem and not give you a solution is gonna end badly for you I was lucky enough to hear : I'll give a leadership talk one time and he says the best way to be a leader is to always followed these three communication points when you go to talk it's tell them what you know tell them what you don't know and then tell them what you think based on what you know and don't know in that order how many times I've messed that up in my life i-it's zillion times because I always start with what I think is gonna happen right like oh yeah I think that we're gonna gonna lose all our data here without talking about I know that we're

running an old version of sequel I don't know if there's a live vulnerability if there is the worst thing that could happen is we could have a data breach what's talking about product management no better friend no worse enemy all right how many of you guys work directly with your product managers in your company they are responsible for everything I tried to figure out like what my product manager does she's actually sitting back there Emily she's in charge of a road map she's in charge of her day to day work and then she's in charge of me showing up and saying hey we've had this bounty submission can we get this prioritized and she's always

like we'll try but is it more important than these 40 other things that were working on today product management is easily the hardest job in any technical organization I do not know how they do it developer relations so we have any full-time developers here good um mark berry gave this quote at a I think it was Ruby comp two years ago and it basically says that developers don't want to do anything but write new code and I had a really snarky slide after this but I took it out because it's true right developers get burnt out when they don't get to do what they want but my job is to normally bring them stuff they don't

want to do as anybody here found a developer who just loves to update frameworks who's like because if you do we're gonna hire them away from you I don't care how much it costs I because it's no fun to update frameworks it's no fun to do input validation every developer wants to go in and build the next widget are the next connector or whatever your thing needs so dealing with developers is super hard here is some stuff that I've done over the last two or three years that's really helped I posted some dev security training I have a company that if you want to talk to me offline I'll give you the day above they do a really great job they

come in and they have live code and they teach them what burp looks like and and gives the devs hands-on experience on the tools that I use every day that they might not have seen so that they kind of understand what these vulnerabilities look like because the sooner they see the vulnerabilities the more likely they are to really dig in to fix them we go through the CTF for our dev team we a host one a Los Chicago hosted a CTF I took three or four my devs two that they loved it it was the first time that many of them had seen a real-life vulnerability and understand what's like it was like oh this is what it looks

like what a sequel injection looks like here's what somebody would sit down and hack this bank and pull this data down it was one of the things just sitting there and watching these guys for the first time just seeing that light bulb go up oh this is what would happen if I could get access to this database you know if I could read this table that I'm not supposed to and then the hardest one that I'm still working on is to teach them basic threat modeling threat modeling is hard for me it is super hard to teach a development team or a management team right you have to say okay you built this amazing product and

now I need you to sit here and just flip the script and tell me how you would break it can you do that how would a bad guy use this right like right an abuse story that is super hard that's something that I'm still dealing with today I don't have a great answer for it but I know that at least trying to get those con substan is where I'm going time management is so hard I like to hack stuff I have no time to hack stuff right I have probably I looked at my calendar over the last month I have roughly fifty to sixty percent of my time our priest standing meetings right that's just

where I need to be in a meeting not doing my job and then you've got just the written communications you needed here right like I really have to get in the slack Channel and talk about what hamburgers the best in Chicago or my opinion won't count right and then you got to work on your tickets and you gotta you know make sure Emily knows what I need I'm dealing with external vendors by the end of the day I have very very little time to hack anything right like in this role running an application security program most of my hacking happens at home when I'm not supposed to be working we'll get to this point in a minute but if you don't if

you're not at a meeting your opinion doesn't carry the same way right I can't slack it hey I need this cross-site vulnerability bug fixed and that carry the same way it is if I go to a meeting for an hour wait for my five minutes to talk about why I need a bug fix so some of its just people management like Rachel was talking about right if you show up people are gonna take you more seriously budget management you have to have a standalone security budget you cannot run a security program on handouts just out of general like you can find a good SAS a static tool for about five grand dynamic tools are super

expensive they cost anywhere from $10,000 to Texas dollars right looking spend any amount of money you want on a good static tools Consulting is twenty thousand dollars a week like that that's the baseline for application security consulting that's why I want to go into application security consulting if anybody's no okay and then growth management this has been where I if I fall down right it's like oh we're gonna spin up a new team right and then I'm not smart enough like everybody else in the company to ask for more money when it's the new team hey we're building a new product and it's gonna be 20% more so everybody else is in line to ask for 20% more I'm

over looking at the slack thread on hamburgers because I wasn't thinking right now let's go through everybody's favorite part we're gonna go through my failures Thank You Winston Churchill he needed that after World War 1 not learning to program I know that this is a really really hot button issue in security now people are like oh you don't have to know how to program to be in security yet you don't I mean you don't know have to know how to like fly a jet to be an air traffic controller but if you want to be able to talk to these developers you have to add a basic level know what they're doing and being able to talk shop with them it's just

what they want it's what they need I have a goal to be able to be able to drop in and do an entry-level job on every team in my technical organization and I think that that's when you know that you're a solid security professional when hey if the Sissons guy is out for the day you could go over there and be the junior sis inch guy today and build servers you might not be fast but you could do it hey is your ruby developer out sick for today can I go and sit at the Ruby devs desk and be the junior person on that team and not need to be handheld all day that's kind

of where I'm leaning not understanding the business I know so many security people who who basically end up like this this is a quote that I love from a comedian who was in San Francisco - lives in New York she says having a kid is like having a tiny drunk friend who thinks you're incredibly rich and sometimes I think security security teams operate under that standard I fall under this all the time I remember one time asking for a $20,000 review on a product that was scheduled to make $40,000 over the course of a year I don't know if anybody's done any Business School in here but spending twenty thousand dollars to have something reviewed that's gonna make

forty thousand dollars will get you laughed out of a room pretty pretty quick I I see people who are willfully ignorant of what their company does and how their company makes money and the security industry to a point where where it's embarrassing you're like oh what is your company do is I always sell something but I'm just on the security teams like no you're on the I sell something team and security is your role there but then this also flips back to that if you don't have a budget there is you don't have a program right I will have open source software I contribute to open source software you cannot run an application security program on open

source software you can supplement your application security program with open source software but it's just not there and the best tools end up getting bought by big companies who have for you know for-profit products I love break man we had brakeman CRO which was a paid product but it was it was super cheap they've been bought by a bigger company right we use the Ruby SEC those guys have been acquired by github so even your open source program those guys are sitting there and it's not hard to see them getting swallowed up into a bigger organization so to think that you can run an application security program with open source software is a fallacy titles matter I can't see that on that

screen I love the office and there's a huge difference between being the assistant to the regional manager and being the assistant regional manager I don't know how many times I've been in a meeting and it's been oh yeah this is a manager's meeting and we're gonna invite the security guy who's not a manager you just don't carry the same weight if you're supposed to be on the pier level with a group to get a decision to made for your application security program it's only fair that you carry that titles right that's why there are CISOs now if we sit at the board level we just need to take that same thinking and roll down if you're the application security

professional and you need to go and talk to development managers about getting stuff done you need to be the application security manager right that's the only way it works I know we want to live in a world where titles don't matter and and you know nobody cares but that's not reality communication I suck at communication I suck so bad at communication that I actually have a communication coach I would suggest that you guys get one it's it's really helpful luckily I live in Columbia Missouri at the University of Missouri we have a student group called the antlers they are basically professional hecklers and we have a very good basketball coach called called cuando Martin he's really good the problem is I

want to thank that I'm the coach most of the time when reality is people see me as the heckler they see me on the outside not knowing what they're able to do yelling at them to hey try harder run faster and they're like oh you didn't think we thought about maybe running faster here you know it's really hard to not be a heckler and application security or security in general and move to being a coach and that's what I like to be a take away from this is to think about when you're communicating is this being seen it's like I'm just heckling these people am i bringing anything that they really see or do I have the

authority do I know what they're going through if I learned to program enough that when I sit down and talk to them about fixing something they see me as a coach so just a quick review you have to communicate period work on your communications I should just have this on your 40,000 times there is no perfect tools but you have to have a dynamic tool and you have to have a static tool if you not have either one of those in your environment today you have to go out and buy one product management is your friend you have to understand what they're going through you have to understand where the business pressure is coming from I had I saw a diagram

they sit in the middle of all those decisions so if you have any security application stuff that you need done they have to be on your team Automation is really hard to do you have to do it anyway nobody is growing their security teams faster than they're growing the rest of their program right so they just expect because they go to blackhat and they go to RSA and the hero easy automation is they're gonna come back and say hey we're just going to automate all this stuff and we're gonna grow all their development teams 20% the security team is staying flat this year so if you know that that's the way it's working you have to spend time to work on your

automation understand your business understand how you guys turn a profit understand what's profitable for your company there is no better way to get a bug fix than to walk in and know how much this bug could cost your company if you know that this bug is in your purchase path and you know your purchase path makes $10,000 an hour that's a number that the business loves to hear right like hey if this is exploited and we go down we're gonna lose $10,000 an hour is a lot better than hey there's this bug on our credit card page we should really fix it it's important titles matter but budgets matter more security tools are expensive and they

keep getting more expensive and you have to communicate but at the end of the day there's no right way in 20 years and I've stepped through and seen three different versions and I don't know I cannot parachute into your organization and help you build a successful application security program I can tell you what's worked for me I can tell you the pitfalls I've had I can tell you where to look but every organization is different every culture is different and you have to figure out what your culture is and what you can do to help steer the ship towards security thank you guys very much I'll throw these slides on a link on internal dev today at some point

I think that we have questions but they're supposed to be on some app so there is one question which is we have a security focus team with buy-in from the DES to perform regular framework updates but updates are a time-consuming task but these features and there are no new features that get built during this time so any tips on moving this process along no no if they could give me some tips nobody likes to update frameworks right you go and you say hey we need you to update this jQuery framework and they're like that takes a week of dev time and provide zero new features I really want to build a new feature you just have to be you

have to go back and say I understand that I would like to be building a new feature too but this jQuery is old here are the vulnerabilities we just have to do it it's it's part of hygiene I have a nine year old and I can't tell them why brushing his teeth every day is important versus that if you don't it's gonna be bad in six months right you have to you have to just take care of it every day to make sure that it that it's good on a whole okay thank you thank you guys [Applause]