We Are The (Security) Champions

our next speaker is steve myrick he's a raleigh area native who does offensive security for avalara he's also one of the people behind the eversex ctf upstairs so if you've played that or played it at a previous competition give him a hand he's going to talk about security champions thank you so much uh hopefully that sounds okay great yeah so today we're going to be talking about security champions it is maybe not the first time you've heard of the idea it is it's not something that i made up and it's definitely uh something that's relatively standard but we're going to talk about a little bit of some of my learned experience in in growing a

security champions uh program over the past year or so avalara so uh with that we'll go ahead and get started um again my name is steve meyer that's my um my pre-pandemic hair i am an adversarial engineer over at avalera we are in durham so i've i've been working there for a little over two years now i've been on team eversec since about 2016 we uh really enjoyed getting to put on the ctfs for uh for for these conferences and others so uh if you haven't check it if you haven't got a chance to check it out definitely do jump upstairs to the second uh second floor or if you are online uh we are hosting the um ctf for

the next uh bit of time uh and then i am one of the few people that i meet around here that are actually from uh raleigh as i'm actually i was born in night vale so we are a rare breed nowadays but uh let's go ahead and jump in with a kind of just a definition of what is a security champion you know we got to start at square one here so um here's a a definition is a member of a team that takes responsibility of acting as the primary advocate for security within the team and acting as the first line of defense for security issues within the team now this doesn't mean that they are a member

of security if you have a security organization this doesn't necessarily mean they're that they could be a developer they could be a qa person they could be a project manager but this person is the person that is responsible for taking the ideas and the objectives that security is trying to promote within your company your organization and really uh making that their mission as well and helping promote that so a little bit of a forced multiplication from the efforts that security has to the kind of front front lines on each individual team so you maybe you understand this you get the idea but why why would we have security champions this is why can't we just build out

security a little bit more we spend more money we hire more pen testers we hire more instant response people um why is that not necessarily going to be the most effective way of us kind of promoting this program or or running security i guess in your organization so uh the first point uh apsec professionals will always be outnumbered you've probably experienced this a security is typically one of the later additions to a company once they you know if you have a product based company and they're making something security isn't always added until you know they have gone to market they have proven themselves that the product is worth buying and they have the budget for it honestly uh so i've heard even

that it's on average about a hundred to one ratio in appstack professionals to developers security doesn't always have the tribal knowledge that the development it teams have you've probably experienced this where the people who have been working on products the longest the people who have uh first-hand knowledge will always inherently know more about the thing that they are building than you will as a security person coming in trying to i don't know either interface with them or read documentation get an understanding of the product right uh they will block security issues before they happen so the more that we kind of take this idea uh that security is everybody's problem or everybody's uh responsibility that

they will start to shift left i mean you've heard the term a thousand times but shift the the point the uh resolution of security issues uh before they make it into production hopefully it's always going to be cheaper and more effect more effective for you to stop the security and stop a security vulnerability than having to put it in production identify it with a pen test then go back and uh remediate it uh ownership drives action uh you you know this if something is not necessarily your responsibility if something is uh not necessarily a formal thing that you are responsible for it sometimes doesn't happen you know uh it people are focused on things that they

are measured by the things that they're responsible for so if they if security is everybody's responsibility then their ownership of security and secure software will drive action in making sure that they do it the right way uh is generally low cost high return on investment you can run these programs for free if you'd like it takes obviously political capital but there is a high return on investment here you it not only saves head count you are reducing the amount of people that security requires in terms of the yearly budget but also there is a there is a cost to vulnerabilities if it if they they're exploited it's a lot higher but there is a cost for

the remediation effort the testing effort all of this uh in the creation of a meta security team so maybe you've heard the idea before but uh what is the meta security team a meta secure team is a team that is not rolling up through some sort of security organization officially but they are responsible for security so these are your these are your partners these are partners in crime i have a few quotes here that i like from a few different organizations that i relevant to security champions that i think would be kind of interesting to talk about so security champions don't need to be security pros they just need to act as the security conscious conscience of the

team this is this is important we don't necessarily need to take uh you know take security members inject them into dev teams no it the important thing is that they just need a passion for it they just need to feel like it is their responsibility it needs to they need to be promoting security within their own small group active members of a team that may help make decisions about when to engage the security team so if you've had problems like this before where you have you know maybe you'll find out much later on after a remediation effort happens or maybe an incident happens um they would say oh you know yeah we we resolve that it's

fine no big deal um well if they don't know exactly who to contact they don't know the procedures they don't have standards for how they engage security then you're going to be in a spot where maybe you'll miss things maybe maybe they will they will follow a incorrect procedure and could land us in relatively large trouble here's another one it's someone who serves as both a mentor and cheerleader of sorts engaging with and encouraging all employees to learn adopt and remain committed to the security protocols i love this quote because it talks about two important things one is learning and two is adopting we are adapting we always need to be focused on improving ourselves as

uh security as security professionals and the and developers always need to be focused on improving themselves as developers which includes inherently security security is an important part of software always has been but definitely is much more now uh in the current age so really what is this what are some benefits of the of this program we talked about why it's important what are some of the benefits what will you receive uh in in creating this type of program one is kind of bridging the gap between dev and security for more cohesive vision probably her cohesive visions buzzwords right but the the important part here is that everybody is on the same page we have we

have no longer do we have this you know silo of security and the silo of development where you know security has their own goals development has their own goals we now have a singular cohesive vision is everybody's everybody's on the same page uh promotes a sense of ownership within the dev teams they are now uh not only do they feel responsible for making sure that they deliver secure software um but they also feel proud of the work that they've done that they've done it transitions security from their responsibility to our responsibility i'm sure you may have heard in an organization mel talked a lot about these relationships between developers and security and how much conflict comes up from these

separations these silos between organizations it really kind of promotes unity here it allows you to dissipate information more easily as much as much as you as much as everybody likes to assume that newsletters big emails can you know be the perfect way to disseminate information a single group of people who have an area to talk about their mission will always provide a better solution in dissipating information you'll have one team member who is focused on a certain problem with terraform and how and how security integrates with that and their conversation their questions maybe will help somebody else in their in their mission you'll always know who to contact if you are if you are dealing like if you are

in an incident response type of role uh this is incredibly important because now you don't have to say oh is it you know who owns this product who has this you know should i talk to the manager is there like a tech lead that i can talk to no like you know that that you have an established roster these are the people who are responsible for security incidents in your uh in your products you can improve knowledge sharing by providing a dedicated platform uh we talked a brief a bit about that briefly but a single dedicated platform will become the spot for security discussion within your organization and it's a little bit more distilled because this

is not going to be every single person in the company uh talking in this platform but they will but they will have an area to discuss things in a safe manner and they will always know exactly if i have a security question i will go here and this is and the security team will answer it for me so you maybe you've gone through a maturity model or an audit where you had where um they're measuring the effectiveness of your security organization many of them are not maybe they're not requiring security champions but maybe they are bsim says create or grow a satellite that ties directly to security champions there are other maturity models that i'm sure you're

probably familiar with you've probably seen a reference to them in the past but um if if they do not require it directly the benefit of having these things will directly improve your scores on these maturity models these audits and then also from another perspective if a team is doing if a dev team is doing a audit as well having these security champions these dedicated people who are familiar with the security policies who are available in understanding of how security works the organization they will be bet they will be your partners and be better off answering questions on behalf of the entire company at those audits so there a couple years ago i believe it was 2017 there was a obasa security

champions playbook published at one of the oau's conferences so uh i'm not going to say i'm going to reinvent the wheel here but there are a few things that this person presented that were um just kind of general steps of how to run a security or the steps to run a security champions playbook one is identify teams two is define the role three is nominate the champions four is set up the comms channels five is built on knowledge base and six is maintain interest we've talked about a few of these and we will continue to talk about them but uh i just wanted to have that available just saying that there are open source standards of how

um security champions programs are built this is again this is not a new idea i'm not presenting any groundbreaking information but hopefully this lived experience will will provide a couple of um things to learn from so a couple of things for pre-work things that you can get started say okay i'm on board i think security champions would be a good idea let's go ahead and get this started what do i do so some some pre-work here first is to find success i think for any large scale project defining success is important but especially for this one what is this oh at the end of the day what do you want your security champions program to

look like what benefits does it have directly for your company um two is get leadership buy-in if you get to a situation where you say hey i want to start security champions program i work in security i think it's going to be important if you do not have this buy-in if you do not have this agreement from engineering leadership you're going to have a problem because they're not going to want to you know spend dev cycles of taking valuable time out of feature development to work on security issues uh this needs to be a collaborative effort between the two departments if there are if they are separate departments uh three designated organizer uh i've been kind of in this role for the past

year where i just kind of am the person who generally just has somebody to go to in terms of uh managing the security champions organization there will be times where there'll be a little bit of you know saying say grunt work for just uh you know making sure that the roster is updated running uh communications uh keeping things together at some point uh four is take inventory this can be very difficult or very easy depending on your organization i i don't know how well you maintain and know exactly what every single one of your systems and applications are probably not that great but if you understand what you have then you can better understand what what kind of

representation you need from a security champion's perspective uh five is defined requirements again you need to figure out exactly what you're going to ask from the security champions is this going to be a super formal program is this going to take up five hours a week what is what exactly are you asking because you want to be clear about it and six is established budget you this is again not necessary you can run the program for free but you will always be a little bit it will always be a little easier to run a program when you do have abilities to reward people with monetary prices who can be a security champion uh easy answer is anyone anybody can be a

security champion the point here is not that we have to have the most security minded person in the organization security passion above all is important this is we want somebody who has a mind uh for has a mind for learning has a passion for uh for security and as long as that is available then that is the key type of person we want we want volunteers or nominations before forced appointment uh the last thing you want is somebody to be jammed into a role where they don't want to do it to begin with and will only cause it will only cause them dissatisfaction volunteers are the best because because they've stepped up nomination is a little better because

you've taken a you know maybe a manager has decided hey this person would be very good for it maybe they didn't want to step in uh to begin with but before forced appointment try to avoid it if possible uh representation uh we do want representation on the different teams uh a good example would be one per business line that's okay that's fine you have a couple security champions um two better would be one per product or one for application okay great now we have the ability to say uh for any application we have here at our company we have a security champion who's responsible for it uh but in my opinion best is one per agile team

one per say you have a large scale application and you have uh 10 teams developed to the various features within the application if you have one per agile team then you have a heart of security at every table in your in your organization and you want to decide is it going to be a one-to-one one-to-many or many-to-many relationship uh you can kind of go either way but is it gonna be one champion to one application is it gonna be one champion to many applications or many champions for many applications i personally take the approach that many to many is a good solution because it will allow anybody who wants to be a security champion to be that security champion as

opposed to saying hey you know we already have somebody on your team or you know we just don't need another right now no we should be able to you know provide the benefits of of that kind of growth so uh get your get your security champions community involved this isn't going to be important for the longevity of the of the program uh if if you just start a program and let it die it's that's what it's gonna do it's gonna die it's not going to to go anywhere but we need to keep up the cadence of you know what do we need to do here uh so some some events would be great i've uh started

one called vulnerability the month where the idea is we find somebody who recently has had a vulnerability they've remediated it that's the important part that they've remediated it and that we can talk about it and we really kind of want to break down the stigma that security vulnerabilities are always bad it's always a something you should look down on if somebody has you know had a vulnerability in their application uh no the the we want to celebrate the kind of the the remediation work that has gone gone on within a team they've done a good job capture the flag shout out everest ctf upstairs uh but capture the flag is a great way to get

people involved gamification is going to drive education people will enjoy doing something especially if they're if their prize money behind it but you also can do things like round tables or tabletop exercises roundtables in my important in my opinion are very important it allows security champions to feel like they are a part of making the decisions within our organization and not just consuming them so say hey we're going to make a transport uh encryption standard we're going to this is going to be something that we're going to pass down for the entire company if we get security champions involved with those conversations early then they feel like they're more likely to comply with it because their voice was heard

right we just didn't uh take some random thing off off the nist shelf and just shove it down engineering's throat right we don't we don't want that tabletop exercises are great because it is a way of testing testing controls ahead of time so people when there is an incident they know exactly what they do they've done the practice before these these security champions are somebody who uh has sat through a tabletop exercise and when uh incidents happen it's just another day training is incredibly important now you want to be careful and figure out if it needs to be mandatory or optional i would say that the training itself should probably be mandatory because they need to have some

sort of baseline we need to understand that everybody's on the same page but maybe the attendance live is optional uh we we try to take the approach that anything that we have for security champions needs to be available later on or doesn't take up uh key parts of their time uh it needs to be both relevant and important it needs to be something that we it doesn't need to be waste of time they don't need to feel like they need to learn about some sort of i don't know obscure sql uh injection vulnerability if they are not dealing with databases at all right it needs to be they need to have a general understanding of security

os top 10 is like a great you know baseline for it um but the stuff that they need to be learning it needs to be important for the daily jobs and input so like i said having their input will help security make informed decisions you're going to save time you're going you're not going to start up a uh you know you're not going to start a standard and then pull it back three months later because nobody can comply with it and it helps not feel like they're ignored right they have us they have a spot at the table their their opinion uh matters just as much as any security professional does so for communications again you've

probably heard it a thousand times but slack teams irc internal social platforms those are all great uh obviously we want to have one centralized location where everybody can contribute to a conversation consider bots consider automation if you can uh i i like to have like slack workflows where if somebody needs to know where to get the um if they need to know where to get the standards for something they can click a button and get standards or click button get training things like that mailing this mailing lists and distribution groups are also important also probably should be automated if possible taking down the amount of management work on your part will be very much appreciated down the road

meetings this is important uh avoid becoming a burden do not have a thousand meetings for your security champions make them come to every single one of them this is going to build resentment because developers already don't have enough time they don't have enough time to do every single thing that they want to do their backlog is probably infinite so we need to avoid becoming a burden make everything available afterwards record it especially when you are working in different time zones if you have international teams you don't want to say oh i have a mandatory um i have a mandatory meeting at 9 00 a.m eastern and then all your west coast people have to get up at 6 a.m

that's terrible one-on-one sinks on request maybe not every security champion wants to but maybe there are some that want to have um this closer relationship with security and you should make that available to them right this is something that will drive them to be better security champions in the future so rewards we talked about uh budget being important right uh if if you can get it changed there are some tangible rewards swag is great if you can come up with some sort of brand for your security champions program and have swag available that's awesome people love mugs and t-shirts uh training if maybe if you want to provide some sort of actual training when it comes to like a

certification attempt or something where you say hey you know we will pay security will pay for your oscp if you're interested in getting it conference attendance like this if we had to pay thank you besides for being free but if you had to pay to go see a um a conference then that would be another great thing to provide as a benefit uh prizes including ctf wins lotteries engagement recognition prizes are great for recognizing the people who really engage with the program but just as important as as tangible awards are intangible awards these can be mentions in newsletter communications kudos to leadership or from leadership opportunities to work with security and develop marketable skills or mentorship

mentorship to or from they can become mentors for other developers they can get mentorship from security whatever that whatever that looks like these things are incredibly important and will cost you nothing it will make uh the entire program more well received now we want to talk very briefly about a controversial topic which is as a potential hiring pipeline the idea is security champions will make good future hires for transfers into security you be careful here there are some positives um you know it provides people with a structured promotion path it gives them things to you know things to work for or work towards it injects organizational dev knowledge into security that it would be very difficult

to receive otherwise you will hire from a pool of known trusted talent to build your security team but there are a lot of negatives to this as well so be careful before you make the decision some challenges people may join solely as a reason to ditch the dev teams and they want to stay within the company they just want to move into security they may do this specifically for that um politics can get messy if it becomes an internal coaching effort right you don't want to pull the best developers off of every single dev team because they have an interest in security uh and defecting security wouldn't remove valuable talent from that team right there are if you decide to

ponder this question you should put some real thought behind it there's a couple things to avoid headaches some tips just quick quickly here focus heavily on maintaining the perception of the initiative we do need to be careful about making sure that this becomes something that is beneficial to the entire company and not just a uh not just a time sync right reward and recognize early and often the better that we are saying hey this is wonderful work that this dev team is doing they are per they are pushing securities initiatives and they are an incredible help to the team incredible help to the organization that is great maintain a cohesive roster this is going to be uh slightly harder than you would

expect when you start getting larger we have slightly under 200 security champions so when people leave and come and join different different teams it can be kind of difficult to manage so automated as possible and be able to answer the question when somebody asks uh who's the security champion for this application who's the security champion uh for this team be able to answer that question quickly it doesn't need to take you uh you know 15 minutes go scour through and associate people to work day or something like that um communicate expectations or communicate expectations early with champions managers uh if if you were going to appoint somebody if manager is going to appoint them that's great uh but if they need to

kind of be coerced into joining the uh security champions program expectation should be important so they don't feel like they're being pulled two different ways and uh lastly avoid the program becoming too much of a time commitment i understand that there are lots of benefits to this but the more time that we you know force this to be is going to degrade the image a little bit and lastly here a this idea of security champions can be more of like a framework right this doesn't necessarily have to only apply to security um maybe you have something for like quality champions where they are you know responsible for the improvement of testing reliability and efficiency within applicant within

uh software development maybe you have privacy champions who are focused on legal compliance data privacy things like that maybe you have devsecops champions who are interested in making sure that the entire security or the entire engineering organization is moving towards cicd automation scalability things like that there are three primary guideline guiding principles here that kind of enveloped all of these different types of champions programs which is really ownership engagement and collaboration we take these principles and we apply them to any champions program or any formalized program that that you have it's going to really push an organization in the direction that it needs to be going uh so finally just a couple of resources a couple references here um

big shout out for the security champions 2.0 presentation where we got that general framework and that has been kind of evolving over time and um denim group has also been doing some some good work in security champions and what that means they've done some really extensive research so go check out dan cornell's um presentation on that as well so uh we are running behind we don't have a ton of time but maybe one or two questions

[Music]

yeah so the question was uh how do we you know assign custom training for uh certain teams uh so personally we have we kind of take a two-pronged approach one we provide training from uh things that are relevant so we know hey we've got a lot of net applications we need to you know make sure that we are designing training that has uh inc that includes net specific uh answers and then also uh we are uh going through an effort of curating uh curating content that is open source stuff that's on you know youtube udemy whatever it is and creating courses and saying hey you know this if you are interested in writing secure.net applications these

are great resources and we've kind of tailored those to the different engineering teams that are able to say okay i know exactly what i need to consume to be a better security champion yeah self-selecting is important we want to always be available to for people to pull rather than push security training maybe there are maybe there are some default everybody needs to know os top 10 uh stuff but maybe the more specific training is is more of a pull method thank you if you have any other questions feel free to find me i'm wearing this obnoxiously orange shirt and so yeah you see me from across the room but thank you so much i really

appreciate it this was a wonderful time and thank you for all the organizers as well thank you so much