Putting Out the Fire: 4 Proven Strategies for Hiring and Retaining Security Talent

thank you very much glad to be here all right so as an industry we have a layer eight problem right and it's our own fault we say that there's a talent shortage and that we need good people but once we get them we have a tendency to burn them out leaving a trail of dissatisfied and unhappy workers behind us so this talk will cover some key strategies to help change that and i want to kind of caveat it to say that we've had some excellent talks here today about careers changing careers finding careers and there's more to come on that subject this is not going to answer all the questions and this is not about

job hunting right the primary focus here is people who are in a position to make changes and a lot of times that's management but a lot of times it's also those of us working in the trenches if we have an opportunity to influence those who can actually make things change then we need to work on that we need to try to take advantage of that and this is really kind of the the big thing here that i want to see us as an industry make some changes a little bit about me um besides what was in the intro there like i said i spent several years doing digital forensics and incident response consulting i moved

from there into corporate in financial services and i ran our security operations team for several years and i've been with red canary on the detection engineering team now for a bit over four years and i lead our search training program as i said for years infosec as an industry has been complaining about talent shortages and retention challenges but from what i've seen for the most part we just keep doing the same thing over and over again and at the same time we hope for different results and so we'll talk about that a bit as it helps frame up the conversation i'll share some of my experience uh starting a new job and also um as it as it helps show why we

need that change right from a personal experience standpoint and then we'll go through some of the things that we've done at red canary to be the change that we want to see and to wrap up with some thoughts on the future where do we go from here all right so if we have a people problem and as with most issues it's complicated this is not a simple thing there's not a straightforward answer but it is a layer eight issue now most of the time in infosec or in any kind of technology field when you hear layer 8 it's because somebody did something wrong but this is really more about the overarching issue and while there is a certain amount of

responsibility for the individual this isn't so much about that aspect i'm here to talk about some things from the business perspective and what can be done from that side to help retain good people and we hear a lot about having a shortage of qualified people in infosec and we talk about the need to hire newcomers with little experience but then we complain because the newcomers don't know what's expected of them and the people with experience think that they know what's expected of them but a lot of times they really don't and so then we wonder why neither category stays at a job for more than six months to a year right six months to a year that's not

longevity so presuming that we're hiring people that we think are capable of doing a good job and we want them to stay around for many years what do we do to change the status quo a little bit of a side here emphasizing the aspect of what it means to be capable to do the job especially when we're talking about newcomers that have a lack of experience being capable doesn't mean that someone needs to know everything or be able to do everything for that job rather it means that they have the capacity to learn what's required for that job can they think through a problem can they apply logic and real world experience to propose solutions to

problems do they understand how theory can be applied to reality these are valuable traits in employees and when we find them we need to cultivate them and help them flourish this is really important a lot of times we we look for people that have everything that we need for the specific position but that may not that's in some ways that's like the golden unicorn right it may not exist so we need to develop those traits in people who are capable even if they're not immediately ready to do that job and as an industry i think that we're getting a lot better at seeking out new talent trying to create opportunities and ways to engage with college graduates

transitioning veterans and people doing complete career changes there's been some talks already here at b-sides on those topics we're trying to bring in new blood and that's a good thing we're also trying to provide homes for displaced old-timers which is also a good thing but when we bring people on we need to make sure that they are properly prepared for the job at hand and then we need to keep them around and this is where training comes in so we talk a lot and we hear a lot about looking for candidates who can hit the ground running because it costs too much and it takes too much time to get people up to speed to do the job and so the average

number that gets tossed around is that it takes roughly one year for a business to start to recoup the initial cost of hiring someone between you know job search or candidate searches to try to find people all the costs going into screening as well as onboarding and then paying out for that first year it takes a lot of time and money and so as a business it's all about that return on investment right it's all about how we're spending money especially for infosec because it's a cost center right so we try to shortcut the process to make it more profitable sooner how do we do that by throwing people into the fire but there they get burned out and they

leave after six months to a year so if it takes a year to get started to recoup that initial cost and they're leaving after six months or a year you can do the math pretty easily it's a failing cycle but what if instead what if we change that cycle what if we made that initial investment and we took some time to properly train them to do the job we're investing in somebody already we think that they're capable of doing the job we think that they'll be a good person and we want to keep them around as an employee why don't we better prepare them for that if they're better prepared to do the work they better understand the

expectations then they're already off to a better start than the majority of new hires in our industry and that's making a difference right there now every industry every business every job needs structure people need structure it helps us to function but within an infosec there are a lot of silos a lot of tears a lot of you can't do that approaches i work in what most people would call a sock and while socks aren't the sole owners of such things they are certainly well known for it just like knocks help desks infosec in general i.t in general the newcomers that we're trying so hard to bring in who have very little experience aren't allowed to do

anything but triage and then escalate to somebody else to do the next step it's inefficient it's unfulfilling and it doesn't allow room to grow so as soon as they get to a point where they know something or they even think that they know something they leave because they're already fed up with the status quo they're fed up with not being able to do interesting things they're fed up with being stuck in a corner and so those that do stick around and do their job well are forced into management in order to advance to receive pay increases to get more responsibility and that's whether or not they want to go into management for the majority of technical people

moving into management is not a good move we have to have a way to avoid all that now i'm a firm believer in the value of stories as a way to help humans relate to one another and so i'll use my own experience as an example and while this is from when i started with red canary it's not too different from other experiences that i've had and i expect that it will be very familiar to many of you so i started with ray canary in early 2016 and at the time there were a total of 10 people working there i was the fourth hire for the analysis team all of whom worked remote and starting off followed the basic

outline here as with so many jobs the primary difference was that our ceo tasked me with taking notes on how i thought things went since i was very much an outsider at that point leadership recognized the potential for problems for issues and realized that the people who were there wouldn't necessarily be able to see them and they were looking actively for ways to identify and fix those things to improve them and make everything better for everybody that worked there and we're based in denver so starting off i flew to denver and i spent roughly four days on site realistically it was three days ish i mean three days is kind of stretching stretching it a bit uh but three days of

training with the lead analyst who is now the vp of our cert who lives in the area then i headed back home to work remotely on my own i'm a good analyst i've got a strong background and it's a job that just fits me right it's like doing forensic work in incident response work was made for me or i was made for it i'm really good at what i do but that was really a challenge and it had part of it was it had been almost a year since the last analyst had been hired and so many changes had occurred in that time things that everyone else just knew but it was way too much to absorb

without a formal program of any sort and obviously working remotely is different than being in an office but realistically the experience wasn't all that different right there was a minimal amount of informal training followed by the expectation that you could do the work accordingly and appropriately and that's just not necessarily realistic fortunately everyone was open to change like i said they recognized the potential for problems they recognized the need to do things differently i had taken a lot of notes and i was the one who kept suggesting ways that i thought would make things better so of course i was handed the task i guess you could say that i volunteered at any rate this was the first place

that i'd been where this kind of change was not just a possibility but it was encouraged so to everybody listening today everybody here that's in a leadership role this is the place where you get to facilitate change listen to your people entertain what may seem like crazy ideas and encourage contributions that wouldn't otherwise happen and for those of you listening that are not in leadership roles this is your opportunity to influence that talk to your managers talk to your leaders talk to your mentors encourage change to occur right sometimes it's one step at a time and it's a slow process but it won't happen if we don't try all of us working together are what's going to

make it happen not one person grumbling in a corner if as an industry we keep doing things the way that we've been doing them we're not going to improve right that's the whole definition of insanity thing keep doing the same thing expecting different results so as a company we identified things that were important to us and we started figuring out ways to meet those key objectives now this may seem obvious but what i've commonly observed is that we tend to establish what we think the outcome should be what the outcome should look like and then we work backward from there instead we went the opposite way we defined an objective not what it should look like

and then we figured out ways working forward to meet that objective so challenge assumptions don't accept status quo figure out what your actual needs are based on alignment to the business because again infosec is a cost center if you're not aligning your output to the business the business is not going to accept what you're trying to do once you've figured out your needs you've figured out how you align to the business in that regard then you can come up with ways to meet those needs and we focused on four areas that were important to us and here's where we are now we're never in an end state right it's always just current as we expect to keep growing

we expect to keep changing and we expect to meet new needs as they arrive after all infosec's work is never done and a big part of that is training so we've built out a training program that allows us to have a repeatable process to help new analysts regardless of their specific experience coming in to be comfortable and competent doing the core job within a month that's followed by a month of mentoring so that they have a specific team member to help ensure that they get acclimated and connected none of what we do is is the proverbial you know rocket science but it is highly technical and it requires understanding of analysis workflow and internal processes

even experienced practitioners need that initial training the ultimate goal of our initial training program is to develop a new analyst's core competence in order to publish timely and actionable threat detections to our customers that's the key thing that we're going for and that's the core job that they need to be able to do and it's important that documentation reflects what we do how we do it and why if it doesn't it's pretty much useless so we've developed internal resources to help answer questions and to serve as a source of record for how we do things naturally those change over time and if they or their contents are no longer relevant then they're changed or purged as needed

now this also isn't easy it's not straightforward it requires diligence it requires work it requires effort and documentation is a challenge for any and every organization that i've ever been in but it's so important if we don't have it written down then everything is tribal knowledge and if it's all tribal knowledge we cannot expect to get good outcomes from bringing new people on a lot of us industry-wide and pretty much everybody on my team has seen or experienced pain from having a tiered structure and this is a pet peeve for many of us so we did away with it yes we have tears with regards to pay grades and higher level expectations but our job descriptions are the same

and we're all responsible to do the same core job outside that core job of pushing timely and actionable threat detections to our customers everyone on the team has the freedom and the responsibility to be involved in other things that help the cert support and align with the business we all get to contribute to things like detector development platform improvement operational efficiencies threat hunting and intelligence writing blog posts and and giving talks like this if our future doesn't look bright it's probably our own fault in order to provide ways for our people to develop and expand skill sets and increase responsibilities without moving into management because remember i said for a lot of technical people if not most

that's not a good move so we initially stood up the concept of having different practices within the cert these still exist to a lesser degree but in order to better align to the business we have defined a specific set of objectives to focus on everybody on the greater team is part of one or more of those objective oriented teams and involved with helping move our operations forward specific to that objective each of these teams owns or has responsibility over certain aspects of our work to drive improvements these provide opportunities to learn new soft skills new technical skills and expand areas of interest and so here's where we go deeper into that four week training of new hires

now some of you may have been a little surprised to hear that we do four weeks of training after all that means that analysts are going to be taken out of production in order to train others that's absolutely right it's something that we expect it's something that we accept we're doing that upfront investment into our people because we think that it's going to help keep them around longer we don't want people leaving after six months or a year we want them to stay two three four or more years we also don't expect that our analysts are spending a hundred percent of their time working on events or working on threats anyway right that's not a hundred percent of

their job that's the core job but that's not a hundred percent of it and training again is something that's extremely important to us and since we all work remotely we now train the same way keeping the same workflow there's no more going to denver for a few days we use video calls voice screen sharing we use technology to help make that happen which in the current scenario was really beneficial for us because when everybody's trying to figure out how to work from home because of covid we were already ready for it it didn't make a difference at least not to our team and thus as the company transitioned over to doing that we were able to help the rest of the

company make adjustments share from our experience to help make that transition to working remotely go a lot more smoothly so diving into the training from day one to week four we have a training matrix and this matrix lists out topics that tie in directly with our documentation which directly relates to our work our workflow this helps us to be thorough and consistent throughout the process and everybody on the team has a hand in training new analysts across all shifts so we realize that everybody learns and teaches differently and having that variety helps on both sides of that equation so not only does the trainee have a better chance of connecting and engaging with the team at large

especially since we're remote they also have more opportunities to learn the way that they learn and at the same time our analysts learn more by teaching others this has led to a number of improvements in different areas now week one week one is handled by an analyst that is a dedicated member of the training practice this helps establish a baseline and it helps reduce the resource load on the greater team day one starts off with typical hr you know onboarding stuff getting accounts set up then it switches over to training we meet up we review the matrix we set expectations for the training process and then we assign homework so we have a technical video library

that covers a lot of the topics of the analysis process our workflow things that people will need to know in their daily life of work and so those are homework to go through to help kind of jump start the training process get some of the language that we use the phrases the terminology things like that in front of them so as we get into the process things will make a little bit more sense than if we just jump in raw day two we start off with the training analyst doing the driving right they're sharing their screen they show and explain the interface the workflow analysis process etc toward the end of the week the analyst the training analyst is

still driving but the trainee is trying to apply what they've learned so far by telling the analyst where to go next what to do next the analysts can guide them through that process right if they're not sure asking leading questions okay what are we looking at here you know what's going on what is the detector that raised up this activity what does that mean in this context all right where are we going to pivot from here that sort of thing it's not about the new person needing to remember all the things but rather building on what they've learned and retained to that point and we want to keep kind of stretching them out right going

from the most comfortable area to a less comfortable area and a little bit further and a little bit further rather than just great here's how we work here's what we do here's the workflow you're smart you've done this kind of stuff before you're on your own get going ask questions they don't even know the questions to ask at that point and so we find typically the second week that trainees are generally ready to start driving on their own even if they're not super comfortable doing that at first with monitoring right we don't just put them out into the wild what we typically do is we have open call with screen sharing the trainee is sharing their screen and the trainer

is you know virtually watching over their shoulder very closely but they're not telling them what to do next but it's not okay go here click there okay move aside let me get the mouse let me just show you right that's typically what happens in an office we can't do that remotely so it gives the trainee the freedom to fumble around and find it to try to remember to ask questions and if if they're really stuck the trainer can say you know okay what do you think kind of guide them through that process help them to the next step again instead of just being on their own and as it progresses through the week this becomes less and less of an active

screen sharing session they're probably going to keep the screen up the entire week but the trainer is spending less and less time looking at what the trainee is doing so part of this is the trainer is starting to reclaim some of their own time to do their core job and then by week three there's usually um an open call but probably not screen sharing maybe at first but we want to drop off of that pretty quickly and drop off a call entirely if that's possible either of those can be engaged as needed but again we want to keep stretching that new hire out getting them outside of their comfort zone moving to the next step

kind of gaining more and more independence taking the training wheels off if you will the trainee can also reach out to anybody on the team not just their assigned trainer for that day right obviously the trainer is there but if they worked with somebody the day before that they really had an understanding they really clicked and the way that the person explained things made sense and they want to reach out to them that's fine there's no like stigma on that we try to leverage this third week as a good part of their chance to fly solo as much as possible and if things are going really well um you know we've we've had sometimes with

people that do have a little bit more experience as they come on everything starts clicking quickly the trainer may take time to help them get set up with other aspects of what we do so building out their detector development environments uh showing them how we gain stats from the work that we do so that we can improve our process tune detectors things of this nature and then we go into week four this is a week to really truly fly solo the trainee works core business hours not on a shift anymore and they do not have an assigned trainer if they run into issues there to do what everybody else does ask for help given that all of our

training is based on a business week monday through friday but our analysts work four 10-hour shifts we make sure that the new person gets a chance to have a weekend before starting their regular shift alright so we don't just go right up to the end and then kick them over with no time off this week also gives the opportunity to schedule time with other teams within the cert you know intel threat research you know incident handlers things of this nature time with our different objectives team leads or going back to setting up you know development or testing environments things of that nature and i want to note we don't expect new analysts to be contributing to any

of these extra areas right from the start right do the core job get familiar get comfortable with that core job that's the most important thing we can branch out from there but we do want to have them exposed to those concepts we do want to have them with the expectation that they are going to be contributing in ways beyond just that core job getting more involved and expanding those contributions is handled more directly by their manager as they progress so overall we want our analysts to have exposure to everybody on the team to have the greatest potential for connecting for building relationships for understanding how we work not just within the cert but as the

entire company and so by pairing them up with the rest of the team during their training working on all shifts they not only get greater exposure to everybody on the team the different training styles etc but also better familiarity with the types of events that are seen on different shifts the stuff that occurs at 2 pm is not the same thing as what occurs at 2 am and if you've got customers around the globe or in different time zones there's a lot of variation going on and so having exposure to those again helps create a more uh well-rounded a more capable person working that core job and during that process our trainers aren't locked into you

teach this you teach this they have the freedom to focus on areas that seem most relevant at that time if you get a little rabbit trail going off the side about a particular thing that you're working on at that time and that you know you kind of loop through some different aspects of what we do that's great all this is going to pull from the training matrix that we have that you know we make sure that we're covering all the topics that we need to they don't have to be in a specific order necessarily right there are some some things that we need to kind of front load but a large bit of it it doesn't matter

if you do it now or tomorrow or you know three days from now so you've got the the freedom to do what is right in the moment and this also helps trainees to learn more effectively and more efficiently and again just re-emphasizing reiterating we don't expect our new people to come out of training knowing everything but we expect them to be able to do that core job with confidence and independence and so following our initial four-week program new analysts are paired up with someone who works on the same shift to serve as a sort of mentor or guide now this isn't the kind of the typical mentor program that i've seen where it's really all about teaching or

training someone this is more focused on helping them get plugged in helping them get connected answering questions gaining familiarity with all the other aspects of the job that we do and if they haven't gotten set up on things like detector development this is when that happens we still typically find that new people take upwards of six months to really start getting in the groove so we don't have specific goals or markers set of you must be doing x by date y managers work directly with them in one-on-one settings to help expand their contributions to the team again the main thing is we want that core job done and even with the um you know the extended training that

we do and the the you know the careful structure that we have with it we still see that it takes upwards of six months right so if you think about that and when i started and shortly thereafter for the people that we brought on it was closer to a year so realistically we've cut that time in half but it's still six months so if you have people leaving after six months or year they don't even know how to do their job yet and then you have to start all over not only is that bad from a business perspective but it's bad from the employee perspective if if they didn't even learn what to do at the last job and now they're moving

on to the next job and the next job is really the same as the last job but with a different process different interface they're gonna be starting all over it's just a recipe for frustration anyway this is this is where i think it starts to get really cool everyone across all teams in the cert so intel incident handlers even technical support people that aren't directly responsible for creating threat detections they still go through an element of this same training right it's tailored according to what their needs are going to be to understand that process of how the soup is made if you will because that's that core work product right our threat detections as a business

is our core work product so we want everybody in the cert to understand how that happens they may not be doing it directly but they're tied in with it and so they need to know so they may go through two weeks or one week of training on it and since we have a formal structured program we've also been able to build out and offer appropriate training to everybody else in the company whether they're in technical like on our our engineering and development side if they're in business side marketing sales and we've had nothing but positive feedback from that and doing so has helped break down walls barriers and build relationships across the business so you think about that typically

it's us versus them right especially when it comes to people on the sales side we don't talk the same language but because of the way that we've built out this training program and the success that we've had within the cert the rest of the company is willing to go through some of that same training and they benefit from it i want to stress that these are things that have worked for us right they've worked for our company i firmly believe that they will work for other companies but each organization has to determine their own needs and drive changes that are going to help meet those needs i'll just try and do what we did if it helps if it works if it fits great

but look at what you need in your organization how that aligns with the business and how you can help improve the future so we started out a long way from where we are now and even after we began down this path we continue to make changes we're making changes to this very day and there's a good chance that by next week i'll have to update this presentation in some way to keep it relevant right to keep it current with what we're doing the threat landscape is constantly changing businesses are constantly changing and the way that we approach training and retaining our employees needs to change from status quo as well now from what i've observed directly and

indirectly throwing people into the fire as seems to be most common isn't really working it leads to problems of skill gaps job satisfaction career progression retention and talent shortages there's no doubt that we're facing significant challenges ahead but i'm confident that if we approach this conscientiously and start being deliberate in the way that we deal with our people problem that we can make a positive difference and that's it for my presentation um if there are any questions i'm happy to answer and i'll also be over in the track three breakout um if anybody wants to to go back and forth on stuff there

you