Training the Next Generation of Security Engineers

our presenter is going to talk about noobs and training the next generation they are going to talk about the fact that we really don't take the time to foster our news and they're really going to talk about some of the things that have worked for them so I'm gonna let you do your introduction since your best knowing about who you are awesome well thanks everyone as Kathleen said we're going to talk about noobs we were all noobs once and how we're making a lot more of them so first a few words about me I'm a security engineering manager on Google's detection and response team I manage the teams that do detection in Google cloud and I also manage the ATC

team that's going to be we're going to talk about today I use they/them pronouns I write for Korra so you should follow me there I read about security and management big tech I love everything in the outdoors especially physical activity and I also threw my own beer so Sara Young's talk with anybody else saw her talk yesterday got into this problem in depth every company wants to hire like senior security engineers with ten years of experience and many fewer companies want to hire entry-level security engineers with no experience and that creates an obvious gap and we don't have a giant set of feeder programs the way the software engineering folks do there are a few

formal programs Carnegie Mellon's well-known but there isn't nearly the same quantity of programs so it winds up happening as folks go through a program certifications but not nearly the same amount of training as a software engineer and then they wind up in jobs like Tier one stock analyst which is a fine job but it's not as good at training people as we'd like to see and people wind up staying in those jobs for a while and really having to make their own education and talk their way into things and that's not really the best way to build a pipeline for the industry so our attempt to solve that problem is by creating a pipeline and right now our

pipeline has two components we have itrp security engineer career path we'll talk about in both of these in depth that's really intended to be an introduction to the security field it's not a job insecurity and we have a team called automation triage and compliance which is your first job in security so TRP iq residency program is a google-wide program it's been around for a while it's not specific to security in any way when folks in the program are hired on a twenty six month fixed term full-time contract and what that means is they have all the rights and privileges of a full-time employee but they have a scheduled end date and if they don't find another job by that end

day they're unemployed to get into the itrp you only need to have sort of general IT skills you need to have a good set of skills but it's supposed to be the foundation on which you can build your career it's not supposed to be the skills that will be your career and we look for lots of signs of promise folks who have shown advancement in previous jobs good GPAs things like that the itrp requires no coding and it requires no security so we consider folks in this program to sort of be raw material in which we can build the core job of the itrp is to run our help desks it makes for great tech support actually someone

who's a great performer on one of my other teams started off and helpdesk it helped me fix my computer like three years ago she's also captaining one of the blue teams and the CTF and then they do a three-month rotation with an engineering team that's supposed to help them learn about what that team does on top of that they're not expected to work a full 40 hours we set aside a few hours a week for them to dedicate to learning and we latched on to that by developing a security engineer career path and the path has a few different components we put together a curriculum for them to study that coverage sort of security fundamentals CIA risk impactor

likelihood what is encryption asymmetric versus symmetric TLS when you use it detection what it is and all these basic concepts at sort of the definition level and we also suggested that they go through some computer science courses Stanford's intro courses online we recommend that a few people actually do that most people just dump right in and start coding the real meat of this is the mentorship so we find senior mentors with about five to ten years of experience and we pair them with about five mentees each they have one-on-one meetings monthly and they have group meetings every week and we find that this is where a lot of the learning happens because folks show up with lots

of questions things they've read in the news projects that they've taken on their own time and the mentors are able to help them understand what's going on and work through it and the one-on-one meetings can be used to tailors specifically for their needs and often that winds up channeling them towards interviews so whatever it is that they need to work on to pass our interview bar whether it's coding security knowledge applying knowledge that gets focused on in those one-on-one meetings we also are able to have some folks do rotations with various security teams and see one of my former rotators in the audience today hi Mario not everyone is able to do that but it's

not a requisite to learn the necessary knowledge so so far we have had nine people graduate from the program and we have 19 currently in the program about half of the people who start the program drop out and that's okay that's understood some people are just not gonna like security they're not going to be good at it they're gonna find some other field they like better or they're gonna have some personal issue that requires them to drop out that's that's part of this this is an intro not a job and of the nine that graduated we've hired six which is actually much better than we were expecting so if you think about it the mentors are spending about

nine hours a month on mentorship and we're getting hires out of this program that's like way way way way way cheaper than recruiting these people from outside so then I'll talk about the APC team which is your first job insecurity ad C stands for automation triage and compliance to be totally honest it's a backronym we had another name before and we didn't want to change the acronym and have to like rename all our files Pearson accounts so it came up with some some words that fit it the agency team operates high volume low to moderate complexity workflows so basically if you can define it in PlayBook form we take it on and run it and we have about

thirty of these right now and we look for things that are below the skill level of a senior engineer that are not motivating to senior engineers that do not help them learn and grow and achieve their career goals and we move that work to folks for whom it is motivating it does help them learn and grow and achieve their career goals so it's a win for both parties we're also able to invest really heavily in automation which I'll talk about and as a result of that we're actually more efficient even in an absolute sense that executing these workflows and the senior engineers are in other words we're doing it better than they were doing it and the most

important thing is we target hiring for the steena folks with less than one year in industry we hire for what's formally called security engineer level two it's the lowest level on our job ladder and previously we looked for three years of experience for our entry-level positions which is frankly kind of ridiculous so we're not doing that anymore so some examples of things that team works on if you'd want to punch a hole through a firewall we review and decide if that's appropriate we use binary whitelisting if you want to disable that on your workstation we review whether that's appropriate if you lose your phone if you check PII and a source control we handle those things and

probably the most complex one that we handle is vendor assessments a lot of companies very high skill team does the vendor assessment process but what we found is that a great deal of the process can be reduced to a playbook a set of conditions you must do these things and you must not do these things and those are just hard lines and a relatively junior engineer can understand those lines and understand whether those conditions are being met or not and then there's a gray area in the middle where they can make security judgments and if they they reach a place where they don't know an answer they can escalate and we'll talk about that so

this is what we look for when we take on a process and if you're looking to replicate this in your company these are the things that you're going to want to look for number one is it worth transferring do you often enough to make it worth the effort number two we're not a 24/7 team we need some time for folks to be able to learn if they don't know the answer to go find that answer so we don't do tight SL O's in practice we actually achieve a higher that's the lower than we commit to but you wouldn't want to commit to a higher SLO your processes need to tolerate errors the whole reason why companies look for

people with ten years experience is they want to be perfect every time but you don't actually always need that if we make an error and grant a binary whitelisting exception to a machine that really shouldn't have gotten one the the odds that that particular machine will be attacked by an attack that could have been prevented by binary whitelisting are relatively low so you combine a low probability of a non zero of errors with a low probability of an attack that effects has security control and we can get away with a few things obviously you have to keep your error rate low and there will always be an error rate even by senior engineers but you need a

process that you don't need perfection on so like vulnerability response is probably not a good for this the process needs to involve some amount of security judgment people won't learn if they're just being asked to bring the server back up we're not an SRE team we're not a helpdesk there needs to be an element of security judgment on the other hand the security judgment can't require too deep or expert skills because that's not what we are that's not what we do but what it can have is places where expert judgment is required because if we can identify where we need a deep expert we can escalate and it turns out that a lot of these processes that are run by senior

engineers really don't need judgment 95% of the time 90% of the time and so we can have folks who are new to the industry handle the 90% case and escalate the 10% and everyone is happy and the last thing is we need a partner we need somebody who when we hit those escalation cases when a person doesn't know the answer they can go get that answer from an expert in the field so we don't take things and then the other team goes and runs away except one time we had a team get disbanded that was a little unfortunate but for the most part we are in constant partnership with the teams we work with we used to include

automatable in our criteria but what we found is that we're able to automate basically anything so we have a few tools to do that we built a very basic dashboard basically just like a grid view with with a ticket IDs metadata we built a workflow engine that is basically a glorified cron job it looks to see whether bugs have been updated and if so it drops them into the queue and it takes automated actions and what's nice about this we kept the bar intentionally very low for coding in this engine there's no domain-specific language there's a very minimal framework because we want somebody to be able to show up on their first day and add a new condition to a switch

statement like anyone who knows even very basic coding can go in and add a new clause to a switch statement and over time they can ramp up on that they can add a whole switch statement they can add a whole new set of conditionals and that might require integrating with an API and pretty soon they're doing real software engineering and they've built that skill to the next level they can apply to their next job whoops the other thing that we do that's that's really helpful you know oftentimes when you try to automate things you find that you don't have the data you need to make so we push those things upstream and because we're both the operators and the

engineers the same person who's getting annoyed by being happy by having to ask the same question over and over can go into the questionnaire and the code for that and add the question to the questionnaire and you think like users generally get annoyed by questionnaires but it turns out that if you give them an automated decision most of the time they're actually really happy to use it because they get an instant answer and an instant fix to their problem we've open-sourced one of our questionnaires that we use for vendor assessments you can see it's quite complicated you can see from the progress bar on the side that is even more complicated that appears on the screen but we get all the

information we need to make a decision so we can follow the process from here very quickly rather than a bunch of round trips at least that's how it's supposed to work sometimes people answer the questions wrong and it goes badly from there it's barely able to operate these processes in general with under one day of latency we process over 11,000 tickets a year with eight people and these are real tickets like every single one of these tickets at some point in time would have been handled by a human and we're able to take all of that the rest of the organization so this is strictly a win for the business even apart from the training aspects the

training aspects have been good to have 13 security engineers come through the team in the year and a half that we've been active six of those have now been placed out with other security engineers or sorry with other security teams and folks have gone on to great success we've had really excellent results actually when I started this I expected we would have to occasionally fire people and when you join the team I give you a pitch you have a year to have to prove yourself and if you haven't done it by then you're probably going to get fired and I've never had to do that we do have some challenges it's not all roses one of the biggest problems we

have is in practice everyone we hired has been in their first year of work we would like to get more career changers but we haven't been able to do that so it's not used to operating in a professional workplace we've had some challenges with people not showing up to meetings on time not bringing their laptop to meetings where it's necessary making comments that were rude or inappropriate and so that requires some intervention and coaching from the manager an interesting problem we have is we do reviews every six months but if somebody shows up maybe a month before performance review they're kind of still ramping up in their first review by two months later there crushing it they're operating at the

next level they should be promoted they're gonna have to wait four more months before they get that promotion so our review system isn't designed for folks who are growing this fast and that is kind of unfair to them so something I'm working on with HR and the last problem we have this we've been successful enough that we have more security engineers than we can place out particularly this team is located in Kirkland in Washington outside Seattle and some of our teams are not so we don't have a forensics team in Kirkland we don't have a red team there and so if somebody wants to do that job they either have to move or change their

career plan so it's a good problem to have we've been able to find ways to keep these people contributing but it is it is something that we have to deal with we also had offices that this program would have a more diverse set of employees than we typically find at indeed the the itrp program has 60% of participants are from underrepresented groups that is not wider Asian males and the HTC 50% of team members are from underrepresented groups interestingly we are no different on gender than other teams we have about the same gender ratio as the rest of the industry but we have a substantially larger number of black and Latino team members we have a

few hypotheses by this tree we don't really know we don't have any statistical data but our number one hypothesis is some statistics show that folks from underrepresented groups leave the industry at higher rates than folks from I don't really they call it regular like fully represented groups I think is the term anyway so our hypothesis is if we hire those folks early in their career that if that hasn't taken place yet they're still in the industry and so we have to include them from there and inclusion is a really big value that's outside the scope of this talk so we do have to retain them but at least if we can get them we have the chance to do

that and we do believe we're doing a good job by the way I've heard someone say is sometimes focusing on recruiting is just feeding more bodies to the meat grinder so we're trying not to do that another benefit is when we look for people with three years of experience compared that to looking to people with zero years of experience we get to evaluate a much wider pool of talent and so we can find people who are brilliant hard workers who maybe haven't had the time to get that level of experience we're happy to hire them and build them up to that level experience actually what we're seeing is people are getting promoted to jobs that

externally we request three years of experience we're getting them there in one year so and then they're going on to how strong careers after that so it's a huge win in terms of finding really great people the last thing last hypothesis we have as an industry we started all of this work to grow the pipeline 15 20 years ago and some of those folks are just graduating around now and so we think that we're seeing some effect from that but again this is these are hypotheses we don't have the data to fully support them yet so a lot of folks might think this is a good idea let's do this an artifact I hope you think that otherwise

my presentation didn't really succeed so you many of you if you're hiring if you're hiring for senior security engineers your positions sit open for a long time I hired a manager and it took me a year and a half to get someone enroll so that does that headcount is doing nothing for you whereas if you hire somebody that's closer to entry level they can start today and you'll find somebody there's a lot of people looking for jobs and they may come in the door and immediately be executing at a higher level right away or you can get them ramped up and have them executing at a higher level by the time you would have hired a senior security engineer

who you also have to ramp up the other thing is that somebody who has been with your organization from the beginning is going to have a lot more loyalty to your organization than somebody who joined you because you offered them money or a meaningful project and they're going to know your text a can all of the usual reasons why it's good to hire someone and keep them with your organization and we haven't had enough time with this program to know whether it retentions above average but I certainly hope so putting that on my performance review the other thing is we're also helping the senior engineers when we get folks doing work that is at their skill level

they feel happier about it helps them morale good morale leads to good retention and it also leads to good performance so some things your organization needs in order to support this work you need at least three security people if you're so small that you only have two people you're probably just not going to be able to sustain it you need to have some operational processes so if you are a very small company that does everything ad-hoc this is probably not for you if you're a consultancy where most of your work is out of clients site this may not be for you you need someone who will be a good code mentor someone who will take a

really crappy CL and provide sorry changeless diff and provide good feedback that helps that person learn and grow rather than just telling them they're wrong and this is terrible do it again because they are you we are gonna get crappy code submissions and we need to improve and we need somebody who will help them through that process and lastly you need room for growth it's a really bad deal for your organization if you invest in training someone up and then you don't have anywhere for them to go and some other company provides somewhere for them to go so make sure that you have something to do with folks we need to develop them so the core

concepts that you need to be sure when you're sure of when you're implementing this in your organization go back to some of the things I mentioned before you need full power work that is broad enough to provide exposure but shallow enough that it's tractable for someone at entry level you need senior mentors who are going to help people improve and answer questions when they get stuck you need your junior folks to be empowered to actually make decisions to write code and to be able to do stuff if you just want a robot hire a robot I'm sure Google will sell you one and I mentioned explicit about the growth requirement it turns out that if you put somebody on a

low-level team with no career path and just leave them there they're not very happy but if you tell someone you have a year and a half to prove yourself and we're gonna invest in you that's very motivating and people work really hard to live up to a citations and I mentioned we've had nobody fail so it's really really been effective and I say that when people join the team I tell them literally you have a year and half we want to reflect back to the process criteria I showed you before these work at basically any organization there's nothing specific to big companies there's nothing specific to tech here if you have processes that meet this

definition you can do them at your company or your organization there are some key things about empowerment that are really really crucial to make this work folks need to be empowered to make decisions if they're not able to make decisions for themselves then you've just turned them into robots that's pointless it is okay that they are required to escalate certain cases that's part of it that's expected it's also okay if you want every single thing they do to be reviewed by someone higher that's a great opportunity for them to get feedback where they're making mistakes there's there's no issue with that they still have the autonomy in fact people a funny thing is managers always struggle to give feedback but one

of the most common complaints is that I don't get actionable feedback so having a review step where people can provide feedback is actually really valuable to the person and very much appreciated now folks must also be empowered to automate you can turn them into thinking robots and still require them to do a lot of robotic manual rubber stamping and they're not gonna be happy and they're not gonna learn the code aspect of the job I know not all organizations require coding but it's always a valued school I've never found a place that actively wanted you not to be able to code but that's saying you don't need to start off as a strong coder you can be a

pretty bad basic coder maybe taken like one Python tutorial and be able to get started because we can start with really itty-bitty bug fixes and changes and you can learn from there you also don't need any of that infrastructure I've I mentioned we didn't start with that we built that all from scratch it was actually built by people who didn't know how to code they learned how to code as they wrote the infrastructure and we started really small and we built on that and it got better and better over time okay so I think we're a time for questions now the rest is left for you to do so ask me anything so this has

been phenomenal thank you David I really appreciate it and I'm sorry that we had to put you on super caffeinated overdrive that's my usual mode so what questions do we have for David oh come on that was phenomenal I guess I didn't mention I condensed a 45-minute talk sorry I talked quite fast so if people want to have more information or the 45-minute talk may they contact you or ask you any questions absolutely feel free to email me you contact me particularly if you're if you'd like to develop something like this but really anything yeah go ahead reach out how many in the room are training noobs right now or working alongside noobs so is this different from what you've been

experiencing or what you've been doing so is this give you some ideas of what to do so can you make me a commitment that if you instigate any of these can you write a thank-you note to David in a year can someone remember to do that actually I would really appreciate if folks do take lessons away from this and apply them in your organization I would love to get any feedback on how it's working for you you know I mentioned we've been doing this for about a year and a half so I'm sure there are so lessons we haven't learned and maybe you'll learn them first and we can we can go from there so yeah because they

let me know if it's did you learn stuff yeah because I you know the the four or five years that I've been doing this we've rarely really talked about training noobs and I think what David is shared with us is we used to do it just like throw them into the deep end and this is actually one of the first presentations I've seen on actually having a process Richard have you ever seen a process like this before you've worked in a few companies so apprenticeships yes but you know this is this is a great training program well let's give David a round of applause [Applause]