SOCs and Shoes

is everybody enjoying the con so far all right cool this is a lot more than the five people that i expected to show up so thank you all thank you there will be uh there will be presents for best heckling or questions

[Music] there we go so i'll do a little bit of an intro before we dive into it my name is sean i am the head of forensics and incident response at yahoo that includes security operations what most people would call sock we don't call it that anymore as well as incident response and forensics program and there's some engineering mixed in i've run a bunch of socks built a bunch of sock programs i was a breach consultant way back before that and i host a couple podcasts when i have time i also suck at twitter but feel free to hit me up there like i answer messages anyway so let's dive into it what is the sock

or security operations center in this because i realized when i throw it out on twitter everybody gives you some cheeky answer that sock stands for something else even though it's pretty clear that this is this is what i'm talking about opinions range pretty wildly right i believe the description of the talk says is just a place where dreams go to die anybody in here actually work in security operations by the way oh good quite a few people of those people how many of you really really love it versus how many of you people are hoping or how many people are like hoping for more i'll take that so disclaimer experiences vary wildly these are different everywhere you go

different programs run different ways i mean we run our program very differently than every other program i've ever seen but let's kind of break into my definition here is one of if not the most important part of a security program now we could toe to toe and say vulnerability management is up there pretty good too because vulnerabilities matter but at the end of the day that is your front line those are the people that are there to stop bad actors from doing something why are we here find badness protect users four words very simple your mission statement may vary but we're gonna focus on kind of those four words for now because that'll help break into kind of

the rest of the concept of the talk so what is it usually like well honestly it has the least experienced individuals how many people think that stocks are entry level jobs that they are how many people think that they should be i see a couple we can we can address that they're given little to no trust or ability so click button do thing follow playbook don't think they have low visibility or an over reliance on tools they deal with a mountain of different alerts which breeds into this concept of fatigue and they have almost no control over what that means for them or in their life they're not empowered to find a root cause any cause to investigate to

remediate they can't fix security problems they can only escalate they face this concept of death by a thousand cuts in a lot of ways and they become a stepping stone something to get away from something to get out of so anecdotally what i will say is it doesn't have to be that way i've got folks who have been with me for 20 years in an operations role my average is somewhere between like three and nine right now because it's all about how you scope this to meet the mission so why is it like this ooh moving pictures why is it like this well we hit it a little bit there's a lack of a clear mission or purpose that

exists there's a lack of trust in individuals there's these tiered structures of these artificial boxes that people are given segmentation infighting ego because we don't have any ego in this industry right a need to justify itself to the business why are you here or what most people would call metrics help and because that's how it's done everywhere i got to remember that the images move when i click the button again so let's dive into these issues so let's talk about that lack of clear mission or purpose so what happens if you have that and i kind of broke down all those problems with the reasons why they exist right so you have that mountain of alerts

you're not empowered to find root cause you can't fix things because you don't know why you're there you're there because you're a compliance checkbox you're there because you're the entry-level position that escalates to something else you're not in a position where you can actually make a decision or better a program so let's break that down for a minute why are we here find madness protect users if you can extrapolate everything down to a very simple mission statement especially when you're in an operations world you can take that whole concept and you can say everything that comes in everything that they want you to do does that meet my mission does that help me find bad stuff does

that help me protect my users no well then we're not gonna do it if it does then let's figure out the best way to go thank you for coming late tj welcome it's tj knoll everybody it's tj noll nerd thank you so lack of trust and personnel this is a big one and part of this is because so many people see in operation center as an entry level job people come in they're not allowed to do something because they click this button and they could break something so let's talk anecdotally about that for a second as i look at my legal rep who if she starts shaking her head i have to be careful so if this if this is

bad like a bad story it's her fault not mine bear so i had an intern a couple years ago that i hired who's now a full-time employee three years we had a big case not going to say what not going to say who none of that big case old intern running the case three years experience still a junior technically in our program part of that case involves reaching out to an external third party cso and basically interrogating the individual who did it me as the person who runs the program our cso no the intern who became a full-time employee why because we trust him to do a good job and what did he do a great [ __ ] job

damn good job if you give people the ability and trust to do things they will surprise you and they will do a great job so let's hit that every person that you hire you hire for a reason from the intern which i actually think my this year's intern is probably in the crowd somewhere to you see so because they had a skill because they brought value what that means is they deserve a voice and trust that's paramount if you want to build a good team or you're evaluating teams that you might want to be on and you want to be on a good team find a place where you're trusted where your opinion matters regardless of your level

so tiered structures swim lanes again a whole bunch of stuff that you have to deal with this all kind of comes back into the trust piece but from a business function perspective this one is probably makes the least sense to me i understand the risk that something can break that's a real thing but honestly call that a learning experience move on it's cool the problem is and there are places lower than the basement i have learned that the problem is why short yourself on people like as a business leader why short yourself on a person if you have a junior who wants to do more why not let them what how does that hurt you what does

that do for you does it hurt you if they mess up maybe give them some guardrails maybe but you're just not getting the value that you could out of a person who clearly wants more and then they're just going to leave and go somewhere else and do more anyway so do away with the artificial barriers so we use tears internally for expectation and it's only for expectation it does not change their job or what they do but we expect more from higher level employees but that way every employee who comes in including our interns gets an opportunity to experience and do cool stuff that's really important that gets people interested involved and engaged and if you want to have good security

people who are actually investigators rather than people who click buttons and paint by numbers you need people who are interested and engaged segmentation infighting and ego yes [Music]

hey

you build growth intern wants to do malware do malware it depends on the stock but honestly it's usually a year or less maybe a year and a half but what i was saying up front is ours i mean we're looking at you know three to 20 years somebody supporting them but my experience as soon as they get good they leave for greener pastures so that speaks to the cultural problem of the sock that they're in right so what if the greener pastures don't exist what if you are the greener pasture so what if you make the job interesting enough to be the greener pasture so i'll tell actually i'll hit on this when i talk about segmentation and fighting and

ego because it's really interesting oftentimes in a security organization there are two separate teams well three there's a detection engineering team and threat detection and response there's a security operation center and then there's an incident response function right detection engineering makes sense that's a very specific skill set you know building things and kind of keeping up systems but incident response and security operations what's the difference anybody okay before and after what depth of investigation okay so neither of those things need to impose a difference they don't before and after is a really good one and i like that because the person who investigates before might have more information than the person who investigates after and you might

actually suffer from that likewise depth of investigation you're limiting folks from being able to go deeper why so we actually it's funny as i give this talk we shut down our sock aka we took our ir function we took our security operations function and we smooshed them together we said hey we own everything from start to finish so we own alerting and monitoring we own threat hunting we own incident response mitigation remediation automation all that kind of stuff it's working phenomenally well because we're taking these skill sets where sock people often have way more experience and way more to bring than what most people think because they're in these logs every day they know what's

there they know how to look at it they might just lack that after piece because nobody lets them do it and some of the ir folks are really good at the after piece but they miss the beginning piece because they don't spend their days understanding an environment or digging into something they only get called when there's a problem so bringing those two things together expands both skill sets in a meaningful meaningful way and like lets people do just cool [ __ ] all the time and that's really what it's all about right like let people do cool [ __ ] how did i do it so we never like shut it down we made a proposal as you do on business

stuff that said i think this would be a better structure for a group of teams people liked the proposal the teams liked the proposal which that was the first group that i ran it through was the teams and we said hey cool we're gonna take this we like it we're gonna smush it together we're gonna try it out we built up a whole bunch of new things that leveraged the skill sets we built a more formalized hunt program and a whole bunch of other stuff and said boom here we go and like it runs phenomenally we have sock people who never would have got to work cases and do real investigations doing phenomenal investigations and we have the ir people bringing so

much of that knowledge and stuff that they have from the back part that they're becoming phenomenal monitors and finding really cool stuff right it's an awesome concept and it works really well but the other piece to it we can't lose sight of why we're here at the end of the day that's what's most important what are we here to do somebody recently said to me i don't care what we do or how we do it i just care that what we do is the right thing so if we could all just take a moment and say what is the right thing for us the right thing was not having a separation between ir and operations

because it didn't really work the right thing was how we best protect the business how we best protect the business is by having full scale whole way through the life cycle people who can provide that expertise that was the right thing the ego doesn't matter none of that matters like we're there to protect a thing and that's everybody in infosec that's not just operations people or ir people red teamers the works across the board like we're all there to make a company safer and like to protect users to protect people to protect whoever like that's what we're there to do if we don't lose sight of that we can actually make really good decisions about how we can best achieve that

outside of our own personal goals which are important so a need to justify itself to the business this is a fun one because this is the one where where i talk about metrics and everybody goes ugh numbers math numbers in math aren't that bad but they do get misused a lot and it's really easy to misuse numbers in math spin a good yarn tell a story and tell the right story the numbers at the end of the day serve as a purpose of telling a story to move you in a direction that needs to go be honest about it like don't manipulate them because that's not cool right like data is important but we want to tell a story so what kind

of stories do you want to tell well we might want to know who's attacking us that's a good story we might want to know what they're attacking that's a good story we might want to know if our work input is more than what we can keep up with on any standard variation so like capacity modeling that's a really good story notice that at no point am i talking about individual metrics because those come way later and for the most part like if you trust your people you have less of a need for that until you don't anymore but what story do you want to tell and then how are you going to use that story to get things that you might need

if you use a managed service company right and a lot of folks do can you quantify how much of the whole managed service companies time is spent in your environment versus versus somebody else's and how much money that they're spending that's a cool story because that could lead you to say oh it would actually be better for me to hire my own people right i'd be better off if i went down that route because i could either be cost same or cost better and get better service so it's all about how you use data to tell a story but data is important metrics are important but numbers for numbers sake are worthless don't don't worry about it

figure out the question that you want to answer and figure out the story that answers that question honestly i actually recommend everybody who does this at an analytical level or like as an ic talk to your leadership and say what do you care about and then you can help kind of build what they care about right so this last one this is like my pet peeve the because that's this is how it's done everywhere i've spent so much of my time like building operations programs fighting with people who say well industry standard is this clearly the industry standard doesn't [ __ ] work let's be honest so i'll say this about it we are hackers

we challenge norms we innovate we adapt we change that's what we need to be that's what we need to do there should be no industry standard that comes across especially one that doesn't work that won't cause us to try to change something or make something different like that's important i realize i'm coming up on time here pretty quick so let's put it all together run cool security operations and let people do cool [ __ ] it's really the end of it but questions folks that works surprisingly well at 20 minutes

i mean i don't i don't see any reason why i can't talk about some of it but i'm getting a no head shake so how do you handle the 24x7 do you have 24x7 and so how do you stamp that with some of your more senior people who have been munched into this thing so we we actually kind of stagger throughout the day so instead of your typical like shift to shift to shift we stagger across the calendar for the most part

so it's just like we're so let me hit the first one first right working with compliance is like working with legal hold on i'm getting to a good part here they don't have to be your enemy you can use compliance so people complain and gripe about compliance teams a lot you know what compliance does though they give you teeth teeth are valuable you need teeth to accomplish things so instead of sitting through the audit and going oh such a pain in the ass like sit down and talk to them about what your real problems are figure out how you can map those to different regulations and then figure out how you can use that to

get the things that you need to move forward that's teeth that can push the businesses to do things that you need them to do or the rest of the business to do those things look at them as friends and not enemies

you always need leadership to buy into a thing like i hate to say it but you kind of always do yes about what his job the is uh he gets to come to me and be like look here's this issue uh we need to change it and i go great let's formulate that in a legal plan and tell the rest of the people on compliance and leadership that this is actually a thing that hurts she is an awesome lawyer you wouldn't be able to do this if you don't forge a partnership with your legal counsel legal counsel are kind of like uh rabid cats you have to approach them carefully and be very nice with treats

and then she's right that i actually gave her the ticket to come here very nice with treats also come up come up and get a present you come up take a present yeah you get treats too good question as far as the cool [ __ ] stuff goes look man it's like it's ir like ihoar is inherently cool like working cases and incidents is inherently cool like that's like the dirty little secret of being in this field like you don't want anything bad to happen because like you want to protect the company but like you kind of do because it means you're going to have a really interesting day week month whatever but by the end of it when it's

multi-month you're like oh my god no i'm done i never want this to happen again i'm not gonna get into that we actually did we did a podcast on that with my ir team on the paranoids podcast if you're interested um yes

which thing

we're lucky we're internal and we're private sector so i don't really have to deal with that

so my background is managed services so i can actually speak to it in that background so when you do any kind of transformation it's really all about focusing on benefit right so in this case we're basically up leveling an entire team and expanding job responsibilities in a meaningful way like that's an upsell that's an up level you're getting more out of this you're going to get better things and this is how those things are going to look oh and by the way you had these complaints we're going to address those things too that's how we're going to make this work

sure so training is almost individualistic for a certain extent right like every person has different goals different needs different ones ways that they want to approach problems for some people that's like sans and official training for some that's shadowing and working with other people i'm an interactive learner so i don't actually learn unless i'm part of a conversation for the most part like i don't do school well i don't do books well i don't do any of that so understanding how people need to learn and then being able to tune what works best for them but i'm there's a phrase that i'm a big fan of you know everybody says like infosec is sink or swim right

i hate that phrase but i will drop somebody in the ocean but i'll be standing right next to them on a boat with a life vest just in case they need it uh who else go ahead right over there i'm gonna ask you you call it trust right but how much of that is actually mentorship and pushing someone to what they need to do to be able to actually get them to do it right i feel like a lot of operation centers you

i would say that that's actually a good question so i think if you're gonna like if you're gonna take the time to mentor somebody you almost have to trust them right like trust is the foundation of all relationships in the existence of the world like you must trust people that's the very foundation of our society if you don't trust the person then you're not going to help the person you're not going to want to interact with the person you're not going to want to do any of those things so like foundationally trust has to come and then all the other stuff like mentorship comes after it that is a good question though also you as well both of

you come get presents also am i like way over now or am i good am i good okay thank you very much

yes however it's based on expectation so like i expect a more thorough something from a mid-level or a senior now there's one stipulation to that and that's that we have i have another team called specialists who are primarily focused on project work which is you can actually throughout our structure without getting too much into it you can go a leadership route into like associate leadership or you can go a specialist route which lets you do mostly engineering project work outside of operations as well but operations is really where all the cool stuff happens because that's the full ir lifecycle process go ahead so based off of your experience i've heard the same type of structure what not in like the financial

sector how easy or how hard you think it would be to kind of push this type of concept

there's a reason why i don't work in government anymore um they're very resistant to change but if you have it ultimately comes down to do you have a good leadership team like i am very lucky my leadership team the whole way up the chain like my boss one of my best friends in the world like my wife thinks we need to go take cute pictures together um and my whole leadership chain trusts me and the team to do what we need to do effectively so building that trust in relationships and again we all come back to relationships and the foundation of trust build trust make change like decent theme hopefully that helps a little bit

depends on the organization probably in the

back about how much so actually is pw crack in here because if he is i'll quote him red teamers don't do any work just throwing that out there but no so look the the red team back and forth ribbing is all fun and i love it but like as long as we all remember that we're all there to do the same purpose and to protect a company like we have a fantastic relationship with our red team we work hand in hand with them on a lot of stuff it's really helpful and meaningful and we can use them and point them in a direction much like we can point our lawyers in a direction or our

compliance people in a direction to solve the problems that we see because operation sees all the problems let's be honest there's probably time for one more folks who's going go ahead everyone okay

um ask me later i'm not going to say that on a record you're

like welcome reaches no incidents like multiple security is great how do you help a leader like chop that up and say hey so at the end of the day the hope is you know for everybody else but the ir team that there's going to be no incidents and no breaches right but that doesn't mean that there's not attack paths that are valuable that doesn't mean that we don't want to think about how we're going to get attacked that doesn't mean we don't want to think about what's going to be attacked and how we protect things and constantly expanding and i don't just do that to explain to leadership what potential risks we have or where we're at right i'm real key on

like even as i bring in analysts like i don't want them to worry about the alerts as much as i want them to worry about like what data goes behind it and what visibility gaps might exist and all those kinds of things that make up a bigger security program like that is my baseline expectation from an analyst which is probably a little different than you know what most people think of like an operations analyst but i tend to expect like think about the why and think about the what if and how could we go do something and i mean we're always busy not like from a breach incident any perspective or anything like that but like there's

always more to do there's more to hunt there's new things hitting the news there's all kinds of stuff to keep us occupied to look for bad because that's what we're there for find bad protect users that's right on time thanks everybody