SOCs for the Rest of Us

all right it started out I want to introduce stand Harold conditioning talks with restless days Packer one that is genuflect professional from Denver Colorado he is currently a senior security architect and he's at he's gonna work as a retail engineer strategic security consultant his own security architect engineer and post officer so it's a pretty vast and length a background in intersect this talks he's going to talk you can discuss key traits or some the largest and most successful security operation centers but he has visited over the last two to get over to de [Applause]

poule all right thanks for having me everybody out this was an amazing amazing then I live in Denver I've never been to Pittsburgh before and I only knew Pittsburgh from monday night football on Sunday Night Football like man who's got a tough time together me and then when I got done I already got to be gotten tough and me I got here absolutely heaven never been a nicer group of Poland's and all the folks on that is its burdens fantastic um so this mock helps talks with the rest of us I actually wrote this along with a colleague of mine and on a mine info bar we can't be here today he's actually in London and that we actually worked this

one for sponsors downstairs we're not here to sell lock it in any way during this box downstairs the sound pretty hard see where that's their job through um so um one thing I do is one which is that it's pretty cool is you get we have like 13 thousand customers so I get to go around and talk to some of the biggest security customers or salon getting on mostly with when I come in we're just talking not really products so much but about how organizations can have used the security products how their security operation centers work and so we found we were actually submitting a talk for the sand sock summit which was actually earlier

this week in DC with anybody there at the Sam Fox on their engine well it's the only other time we've given this talk is to say there's no duplicates there um so one of the things we do is travel around the world a lot so I'm one of those guys that travels almost 100% and so we thought we would do some of our experiences make some hypotheses about some things that we believe that the most advanced security operation centers are doing and just try to prove or disprove those hypotheses by talking to some of these very very large organizations which you can't mention who they are but they're literally the name brands in many different industries

healthcare computer software online service provider kind of technology companies defense contractors government agencies etc so those are the folks we talked to and doing the research for this pop - to redistrict Care Committee now we always have to put this layers in or doing some caution this isn't a swamp Rock so I'll get there right now um yeah I don't know going to get down going to the bottom but if it's Canadian dentist billing - passionately these molars and foreskin working what's my favorite part over this way work um so look guys my in terms of the document a nice job thanks John for introducing me so I work as well I've been mostly on my career I was

a security practitioner for about 15 years about five years ago or so I moved over to the dark side and ended up working at a vendor so don't hold that against me too much I've been involved pretty much with our been involved a lot of stands over the years side of the BSE serve and a better mentor for them and sometimes speak at their at their conferences as well um my buddy Ryan is not here he also has a long long background so it isn't as approximate knowledge of anything so so I'm not here to speak from an ivory tower we just had a unique capability over the last couple of years to talk to some of these really really

giant organizations and we figure that if we could pull a few things about what they're doing successfully to make them successful and share them with the community - that would be of value so that's what this is all about mentioned a couple times not here to sell anything and please feel free to ask questions interrupt and stop me if you want I'm poppy talk realizing that that that emma dates me I think all right so the way we're going to we were going to do this talk is we'll talk a little bit about some experiences and Padma that I've had some hypotheses that we came up with that we try to prove or disprove how we did that and then kind of do a

mythbuster style analysis of those hypotheses so oh yeah this is that this actually is one of us but it was just used to make a QQ map loads of everywhere that I traveled the last few years so it's a little we can do here about a bill for a long time but safe to say that I'm not I'm not lying about how far across the world we blind and plus secured you that right so you know if you came out all right so the customers that I generally talk to you like these organizations or everyone from these giant banks in Hong Kong you said to these guys you know all of them right so Brian and I will jump into power

settings and meeting with everybody in this building but very large companies the very small ones and sometimes a small one surprised us with the very needs and things they're unique things that they're doing see everything I so see organizations that are you know gotta be ok bought every product is out there right ten moment agreement together right at Keens of people who know how to use those products they're able to detect and prevent pass very quickly in their environment yeah kind of the middle of the road folks right I got my Palo Alto door my fire I and I rely on that almost exclusively but for the most part we got this and then and then the

ugly right to just you know I we got a guy who comes in every two weeks and changes the back updates whether they need it or not and that's that they're not only ITT but also their security team so let's talk about these hypotheses we wanted to go over here a little bit so the first one is is the concept of Tears in theory operation center so we see over and over again you see organizations struggling with staffing right that especially Tier one zapping just seems to be this constant just there I say hamsterviel painting just organizations just constantly trying to find good help it's here one and trying to retain them right for more than just a few months

which it seems like that's a unfortunately that's us as an average that we see happening a lot in that so a lot of organizations that we see the high-performing ones actually you're moving away from that idea of Tears and moving more towards other other things I'll share with you in a few minutes here but that was one of our hypotheses there's also this this concept of tearful on jobs being replaced by automation so it's a very popular thing to say a lot of folks are talking about this we're going to replace Tier one with some sort of automation right that's really just a response to tier one is so hard to staff or and to constantly

manage that there's got to be a better way right so those other thing that we investigated that Fox have data scientists right so this is something that that I think is the popular concept so we kind of set out to understand if that's really the case if we really are seeing data scientists pervasively in these high performing security organizations no one has a good incident reporting tool bit ironic considering my employer but this is we tend to when we talk to these high-performance companies say okay what do you use for your incident response and they'll be like oh yeah we use you want them you never know right here we use weird on it on a file share

or you know something available use a box with something like that and that's surprisingly right because usually these organizations can buy whatever tool they want right the kinds of budgets that allow them to do that and in fact you know we can tend to not see that right so that was one of our hypotheses and also that there is a lot of commonality as well among among different goals and different tool chains and things like that that are in use at these organizations so in order to be someone science that they not not terribly quantitative but we kind of wrote down these people policies and we lined up like seven or eight people that were

willing to talk to us and spend some of their busy time and weeks we came up with a bunch of questions I know you can't read that but you'll be able to see it if you download it but use the questions that we asked to try to uncover the answers to these questions but ultimately there's plenty of my humble opinion sprinkled throughout here oh then we're here to share the best with you Ryan makes great slide economist look important um and we'll share this with you um all right so people are more important as well so before I jump into kind of proving out those hypotheses there was just a few things that we learned that were that

were pretty cool I wanted to share that first one is super important because people are more important than tools right and so whenever you go into an organization this is something that that I do a lot of my day job is we come in and talk to you a customer and they think I'm gonna start talking to them about this product and I say forget about the product is your ever ever going to be able to use that product the way that you're organized right now the way that your security team is set up who your management it is it's ways that your management ears are set out the way that your security engineers and your security

analysts are trying to kill each other all day long need to fix some of those things before you can actually do the tool right at the start bottome and so people were important the tool that's me and Ryan didn't Hong Kong long time very nice um so Trail of Tears right so you can get the reference um for my daughter's a specific project Trail of Tears I thought that was funny I'm not hungry honey definitely not last time I ever but the concept of rigid structure within the security operation center of you know this is Tier one and this is all that they can do this is tier two this is their job they only get things that are

activated to them from tier 1 and this is G or 3 and they're the only ones that can do hunting in your report you know prescribe some some action or something in the environment that's really going away I think the one of the things that that we found as well yeah hey what's what's the perfect Security Operations Center personnel look like to you and these folks every one of them every single one of them said the very most important thing was its curiosity right they wanted their security people to have a curious mind right to be somebody who's not satisfied with the surface level answer the ones that dive deeper right and get to the bottom of field of

the root cause of a problem and if you can think about those types of folks right and creative people serious people probably the worst type of organizational structure that you can possibly think of is is what we look at is the traditional tiered model right where you have you can only do this you can only do that right you're sort of working against their nature by doing that also the most valuable technical skill we talk to these folks was coding right we said what's the most valuable technical skill and they all said it coding right maybe it's recorder powers of PowerShell or goes and a little mentioned go which I was impressed with it I don't know go myself

but um you know playing on certainly so you get the ability right to tie different products together different processes together into some kind of scripting or coding if you don't have to be an application architect right you will need to be a software developer right professional for say it you need to be able to pull things together need to be able to write code very practically so very very important a lot of the boats when we talk to you said hey do the folks in your sock have degrees and a lot of people don't and that's a great note no worries but of the people that do that a what kind of degrees um do people have right in

desert mr. manager of giant high performing sauce or just miss manager will pun intended um what you know what types of degrees to your people have right and there's you know normal stuff computer science and certain types of engineering and that sort of thing but also we're really surprised to find there's tons of folks with with psychology degrees philosophy degrees use it right other other types of degrees where we're critical thinking or report on sort of organized patterns of thought were useful right but not necessarily technically oriented oh yeah this is the best so learn to communicate clearly overwritten and gooder speaking um so hope this is super important right like even is technical right the most

technical person in the room you still need to be able to sell your ideas and communicate them right everybody in some ways has to sell something to somebody whether it be your technical approach to a problem or you're getting buy-in right it's your analysis of a breach or something like that is correct very very important to have those communication skills oh yeah so the sort of emergence of data signs in security operations is an interesting one one of the one of the guys who talked to you guy in Denver runs the security operation center for one of the biggest healthcare companies in the world gichan said you know we're actually not trying to automate to your

one right I mention that earlier he's like we're just trying to build such better detection mechanisms user data science that the vast majority of the things that your one traditionally has done they they just do this aren't there we've eliminated those things right they're not there because the detection mechanisms are more solid because they're based on some of the premises in data science very much a buzzword of course there aren't that many true security data scientists around the organization we thought to add them as their the big ones it's important that type of thing however the I think the premise right that the pieces and parts that go into a data Sciences those those things are

becoming more and more important definitely something to take away right you don't need to necessarily be a master of machine learning right but you should be able to understand what's the distribution of my data right because it makes sense for me to do a standard deviation analysis or the interquartile range or something like that it's just very quickly packing to do a analysis is the spike in activity right into some anomaly those types of skills like practical skills that our data scientist will be a master of those things but security ops people need to have those skills in general right those teachers who borrow from the tactical the tactical parts of that versus on-prem so

this is something that is fascinating to me this is mostly happened since I've been not so operational from the security effective but that's the fact that many companies right actually has less and less infrastructure that's traditionally on from right in their four walls like the company I work for you know we actually have almost no no infrastructure inside our four walls right all of our you know files are stored in a box you know our customer data is stored with a operating service company our you know most of our infrastructure is provided by cloud service providers etc so you start thinking about the things that security operations center needs to do like an imperative perimeter maybe look at some

some post bases post based data that sort of thing that's all kind of flipped around right and collecting the right data through security operations center is also become quite a different quite a different endeavor than it used to be not just for forwarding a lot from servers just logging company from a network device but pulling data maybe from an API or maybe if you had a service that posts data back to you right look at platform as a service it will say you know we're going to actually going to just give us a RL listener we're going to post all that data back to you that's your mob data right I'm so lots of different them lots

of different paradigms going on there for certain yeah I mentioned just earlier the most important analyst trades right and talk to these folks was curiosity and and I think this probably surprised anybody but you know I think that's you know that that means is to hire people right that aren't satisfied with simple answers we're willing that something's just bothering them so badly after they get home from work right that they're like oh I got it I got to dive in right I got to find out why you know that packet looked like it that that log error message said what it is writing and diving diving down deeper so so I do these little hypotheses of

you pop it on earlier so the organization couldn't ask tears and that tear when jobs are being replaced by automation the best Doc's have data scientist no one has a good reporting tool and socks have similar people and toolset so and we're doing this mythbuster style even though they went off the air sadly a new one doesn't count so organizations have new packs here so I think this is something we have a lot of these organizations I mean how do you actually organize your security operations team and many of them told us you look at it as a one of them actually use these exact words and we look at it like a draft right when we go out to hire we

find sort of the best person available right and we don't know necessarily we're not that necessarily is worried about exactly what what their precise skills are we have sort of a team of generalists right they can code the has a curious mind right if they're good communicator and they work well they work well in a team one point gentleman mentioned that that who looks at it is folks on his operations team and it's like a almost like a mission team right where you have this you have a somebody who's just as comfortable working an incident right and responding to an incident on one day as they are the next day like going to the security

engineering team right and say oh let's let's pull something off you that needs to be done right these two systems need to talk to each other you know this thread is healthy needs to be automatically downloaded and ingested into our indoor stand or something like that taking projects like that and running with those and implementing those and then the next day maybe you know working in incident again invaded a next day like doing some threat hunting something like that right so I'm trying to play this you know sort of structure the job and the requirement of the job or to satisfy the types of people that he's trying to attract right trying to attract people that like to do different

things that have a broad skill set that are generalists and communicate on that so make a job being more aligned to that and was pretty cool from the times when we asked a couple of these folks mentioned that tears are actually still a lot of the reason why period exists in the first place is because they might give an NSF fee so they might say our Tier one environment SP so which is make sense that there's these kind of boundaries around them but for a lot of these other organizations I was pretty cool this is the generalist idea because that's a job on a blog right that would be super fun super fun sort of rule to

find yourself in all right automation

so everybody said they want to replace the tier one analyst with automation anything to say eating things is for a vendor to say just very very hard to write hard to realize that I don't think that we see we don't we don't see you I first started thinking about automation for security a few years ago or automated response that sort of thing I kind of I kind of wrote it off and said it will never never work because I had a kind of a narrow view in my own head okay there's alert Flyers then we're going to go and do something automatically right and it's not there's too many things that are scary about that I would you know be suing things

that you wanted human being to look at and confirm and you know make sure that that's the right thing right but and then that's true I think that's still true winning most of us who run security programs we don't want to say turn off our production website right in response to some alert that we got from a certain system in our network we want to keep it being to see that alert and make that halt however that that a narrow human right of automation was wrong on my part right because there's so much more that you can do with automation the big thing I think that we see is is sort of the automated triage and automated context

gathering so we see an alert right and I'm going to go take some crazy action based on that that you're going to go pull a whole bunch more data from your environment right hold the list of running processes from a machine or below maybe even a memory down forensic disk image or just you know say okay we got this alert for this user let's find out the last ten systems of my user logged into or let's find out the last ten email domain they received an email from with a name of the last zero 50 attachments and they received or whatever these are things that are pretty easy to automate in a lot of systems if someone you do a job pretty

well as you can pull these things together right into what I've heard described as a triage package which I like that a lot at that term so that the first time if your analyst rank is really expensive people is really valuable people that we've been talking about the first time they laid eyes on it there's a whole bunch more context rather they have available to them but the first 15 minutes of activity that they might they might normally do to go you know I see this alert you know my normal response is to go look that up in my inventory system and see drones and I maybe search for logs or whatever is all these manually steps right these are

things that could be automated and that's where I think the biggest benefit of automation is coming is coming in every once in a while I can residue somebody says yeah we actually making the actual block something will block an IP address or something into surgery we see a certain moment that is generally the attraction and definitely not the norm so the best Doc's have a design so I kind of dwell on this a little bit earlier jump go ahead but we are speaking this we are seeing the most advanced stop going in this direction I think that there's it's hard to find in security data scientists right there was some great training after data rooms at

the training that's besides yeah so you guys got in the morning from John you got a whole bunch of great information about machine learning in science and so that information starting to get out there and perform with local training sessions and things like that I think the in my opinion racist unless you're going to devote your career kind of to that you might want to do more like what I described earlier right which is to say you know I should know the basics of I should know the basics of how to how to select a machine learning model or like what are the categories of different types of machine learning or just basic statistics right how do i how do i

calculate an average in a meaning and a enter deviation and therefore power range these things are pretty simple like you can do minutes Allen and other tools on those kinds of things right that I think are achievable for most organizations professional and a security data scientist although you do see in more little more frequently than some YouTube no one ever goes into their reporting tool so this one is funny because we I haven't worked for a company that sells such a thing often but we're not seeing anybody out there saying and this is the best way to record all the aspects of an incident or record all the aspects of a threat actor right there's tools out there

I know those vendors out there Drakken act good root man and others they're diving model approach to collecting indicators and information about adversaries and then those kind of things good solution but consistently blown away by everybody's like oh we definitely use this tool or that tool or yet sometimes we've seen both with Chris or something like that for indicator posit or II but definitely definitely not not even a classic tool really that that are that we see consistently out there so anybody wants to want to start good start up ideas you down so this one security operation centers has similar people and cools us so the idea here that high-performing organizations are trending in the same

same direction right moving towards moving towards folks with similar skill sets and you're definitely seeing this you know this this concept of the generalist and the security operation center that can do internet response it can do a little bit of coding and do a little bit of fret hunting and then there's still you know a lot of specialist right product but I call them gilts sometimes registers groups like reverse engineers or really deep threat hunters for her divorce evening operators that really beat cyber threat the threat intelligence analyst writing track adversaries over long periods of time maybe has language skills and you know that support you know the type of adversary they're going after those

gilts still exist those specialty still exist but it's often that we talk to they're looking for ahead folks you can do boxes a nice flow alright so takeaways um we're gonna actually going to write all this up into a white paper full of blog posts probably coming out soon and so and without all the typography these questions attention not the names but the types of organizations that we talked to there hopefully you know that would be useful to to to a lot of folks in this room a lot of other folks to kind of want to model their security organization after the highest-performing lords in the world and with that and we'll finish a little bit early but my contact info is

out there enough to connect with any and all of you and now we have our table downstairs but feel free to come down via t-shirt we have some other goodies down there and with that I will give you a little bit of extra great time this morning thank you very much [Applause]