PG - Building A Teaching / Improvement Focused SOC - Andrew Gish-Johnson

okay so as he had said I'm Andy Gish Johnson a pierogi powered on Twitter if you'd like to follow me we're gonna I'm gonna present about building a collaborative or a teaching sock so first a little bit of Who am I I'm a blue Timur I'm from Pittsburgh I got my start in the steel industry then I went to Highmark Blue Cross Blue Shield it's always been blue team and now I'm with Carnegie Mellon University we're trying to build a we're continuing to build a student-focused Security Operations Center related to all of that I'm also an organizer for b-sides Pittsburgh we celebrated our eighth year five hundred attendees our conference is also at a casino so what am i doing

so at Carnegie Mellon University we are at trying to implement a teaching sock that'll be follow the Sun so the overall idea is students will learn a theory in class they could come to us and then apply their theory in a real security operation center using Enterprise Products it's going to also be a fall of the Sun like a lot of popular security operation centers so students at our main campus in Pittsburgh will be working with us they'll be working with professional and fellow student colleagues at our cutter canvas transitioning work as you know our day ends and their day begins part of the job I'm gonna try to or try to focus on learning compliance we're gonna be doing

PCI and NIST and then additionally there's going to be a collaboration with a research community Carnegie Mellon's a research organization we have a lot of people doing research rather than them build test environments they could come to us collaborate with us test different ideas we've got a network we've got thousands of people thousands of different products let's see what we can do so how did I get here from there or how did I get here from where I used to be I think we've all had the same kind of experiences you know you're working in a blue team space there's this compliants obsession you've just got these bad alerts you've got a missing alerts you

know how many people have the same like low quality high volume alerts like internal failed logins or my pet peeve I just hate them they're almost always an IT issue it there's no real value unless you are focused on meeting that compliance goal or meeting that SLA like I need these open tickets so I could close them so I could meet my goal of you know I need 13 or 20 tickets a day 20 of them are failed logins boom I'm done in a half hour I open you know service now or service manager tickets to all the IT organizations time to sit back and relax related to that you know if you got the

siloing where you're just hunkered down on these tickets you're throwing out ServiceNow tickets to groups you're sending emails but nobody really wants to respond to you or are you really making a place any better your emails get ignored maybe you're using the wrong language hey database admin I saw these failed logins against this host what database is against you know I got a hundred different databases running on this host let alone this is a clustered host like it you know you're not helping me you're just sending me a ticket that I'm also gonna ignore and then finally career pathing like I really like the blue team but not everybody else does I like looking at logs not everybody else

does when I've had junior colleagues like what path do they have how do you get people to find something not everybody wants to go into AI or not everybody wants to go into sim management how do you provide them an internal career path so that you can continue contributing to the organization and they don't just leave and then how do you find people within the company that you know want to help you so I mean that those were the experiences I had had and then finally when I reached a senior level was like it's time to fix all of this so one try you know use compliance for good like map your goods security alerts

to these compliance controls so back to internal failed logins how about you know instead of that because it's always an IT issue almost you've probably got password lock out so you've got to factor authentication or even got like these mitigations maybe of auto block for an IP do you really need to look at you know this alert this system tried to log in five thousand times with the same failed password that's a waste of your time you've stopped it in most cases sir for instance maybe of an alert hey instead of looking at failed logins I wait till a user gets to like a duo prompt and then instead of completing the two factor the second factor they

just sit now that's a much better alert so explained in relation to that to either your leadership or your compliance person so you know hey this alert that's you know a user is stuff to do oh that relates to you know PCI ten to four and it's three one eight it's an invalid access attempt I'm not looking at what's typically an IT issue I'm looking at this because this is somebody most likely trying to access the system that doesn't have access my alerts explain that so the junior analyst can see you the manager can see it anybody looking at the sim I'm running seeing the alert knows how this maps to compliance it explains what we're doing

so you know as a junior analyst like how many people didn't talk to compliance till they were a senior like that was me I just kept avoiding compliance and then suddenly it's my turn to talk to audit and I was just like oh I know PCI exists but that's it so now you know your refresh you're not refreshing you're learning the whole PCI we're like with this junior colleagues that were working on they understood PCI they understood nest because they've been seeing these alerts they've been seeing you know these phrases they they understood an auditor could come why they could talk to any of us I don't have to be the guy who's there like anybody in the room could

explain how we're compliant including my management so so for instance I'm so I've always used Splunk for enterprise security this is a screenshot from for enterprise security that's super redacted if you've never seen this before you go you have like your alert you've got a description you've got some additional fields to help you some next steps that's not really important but you can kind of see in the middle how this alert which was a user performed an action on a splint object-- so this alerts related to you know NIST control 331 about protecting logging systems and PCI control 10 to 2 so an analyst looking at this you know it's not front and center but you know we're

trying to do our job but through repetition of seeing these alerts with no oh you know this NIST control or this PCI control this is why I'm doing this job this is why this makes sense it's not fancy you know Splunk isn't the prettiest interface but you know it is what it is but you've got compliance there you're seeing it every day you're understanding why you have compliance it's not this once every six months there wants every one year thing it's cut gone to a continuous thing so transitioning from the compliance thing the second part I was doing there I am doing is collaboration so in addition to having compliance throughout the workflow we also do collaboration we meet with

fellow IT groups fellow groups within the organization to review what we're we're working on what we're doing hey here alerts we're doing here's how it relates to compliance so you know back to database team here's what we're doing do you see any gaps in what we're doing you know we're focused on the IP stack maybe we should which for a database we should we should be looking at database logs you know that could be an entire giant oversight in what we're doing are we looking at alerts that are entirely worthless are there things we're missing because you know IT people they know what their systems are up to or it could be something like well I see you didn't

get any alerts over the weekend we did a major upgrade why didn't you see anything guard did you know your logins not working anymore or I see your alerts are only for like one log stream maybe you're looking at the operating system and not any of the application logs you see us logging in but you don't see any of the users that sort of thing I've been bringing junior analysts along get them involved you know they were involved in compliance let's get them involved in improving the sim let's get them talking to people improve relations do a cross-training so if you're the junior analyst that maybe has more of a network focus like when we

talked to the networking team come on along like let's talk to them you could kind of build a bridge with that group get your name out there yeah so here it's a example dashboard I mean again it's this is a pretty like a lot of different group products but I'm using spawn so so you know in these meetings like for instance this is back to the do authentication system I'll show a sim for a the duo group it's here's all of the searches we're running but you know here's a sample because you're not to do a team you're not compliance if you do want to see everything Carnegie Mellon's hiring so so like here's the searches were running

for monitoring duo here's the PCI controls the name kind of gives a explanation but we're also there to explain everything do you see anything we're missing there'll be different reports of the kind of logs we're seeing also get these groups integrated into using Splunk so you know here's a sample dashboard of some administrative activity so if we were talking they make I we may say hey I see there was a to FA error message over the weekend do you know what that was about or they may say hey I see there's a Sierra you folks in the Sauk didn't contact us why didn't you contact us and we could say oh well you know we

we did investigate but we had the necessary logs or oh we just totally missed it we didn't even know a 2fa air was something we should focus on so and then also you know sample notable events which is a Splunk term for incidents so here's you know a sample of the alerts we've seen are these things we should be looking for so looking at like this maybe that top alert where the source was in the 199 a range hey don't bother with 199 that's test that's always going to have crazy stuff going on or why was there an alert from the 100 source we could also talk about like you know this reviewer man they really know do oh

maybe we should talk to them and see can they help us maybe they have an interest in being a duo admin in the future back to that collaboration or career path thing or you know explain terms like for the middle one what's in non-anomalous loggin if you know the IT person had a question or why did you give that such a short response well you know in that case we really need to talk to that sake analyst why did they put three words that don't really tell me anything about what they did you know but we show this is what we've been doing monitoring your system this is the value we're trying to provide to the

organization this is how this maps to compliance controls we know you're getting audited we're getting audited you know we're all part of this one big organization let's improve this together this is what we do in the sock all day we're not just sending you weird emails or giving you calls or booting you off the network continually like we're doing things because we have compliance because we want to make the place a better organization can you help us so I've been in this setup for going on two years between the previous two employers I've had pretty good outcomes so like the first year of building a sock didn't even detect the penetration test it wasn't until they were enterprise admin

and they got loud on purpose just to see if we would detect anything did I finally see an alert I mean that's bad that's terrible the second year we didn't block the pen test but we detected them we detected them within hours and alert within days or hours is better than an alert within months or finding out on the news you were breached a year later right this is all continuous improvement sock morale improved we went from hey you know we've got this glut glut of alerts we open alert closed alert our SLA s are amazing but you know nobody was happy to the socks improving people were seeing interesting alerts people want to make the sock better the

whole team felt like they were contributing it was no longer well I'm just the junior sock admin I get alerts in the morning I close them I go home and then I repeat the next day it was Oh what you know this portion of the sock I helped build I helped improve we went from not monitoring the mainframe at all to monitoring the mainframe and it was myself and one of the junior analysts and he was like you know I didn't know anything about a mainframe now we're monitoring mainframes no and then nobody does that I mean it was amazing that we built mainframe monitoring the stock visibility improved people knew what we were working on they knew why we were

working on things you know we had the fishing intake hey I got this phishing message but people were also notifying us hey I did the system upgrade or hey this system had behaved strangely did you folks see anything it people knew about the suck they were trying to help us we weren't just the random people in that room with the locked door that you just kind of walked by or you avoid because they're just like the auditors and they make my life miserable you know but yeah so that's where I started we're in the process of building a student security operation center we need students so come on in a row at Carnegie Mellon University questions up we got to

get you a mic yeah so from Ground Zero and your first like the sock was even just a an idea at that point like where did you begin like how do you how do you start so it may admittedly the first time around you know I had Splunk for enterprise security with all the generic searches finally after like a few months of just spinning our wheels we turned everything off and it was okay let's go from system to system and figure out what makes sense to monitor for some systems like okay if you've got a Windows domain controller there's a wealth of resources online for building decent alerts for other systems we sat down with those groups or

just said hey we're wasting our time here can you help us figure out what we should be monitoring sometimes you got to you know grease the wheels with a box of donuts or a cake but yeah you know you do what you have to do and every group has the person who wants to talk to you and help like yeah you also have to search out that person can you tell us more about how you set up that mainframe monitoring yeah so initially it was flow control just looking at like could we get flow data at the edge of the mainframe we did a little bit with intrusion detection like looking just for like the error message and then the

mainframe team already had reports that were being exported so it was logs that were 24 hours old and then finally we bought a product sync sort that converted mainframe messages to syslog but still that was the you know no offense to them that was converting messages to syslog and then it was back to reading mainframe documentation talking to people like phil young mainframes 767 what should i be looking at and then we got the terminology down so we knew like you know behavior profiling the things that were valuable and not just failed logins so like these users should only run these jobs which are equivalent to program sir [Music] Phil young he's mainframe 767 he's I

think he's running evil mainframe hacking tomorrow afternoon here but yeah he's the guy to talk to for all things mainframe yeah follow-on question about replacing Splunk enterprise security with relationship building I'm doing something kind of similar to that but a little terrified about how long it's going to take like I'm fine with baby steps forever toward the horizon always better but at some point I want to be able to point out and say hey look what we did and and I'm a little terrified about how long that's gonna take so can can you talk about that process a little bit please um yeah the first time it it took about a year so we were doing well

so there was a lot of panic as you know the audits were coming back yeah the metrics were shot exactly but yeah we started turning things off turning new things on as we would write them and because you were working fast anyway now you got a whole mess of new alerts that weren't very good but the junior team understood like hey these are only going to show up for a little bit so where I met yeah so our metrics were wild for a good six months of like teach analyst has a hundred events to each analyst is five to 100 which you know you spread average that out over a month that didn't look too bad fudged

the numbers a little bit yeah yeah oh yeah and then you know we tracked a these renew searches or correlation searches luckily know that we had implemented and luckily we you all purposely we had everything mapped to compliance so we could explain why we were doing it we weren't just throwing you know roles researches at outs like we did this mapping to this compliance role or requirement so I'm I'm very interested in seeing what it's going to be like in another semester or another year because I think you're going to have some really interesting metrics on how you're driving the arms race because your mainframe student and Sox students are gonna get much better and your

little your red team students are gonna get better and each one is going to have some sort of area it's gonna be interesting to watch the results I'm very excited for the red team students which Carnegie Mellon has way too many awful [Music] way too many in a good statement just as a blue team or you know how do you or do you find that analysts run into like I don't know I don't have a name for it sock burnout we're or alert burnout oh yeah we're there stared to you how do you deal with that do you rotate on what they're looking for do you test that you know um so I tried to figure out you

know what do what are people's interest so you know are you a network person why don't you start meeting with the network team with me or meeting with them even more like for the improvement side of what we should be monitoring why don't you try writing some searches and see what happens I'm here other Splunk professionals or in the organization that could give you guide points because you're still contributing to the sock and you're still making us better but you know how do you take a break without but you're still helping us yeah and then for reference that the sock size is around a dozen people so I don't know how this would scale for like the

200 person mega socks that some of these banks have so being a university are you actually using students as your junior analysts like part-time students or so we started over the summer with a small test group this coming fall will be go live the spring and summer have been let's build this out give this as professional as we can once the students show up as well as the reference follow the Sun the students at Carnegie Mellon Qatar when they start showing up Pato how do we perform that transition like a you know junior professional analysts have done transitions where I've been elsewhere but students will be an interesting change for that specifically I've seen two more hands here we'll do

these last two questions and then we'll call it so addressing sock burnout and low morale we're to the top two three ways that you've increased or been able to increase the morale within your sock one get rid of terrible alerts like the failed log and I hate it every analyst I've ever worked with has hated them if there's an alert you keep seeing that you don't like flag it like let's stop looking at bad alerts there aren't too many people who duck who don't want to see an alert because they just don't like it it's usually well this just doesn't provide any value and then trying to get everybody to feel like they're contributing breakdown like

you're a senior you're a junior like we're all part of the sock I don't know everything you're new you should be bringing new insights like you know I'm getting trained in my way like I'm getting old tell me what I'm doing wrong break these habits you know I used to have the manager that got hit with the worms and that's all that person had ever thought about I've been in a lot of fishing and now ransomware in the news like am I going to be that guy that's entirely focused on ransomware because I saw one occur I wants like these new people should be telling me like hey no you you know you're stuck in your way right

hopefully far we're gonna keep getting fetched forever but so may I have a last question if you are in a situation of building go stock in a company from a scratch I don't know sock whatsoever whatsoever and you have a couple of people only as the resources would you be able to define place the main focuses what what should these few people focus on concentrate yeah so if you were building a sock and you had limited resources which I feel like is every sock I've ever been in I've always gone to mail filtering and specifically egress because usually of your ingress relatively secured but the fishing's always going to be an attack I mean we

see that and all the other talks everybody basically seems to come in understanding email too so you don't have that giant training barrier web applications uh everybody I mean that's tough I have trouble with web application attacks so I maybe I keep it on the back burner because I have trouble but people seem to understand web browsing because they could browse the Internet and build the same traffic people seem to understand email because they're sending and receiving and those have always been my two focuses [Music] the failed login thing I mean it it's such a mess from scripting and such all right big round of applause for Andy

[Applause]