PG - From SOC to CSIRT - Ben Butz

welcome and have a good time good afternoon can everyone hear me all right so my name is Ben butts and I'm going to be sharing with you my experience in moving my incident response process from a security operation center to a c-cert before we get started do we have any blue teamers in here any response people oh wow good portion I was more than I was expecting all right so a little bit about myself I've got about 13 years in information technology with about eight of those in information security with the last six years being dedicated to Incident Response I got my start in the United States Army after the army I moved on to a company called Alliant

Techsystems if you haven't heard of them they make space systems advanced weapon systems basically things that give our military a competitive advantage and now I'm with target's cyber security incident response team as an incident handler so I've had the pleasure of being part of the foundation of two Incident Response Teams so at ATK we formed the intelligence and response unit and now at Target I was there at the beginning of the cyber security incident response team right so I put this talk together because I I didn't see anybody really addressing the issue that there's a lack of Incident Response knowledge skills and abilities so what I'd like for you to take away from this presentation is an urgency to overcome

the skills gap so if you're new in security I'd encourage you to pursue a career in Incident Response it's very rewarding it's exciting it's fun if you're not if you're in a position to influence your company's security program I'd encourage you to emphasize developing your incident response capabilities and if you're in academia anybody from the university here a couple all right so I'd encourage you to teach this stuff I don't I don't see this as being taught so it's particularly important there couple of recent events here you've got wanna cry and the not petia that have had global impact so they've affected organizations around the world and there's a lot of focus on well they're

not patching good enough they're not and that's true right but what I don't hear talked about is they're not responding to incidents correctly like even if you're you know you've got a patch I don't want to come off as don't patch but you've got to be able to respond to your incidents right so when I started at ATK that was my the beginning of my my path and incident response that was an interesting experience for me I got a lot of experience quickly in dealing with very advanced actors and responding to those types of attacks so and that's when uh the the word threat changed for me so threat before it was it was a

virus it was exploit it was something you know something created tangible @atk I realized threat meant the person on the other end of the keyboard controlling systems on my network trying to accomplish a very real mission right and that that threat could respond it could change its tactics based on what I did that's when security really became cool for me right and uh developing target see search so it caught me off guard when staffing our see serve we have a 24-hour operation very well staff well staffing that out it caught me off guard that the the lack of people applying for these positions I had Incident Response experience so we hired a lot based on

potential and motivation willingness to learn and we trained them up to get them up to speed so apart from my personal observations there are a couple of studies out there that support that the idea that at least there's a perception of a skills gap so the Center for Strategic and International Studies published a report they called it hacking the skills shortage wherein 22 percent of respondents reported that they suffered reputational damage as a result of a shortage of security skills over half of us respondents reported that they outsource their cybersecurity services so I'm not saying outsourcing is bad or you shouldn't do it it's not good or bad it just is but what that tells me is

companies don't feel like they're able to do it themselves so they need to go outside and and staff it elsewhere sans also publishes a report the incident response capability survey where 76% reported that they did have a dedicated IR team but 65 percent reported that their IR team lacked skilled personnel so the organization was there but it wasn't staffed with skilled personnel and one thing to call out from the slide for the maturity of respondents IR capabilities that adds up to seventy-five percent rated themselves as either immature or maturing right so why do we have a shortage in IR skill sets I think some of its got to do with what we task our sock to do right we've

got them our security folks are tasked with a lot of security engineering stuff which is great you need to do security engineering but while your sock is focused on on doing these things they're not focused on responding to incidents so I think too often Incident Response is viewed as an additional duty and not a full-time job right PCI other compliance also complicate this so here's here's what PCI says about Incident Response like that's it like be prepared to respond immediately to a system breach be available 24/7 to respond to alert staff but provide appropriate training to your staff it's it's very very high level and if the the people at the top don't understand Incident Response how could they ever

hope to to do these correctly right so an incident response plan is its if you just need to check a checkbox that'll do it for you but you need to have people that come in day in and day out and search for adversaries on your network ir is not something you can do on the side you might ask well if I don't have constant instant incidents why do I need constant response capabilities and so maybe you don't but what you do have are constant attempts you're constantly somebody's trying to get in phishing attempts drive-by web browsing so while you're incident responders are not responding to an actual incident that need to be hunting they need to be reviewing emails for

malicious documents analyzing links detonating attachments to see what would happen if a user actually did that and then take that information and look to see which users did it you also need to be looking for signs of undetected compromised so things like malicious C to traffic command and control traffic looking for signs of persistence mechanisms there's a whole host of hunting strategies out there but basically they need to be looking for the presence of presence of an attacker on your network right so if you've got a sock right now if your organization has a sock and its compliance focused you're probably spending a lot of time in the availability portion of the CIA triad responding to outages network outages

system outages figuring out what change happened that made it rolling it back doing that kind of work you're probably doing a lot of helpdesk stuff to advanced helpdesk right if if the security of the network engineer can't figure out you know why something's not working suddenly somehow it becomes a security problem just send it to the sock they're there 24/7 anyway they'll figure it out for me you got to get out of that business all right so if you were following PCI guidelines some alerts you'd be interested in looking at look like this multiple failed logon successful logins so this is this is kind of what I experienced when I arrived at Target in 2014 it was very compliance focused and

for a retail company that's important compliance is important you have to you have to get your accreditation and everything that's all well and good but while we were doing this it was bogging us down in non-adversarial activity right so our top firing rules included they had names like excessive successful attacks access failure monitoring right and and when these happened multiple times a day the resolution is you know hey who made this change oh that was me okay well it's business justified move on right you're getting away from the important adversarial detection and response all right so what we have in place now we stood up our cyber fusion Center in late 2014 and this is what our C cert looks like so

we're made up of three primary groups the incident response team is 24/7 there are people right now reacting to detection alerts doing hunting looking for adversaries on the network the threat intelligence team is determining which which actors may have a significant impact in the organization discovering what they're TTP's and i OCS look like they feed those over to the threat detection operations team who builds detection for the incident response team

so one main thing that we did is we took our compliance monitoring and we couldn't get rid of it you can't just stop doing compliance monitoring but we paired it down a lot we took all of the rules we looked at them and we said is this actually providing is this meeting that compliance requirement if not get it out of there we don't we don't need that but the ones that that were providing compliance requirements we offloaded this to a separate team it's a completely different set of people doing compliance monitoring than doing incident response so we can free up our incident responders to do the adversarial the in my view of the important response work

all right so in staffing out our C cert right we look for a course of skills for our for our analysts so they've got to have fundamental IT skills you know they got to know the systems in the environment they have to know how networking works the basics they need to know forensics and I'm not talking about cop style forensics get a disk image you know do the ash do the the change you know all through the line we're not trying to prosecute anybody right our mission is to stop them from from accomplishing their mission on our network so we do very fast focused sniper forensics we identify the the artifacts on a system that are likely to

contain the evidence that we need that would paint the picture of what what happened they need to be familiar with enterprise logging the logs that we have available what they mean how to get useful information out of it how to search it effectively and at scale and they need to be familiar with Network forensics so we use bro IDs it's phenomenal I love it it's we've got a very large sensor deployment so it basically takes your network traffic it turns it into logs and then you feed that into a similar or something else to search on it's it's fantastic you can build reporting out of it reporting alerting searching anyway your your analysts need to be familiar with

Network forensics they need to know how to capture packets and how to analyze packets differentiate normal traffic from abnormal traffic they also need to be effective communicators so they need right--we're 24/7 so somebody who's working on a case needs to be able to hand off the case to the next shift if it crosses shifts it needs to be effectively communicated keeping great case notes and they also need to be able to handle high-pressure situations it can be very stressful when something big happens when something important happens you typically get senior management who at least want to know an update you know every so often which completely understandable but it adds to the stress level you're trying to figure out what's

happening you got to deliver you know what you know what you think oh what you found out you thought you knew no longer is the case you got to keep monied to be able to handle high-pressure situations all right so I'll walk you through a scenario an incident response scenario and when I do this try to envision this in your organization if you have an incident response team now kind of compared against your processes and how you see things playing out right so we start with intelligence our CTI team so in this scenario they're monitoring an advanced group that they suspect is is you know posed as some kind of a significant threat to the organization

and they observe this group registering some infrastructure in this case domain names so they send that over to a threat detection team who builds the detection for it and before you know it we're getting alerts for traffic to these domains to one of these domains so the C cert sweeps it they investigate the network traffic they find all the hosts that have been communicating to it they see that it started about a week ago they've got a timeline to start with and they start triaging systems so while conducting the system triage they discovered that the traffic is being generated by a malicious process that process is so it's making the communication but that process there's a TTP in there where

that process is maintaining persistence on the system so when you shut off the computer it turns back on it's still running for them so that's what we call a tactic technique of procedure that's what that adversary uses as a method so it's not a MD of that file that that virus I don't like to use that word but you know what I mean so we have a tactic technique and procedure to search on so we scoped out the environment we look and we find additional hosts that have the same persistence method with different malicious processes and those different malicious processes are communicating to different command and control infrastructure right so now we've gained the information we we know about more

affected systems more of the c2 infrastructure that we that we previously didn't know right and so investigating these systems we see one of them me me Capps was run dumped credentials and we can see that one of the accounts started being used on other systems so now our C sir has identified systems that the attacker has access to as has touched logged in and manipulated without the use of malware right they just logged right into them all right so after all the identifying all the systems with the c2 traffic all the systems logged into with the dump passwords they think they have a good list of systems in scope for this incident and they implement a host based

isolation effectively shutting down all network traffic to those two or from those hosts except for traffic that would allow the analysts to continue investigating they also implement a DNS sink and a sinkhole to the domains that we know about and so that way any traffic from hosts that haven't been isolated will you know resolve to a system that we control we can alert on it monitor it or even manipulate how the server would respond to those traffic calls all right so once we have all the systems isolated and contained we can start digging into the systems individually try to develop a timeline of activities what happened on the systems and trace it back to a

root cause so in this case we were able to identify that it was traced back to a phishing email where an engineer was tricked into installing a malicious flash update right so in your organization's if this sounds similar to your processes and procedures for responding to an adversarial based incident response good on you you you're doing a great job if it's not right you got to make some changes because that's how you do Incident Response how do you get there so branding I mean what's in a name a lot of things are it is just a name so if you continue calling your organization a sock Security Operations Center that's got a connotation with it

that's got you know the network people say I can't figure it out send it to the sock it helps to change the name so ATK we had the Incident Response Unit diligence and response unit we had a target it's a cyber security incident response team it's useful in differentiating the change in the organization but more importantly you need to allow your Incident Response you need to allow incident response to be a career path it takes a lot of personal commitment to learn these skills learn how to do incident response and so you need to make it worthwhile for someone to do that they need to have the next step and two steps higher than that they

need to be able to advance in the career doing incident response right and you need to invest in your people they need training so you're like there's no bachelor's program out there called Bachelor of Science cyber security incident response right if there is I haven't found it but it's just not a thing even if it was a four-year well-rounded degree it doesn't have time to teach these different things it it's not you can't come out of college with Incident Response people at the bachelor level masters programs you know I'm seeing a lot of policy programs risk management programs I've seen cloud security network defense those are all great emphasis but they're not Incident Response I've not seen Incident Response

in those offerings and lastly you've got to get your people out to conferences like besides Las Vegas they've if you can't get everybody out to Las Vegas once a year get them to attend the local security conferences local information security meetups there are local hacker community events capture-the-flag events and if there isn't anything local maybe start something but basically you have to get your people out there communicating with their peers communicating with subject matter experts and networking because this is changing all the time all right so I just want to emphasize before the end of it here you're not doing incident if you're not doing incident response correctly do it if you're interested in a career incident response reach out to

me on LinkedIn on on Twitter if you want to find out more about this or just have questions or anything reach out to me I'm more than willing to to discuss alright well that's all I had do you have any questions yes I to questions if that's okay you answer the three teams you mentioned the detection team being 24-hour operation would you recommend or do you the other two teams also operate on a 24-hour operation so the way we have it is they are they're reachable 24/7 they don't all need to be there not there all the time staff 24/7 but if if the our nightshift needs to reach out and say hey we need detection on this thing now

they can get ahold of the right people and make that happen thank you second question on your stakes your case study you mentioned looking for processing services on other hosts oh sure um you you mentioned brewer's tooling but do you have any tooling for host based indicators that you might recommend so we build our own there are free tools out there so if you're looking for processes there's the proc proc Mon and I don't know why I'm brain blocking right now but there are yeah nurse oft has some great free tools to use yeah tasks list schedule tasks yep you mentioned yeah of course most places are compliances of course is important to most people but um as far as focusing on

that actual skill set or certain training is there anything that you could recommend where people start like some people are just looking for a really good starting point to see where they can move forward to actually go on the right path versus the compliance and policy sure I've attended sans training sans is expensive it's hard to do I've been to Carnegie Mellon's software engineering Institute they had a great program yeah I would recommend some of those I don't know I'm bringing brain farting but research it they're out there some google search and can find them yeah you want to talk about the level of automation the level of automation that helps you put those three teams together

sure so we we're trying to build our own as much as possible its pricing licensing it's got you know to do with if if we have an issue you know if we reach out to a vendor to change that issue that vendors got other customers with competing priorities whereas we can just make our changes but yeah automation so we have we have an automated process for collecting evidence off of the system and I don't mean to give plug outs to you know any of the vendors I don't I don't work for them or anything but we do use tain iam it deploys our custom-built IR package brings the stuff back it also does our host base isolation tinium doesn't our

well tain iam claims that they offer something similar we have built our own and use stain IAM to get it there thank you all for joining us today yeah we are ending for everybody going out to lunch so thank you for your time and [Applause]