BSidesCincy 2015 Livestream

Alright, now down, is this stuff saying stuff saying stuff? Alright, is that me? It's too delayed?

Alright, well I'm in front of the camera now. The whole delay thing does compli- yeah. Cause I'm delayed

Right there, I'm delayed a little bit. Noise been away?

Are you hearing it okay?

Okay, they got theirs on channel 2. So I'm curious, if I take this to channel 2,

I changed channel to channel 2 and I want to see if that screwed anything up. And I have this going for recording and I'm going to stop this recording to see if channel 2 gave it any more noise.

shitloaded ground noise there. And I want to see if that screwed anything up. And I want to see if that screwed anything up. And I have this going for recording. And I'm going to stop. Alright, so I'm getting you on the broadcast, on that mic, and it sounds good on the radio.

And where did you have to open that? I mean, how did you get to the stream? I searched for YouTube, besides that, and it was published.

Alright. Cool. Hmm?

Thank you.

Pretty cool. Audience coming across perfect. You all set? Yeah, I think so. Can I get you anything mainly? I'm good for right now. Do we have any soft drinks? Diet Coke's anything? Diet Mandu, anything like that? Where they at? I can go get it myself. In the lounge. Where's the lounge? Oh, okay, same room, okay. Oh yes, you will need my mic. We have to use both microphones. I'm getting the recording audio and the stream audio from this. Okay. As long as it's on, bit light, it's on. And, uh... Should I switch it off now and say battery? Yeah, but...

Red light on that one. Yeah, because we've got two different ones. One is going to stream and recording and one is . Yeah. We don't know how to computer around here. Is this? Yeah, we're good. Mic's on.

Yeah. Is that mute? It also mutes in power. Okay. Jason, you want to drop the podium mic and see if this works? Check. Good. Everybody can hear me? Awesome. Well, let's get going then. So, welcome to B-Sides Cincinnati 2015. I'm Josh Omer. I'm one of the members of the organizing team here. We just want to welcome everybody helping us hopefully make the second year even better than last year. So for anybody that hasn't attended a B-Sides conference yet, just want to cover what we're all about. And this is a quote stolen from the Security B-Sides website. It's really a community-driven framework for building events formed by information security community members. So it kind of takes the larger vendor events out of it. It's all about

us and the content we're concerned about. and the things that we think are interesting.

So I just want to talk about kind of the building that we're in today for anybody that isn't from Cincinnati or is new to the area. You're in Union Terminal. This is probably one of the most iconic buildings in Cincinnati. It opened up in 1933 and served as the main hub for passenger train service in Cincinnati up until about 1972. And really from 1972 until about 1990 it went through a couple re-images, different type event space. It was a mall at one point. Really until the city of Cincinnati took control and decided to turn it into Cincinnati Museum Center. It's pretty much renowned for being, for its Art Deco style. As many of you

You may know it's also the backdrop for the Hall of Justice. As you can tell from the t-shirt design as well. So now it serves as the Cincinnati Museum Center here in Cincinnati. And that encompasses the Cincinnati History Museum, the Museum of Natural History and Science, Duke Energy Children's Museum, Cincinnati History Library and Archives, and then the OmniMax Theater. If you've never heard of that, that's something really awesome. You should check it out. if you haven't. But they recently just voted to renovate the entire building. So you've got about a year to enjoy everything. And then the building will shut down actually for three years until 2019 when renovations are complete. So if you get a chance to get back, if you're not

from here, definitely spend some time in this building. There's a lot of cool things to see. So give you kind of a lay of the land.

you are in the Riocart Auditorium. And there are bathrooms directly across from here. The women's is off to the right, men's is to the left. We have a separate hangout room in the Losanthaville Cafe, which is if you go out into the Rotunda and out to the right, you'll see a B-side Cincinnati sign. That's where we'll have lunch. We're also streaming this. into that room. So you all got two drink tickets. They'll be serving alcohol in there later today if you are of age. Feel free to grab some drinks, watch the stream from there. We've also got a lockpick set up in there. There's, I think it's Cameron is his name. Came up and is

helping us out. He set up a bunch of lockpick stuff in there. So he'll help you walk through if that's something you're interested in playing around with. He'll be there most of the day. So just some guidelines. While you're here, please keep your badge on. That's how we'll know who's supposed to be here, who's not, who gets food, who doesn't eat today.

But we'll make sure you all get food. We'll do our best. Respect the other attendees. They're all here. We're all here for everybody. This is our event, really. I mean, we put it together, but it's all about you guys. So, respect the attendees, your speakers. You must be 21 to drink, and if you get too drunk, let one of us know, and we will get you a cab. And if you have any other questions or issues, just find somebody in a yellow shirt, and we will help you out. So, just kind of go over the agenda quick. Coming up at 10, we have Harlan Carvey, who was kind enough to come down from DC, right? And he'll be talking about automated, or I'm sorry,

lateral movement. Then followed by John Davison, he'll be talking about automated detection strategies. We'll do lunch at noon, and unlike last year, we'll stop for lunch. There will be no talks during lunch. We'll pick back up at 1 with Jesse Lanz, PowerShell for incident responders. I think that title has changed from what he told me. 2 o'clock, Coleman Cain will be covering cyber intelligence, collecting all the things. From 3 to 4, Justin Hall, response ready infrastructure. 4 to 5, Mike Reeves, distributed computing approach for network detection. Then 5 to 6 would be Chris Tyo with the value of a simple DLP program. And then at 6 o'clock, I'll wrap things up and dismiss everybody. So that's kind of the plan

for today. On video, Adrian Crenshaw was kind enough to come up from Louisville and help us out. So we've got it streaming over YouTube. I don't have the YouTube details up here. But Adrian's a seasoned vet. He does a lot of the cons. We're very thankful for him to come up and offer his help. Thank you, Adrian. You all should have the details for the wireless, but here it is again. The access points B-side Cincinnati or Cincy and the passwords Queen City.

I just want to say thanks to our sponsors GE Aviation, Ashland, CBTS, Puppet, FireEye, and we didn't put ThreatBud in here, but they agreed to let us use their stuff too. We will be sending some feedback out after the event to kind of figure out what we did wrong, what we could improve on, what you guys enjoyed. So be looking for that email. And then unless anybody has any questions, we'll get going. And, you know, thanks for coming. Really appreciate it. Cool. We'll get Harlem ready to go here.

Well, I was in the Marine Corps, so. Good to say, yeah. This is probably a lot. My dad is coming up. My dad's a Marine. This stuff is probably a lot cooler than the stuff we had.

I actually had dinner last night with one of my college classmates. We went through training together. Oh, yeah? Yeah. It was a good time. Hadn't seen him since we graduated comm school in 1990. Holy crap. Let's see. So which one should go first? I'm going to say this one. The recording one I'll put up higher. God, look at all this crap.

Yeah, I'm good. And it's just PowerPoint, right? So I just cycle on through it? Yes, it did. Cool. No, I got a

Hey, which one of these is yours?

You don't have a parabolic mic or anything? No. Okay.

Or a boom mic.

This is so bizarre.

to stay up here? All over. I got the entire room, right?

I mean, I got all this swagger, right? Look at me.

Yeah. Yeah, actually, that would be pretty good. That happened at Black Hat. Everybody that was in the room, it was a, no, it was a sans Windows security, and they were holding it at Mardi Gras. So there were like three people in the presentation, and they were all sitting behind the Klieg light from the video. So when somebody raised their hand, you couldn't see them. Yeah, I'll do that. I'll just stand right in the light.

Do you need me to be in a particular place?

The podium. Yeah.

You know what, I just kind of refer to him. And I might point some stuff. OK. Whatever works for you. I'll try to keep the movement down to a minimum though.

I think you submitted for DerbyCon one year, but I don't think you spoke to some. DAVID MALAN- Which one? DerbyCon. DAVID MALAN- DerbyCon. No, no, actually, you're the second person that mentioned that. I've never submitted for DerbyCon. But you did. Maybe you're a fit with someone else. DAVID MALAN- Yeah, you know those bald guys. They all look alike. The two names that's on a key detective or something. DAVID MALAN- Keydet.

all my swag. I'm told I have to wear

them. Knowledge. Knowledge and confidence. Free knowledge. There you go. When do those come out? What's that? Oh, it's all right. What kind of beer did you guys get?

Stare at people, so please go through. Yeah, you need your code. They were talking to this gentleman over here. And he just goes, what are you selling? It's hilarious. Because you see it on the movies. You never really realize where that stuff comes from. Because it happened to somebody, and they go, oh yeah, I'm a writer. I'll use that. I don't want to criticize anything that's not for them. This isn't nerdy enough. We criticize it more than . Sighted. Like an hour or so? Oh, more than that. Oh, OK. That's on a good day. Chantilly? Oh, yeah. Oh, really?

Yeah, for two years. Oh, okay, gotcha. Some apartments right across from the... Yeah, yeah, that's Centerville. Yeah. Uh-huh. Yeah. So it takes a while to get in, so. Yeah, it does. You know, when they have summit, I'll stay. I'll actually... So, which marketing pays for, so. But it does take a while to get in. Fortunately, those things start, like, lunch at like 1130 or noon and then they kick off the summit at 1 so I can go in after you know traffic dies down and after the HOV is left nearly as bad but my wife and I would drive to Frederick Maryland for dinner on a Friday night rather than driving to DC

yeah Maryland's not bad depending on where you go

Yeah. Well, it's been in the news a lot. True.

Greg Marriott. Hey, Greg. I'm Harlan. Hi, Harlan. Oh, sure. What do you do? I drink coffee. Oh, awesome. I drink beer. I drink coffee, too. Awesome. That's right. That's right.

Siemens.

I'm a little bit too. I have an interest in political affairs and things like that. So I like to decide what

Yeah, yeah. Yeah, oh yeah. Well,

where the idea of a sniffer was to go down in the basement with matrix printers and hook them up until you found out which line they were actually coming in on. Did you ever read A Cuckoo's Egg? Oh yeah, it's a fascinating, fascinating book. Because this was before Sniffers, and this was well before the FBI could set up traps or anything like that. And this guy tracked nothing better to do than track down a 75-cent accounting error. That's what I do. Yeah.

Incident response. The guy simply did not say, I can't.

Basically a hacker, like the Apollo guy. This is all I've got in front of me. with this. Yeah, and he . Matrix printer. Sleep in the . Like that.

Well, and he also set up the first honey net.

Once you understand the title, the book makes sense. that because it's you know good meeting you well I get to leave on a flight but I'll be here probably through lunch all right no that's right I got plenty of time it's not that far I mean really Dave where's everybody going

You're going to do a time hack towards the end? Yeah, I can give you a 10. Yeah, give me a warning of a 10 minute. Yeah. Two hands is 10 and one hands is 5. I think I counted that right. OK, that'll work. That'll work.

Yeah? OK, you do that. You want the mics back? No, you can't. You want me to completely undress and give you the mics back?

Okay, so that's recording. Yeah. Okay.

Give me a sec to get all my Borg stuff. Oh, that's on. OK. All right, great. Thanks, everybody, for showing up today. It's great to have this many people out on a Saturday, especially. And thanks for Josh and Justin for inviting me to come out to this. I'm peripherally familiar with the B-Sides conferences, mostly by sitting staring at the screen wistfully, wishing I was at one. And so it's great to actually be at one. So we'll just go ahead and press on.

So a brief introduction about myself. I'm with Dell SecureWorks. It's kind of an awesome place to work. And I've got a really cool title, Senior InfoSec Researcher. I've been involved with information security for, if my math is correct, 26 years.

I started in 1989. Who was born after 1989? Yeah. OK. So I don't feel too terribly old. So that probably explains why I just had no idea how to use Uber. That's probably the explanation right there. No, no, I'll walk. I've written a couple books. You might recognize the cover. One of the issues I've had with the publisher is that even though the covers have different title, they all have the same color. So I'll take a new book that was just published, and I've got the only copies with me, and I'll have them at a presentation. And somebody goes, no, no, it's OK. I've got that. I was like, how did you get this book? It's like, I went to the bookstore

a year ago and bought it. No. And I tried to explain. You've got to do something about the colors and the color scheme. So yeah, this is a fourth edition of Windows Forensic Analysis. Has anybody seen that? Has anybody looked at that? Has anybody got a crease in the spine at all? Anybody? I know my first wife read one chapter of the first book, and after that, she just doesn't go beyond the dedication page. But that's okay. So what we're going to do is we're going to talk about lateral movement today. Is everybody familiar with lateral movement? Okay. Guys like me, for me, lateral movement is Michael Jackson doing the moonwalk across the stage, right?

What is lateral movement? Anybody?

Okay, excellent. Excellent. East-West traffic in a network. That's a good way to start thinking about this. To give you all a little bit of background, when I originally was contacted by Justin to do a presentation, I was going to do it with no notes. And then I started thinking, you know, wow, the talk I was coming up with was just kind of not the right approach I wanted to take. When I give presentations like this, what I like to do is give you all something that you can use when you leave. So what I'm going to do is give a revamped presentation that I gave once this spring. There's a little bit different information, a

little bit more information in it. But the idea is that in 2012, I was at the DoD Cybercrime Conference in Atlanta, Georgia. And I was just hanging around some of the other attendees. And I heard a couple guys talking. And what they said, what one of the guys said was, there are six presentations here with APT in the title, and not a single one tells us anything. And I started thinking about that. You're at a conference with a bunch of people that investigate these types of instances for the public sector and the private sector, federal government, all that. And you've got all these presentations going on. I sat through three of them. In fact, I

sat through, and I guess I'll just share this because the company no longer exists, the H.P. Gary presentation. And the guy speaking said, yeah, the bad guys move laterally. And somebody raised their hand and said, what does that look like on my network? And the response was, the bad guys move laterally. That was it. So I started to see what it was people wanted. And hopefully, I'll be able to give you some of that in today's presentation. So that's my goal. Like the first day of high school, if you're not here for that, then sorry.

Everybody familiar with the kill chain? Everybody seen this before? I don't have to go into too much detail about this. You don't have to be able to read these blocks right here, do you? OK, so see the yellow one on the end? Everybody just kind of do this. Really, seriously, go like that. Because what's happening is, is this kill chain is tipping this way. OK? I don't get to see a lot of this stuff. Sometimes I can find this. I found this a couple of times. As a forensic analyst, I don't really get to see too much of this. I get to see a lot of this. We have network guys that do this. But

most of what I do is involved with this area. So it kind of like tips the kill chain that way. OK? Does that make sense to everybody? Because once they're in, there's a lot of activity that occurs. You know, after they get in, that's pretty much it. Everything goes down there and the entire thing just kind of tips up on its side. So again, lateral movement is movement between systems within this infrastructure, east, west, north, south, however you want to put it. Now, most of the techniques are via the command line. So why does it matter that I've got this bullet on the slide? Why does it matter? Anybody? A hint to where things will be. As an incident

responder, when I go on site a lot of times, and I'm not trying to point fingers at anybody, but I'm going back to, by now, over 15 years of experience. When I go on site and I want to help somebody, and the first thing I say is open a command prompt, all the hearts in the room stop beating. Nobody blinks. Yeah, I did take, in about 99 or so, I did take the MCSC exams back before they became the adaptive exams. And what I found was it's all based on the GUI. In order to do operation X, like add a user, you have to click here, click here, click here, click here, and click here. The bad guys do stuff via the command line

because it happens underneath that. You don't see any of that. And it's easy. And what does that also mean? Yes, go ahead. It can be scripted. It can be scripted. It's fast. Really? Come on. Oftentimes, really? Seriously? You're being very generous, sir. Thank you very much. Well, more importantly, a lot of the bad guys don't have to bring a whole lot with them. If the tools are already there, why not make use of them? And the fact of the matter is, The vast majority of admins that I've dealt with, and I'm not trying to bust on admins because there's a lot of very, very knowledgeable folks, very dedicated folks out there, but a lot of the admins that I've dealt with, they don't

even know these tools exist. I'll give you a little, you know, a buddy of mine told me to start with a joke today. The stuff that I'm going to share with you is real world experience. And some of you are going to laugh and some of you are going to cry because I know I did, OK? And so basically towards the end of presentation, I put this early in the presentation because we want to keep this in mind. As an incident responder over the past 15 years, one of the things that I really, really look for, I'm really hoping people start employing, is process creation monitoring. Because yeah, people say, well, ask for logs when

you go in. And everybody knows, look for logs, look for logs. I can't tell you the number of times where, I'll give you one great example. And this is not part of the joke package at all. A device was issuing syslog for its logging. Somebody had repurposed the syslog server two years prior. So if you put a sniffer on the network, you'd see all this UDP spewing out of a system and not going anywhere. It was on the network, which the network retained stuff for how long? OK, so by the numbers. Now this is a slide I added recently. And this is pretty much the extent of my PowerPoint capabilities with the picture up on the side there. But who here reads the annual

reports that come out? Anybody? You know what I mean? MTrends, DBIR, they did reach from Verizon, Trustwave has got one. It's kind of like the thing that cool kids do, right? They have an annual report. So in a lot of the reports, and different reports call it different things, like think, for instance, the Mandiant Report calls it median time to detection. So they want to have a bunch of statistical analysis and things like that. There's other reports refer to this particular metric as dwell time. And basically what this means is that when the incident responders go in, they start looking around. So they're called in on a certain date. And while they're doing their analysis and their investigation, they're able to find indicators that tell

them how long the bad guy was there. Does that make sense to everybody? That's kind of what the metric means, right? So there's indicators within the infrastructure somewhere that allow the incident responders to determine how long the bad guys were there. Now, as an experienced incident responder, I know a lot of times you can't get back to the original phishing email. You can get to the earliest known indicator, OK? I've seen things that are two years old. I've seen things that are four years old. I've seen things that are six years old. very, very strong indicators of somebody having been in at some point. So what I thought was kind of interesting when we're looking at some of this stuff is particularly the mTrends report, when we're looking

at the percent of external notification, the percent of their clients that were identified or were notified of the incident by external notification, 69%. That means somebody outside the infrastructure, FBI, notified the organization of

that they had an incident. They didn't know it themselves. They had nothing going on in the infrastructure that told them, hey, this happened. What I think was really interesting about it was the median days to detection was 205, with the longest persistence being over eight years. OK, so I'm not sharing this with you to say somebody was in the infrastructure for eight years. I'm sharing this with you because what this tells me And it should be telling you all is that these investigators are finding things. They're finding, what's the term we use? Artifacts or? Indicators. Indicators in the infrastructure. There's stuff laying there that they see. Chris Pogue referred to this, he's now with Nuix, but he used to be with

Trustwave, referred to this as expert eyes. People look at things differently based on who they are. Most of us when we go walking in the woods, we see flowers and stuff like that. You get somebody that's military, they see things differently. You get somebody that's a military sniper, and you don't see them at all, sorry. So actually it makes a very good point. Having been in the military, I can't go to like a wedding or a function or anything like that without feeling completely out of place because everything's just not exactly the way it's supposed to be. Maybe it's not military, it's OCD. But what this tells us is that there are artifacts in the

infrastructure that those who manage the infrastructure, if they knew about them, could actually find them. Because the fact of the matter is, you've got guys like me that show up, and there's no logs. OK, so if I don't have logs to look at, then the admins aren't going to be able to look at logs either, right? So there's no logs. So I'm looking at the endpoint. And there's stuff on the endpoint that I can easily find. I'm not talking like in a month of analysis, I'm talking within four hours or less. There's things on the infrastructure, within the infrastructure that are sitting there that tell us these things. Okay, and we're gonna talk about some of those in the presentation today. So that's just about enough

for the warmup. So some of the first steps. Common aspects of lateral movement. We have two systems. This is something very, very important to keep in mind. One of the things that I found about the incident response field and digital forensics field is they lack a specificity of language. So there will be a lot of times when somebody will say something, and you'll say, OK, so which system was that? The source or destination system. And you get, uh, is your response. OK, so with lateral movement, there's always a source and always a destination. Can we agree on that? Anybody disagree? Raise your hand. OK, all right. Well, we'll go on then. I was ready for a fight. So how does the adversary access the source system

itself? There's a couple different ways. They can use web shells. Is everybody familiar with what a web shell is? Do I go into any detail? Anybody not know what a web shell is? Put it that way. OK, because I've got some good pictures of a web shell. Actually, the screen's telling me it's on the next slide. We have rats that get on the systems via fish, Strategic web compromise, now those are always interesting. Has anybody investigated a strategic web compromise? Those are pretty fascinating, yeah. It's pretty interesting. Remote access via VPN, via RDP. Believe it or not, this happens quite often. I've seen this a number of times, a lot. And I see some folks in the back with yellow shirts, little minions back there laughing. Have you guys

seen this too? Banana? What?

All the things, yes, yes. We've seen all the things. Yeah, OK, good, good. So mapping shares, I've seen that quite a bit. Scheduled tasks, this is big. And actually, believe it or not, if they have access via RDP using a browser to collect information. Quick story before we move on to the next slide. I was doing some incident response for a client. And they had a digital forensics and e-discovery department, which was disguised with an army of one. And pretty much what he did, he was very proud of this. And it actually, it helped that a lot. Does anybody here have kids? Anybody? OK. My kids are your age. Anyway, they put Spectre Pro on a couple of the systems that they knew the bad

guys had been accessing. Everybody familiar with Spectre Pro? It pretty much takes like a movie of the desktop, right? So they showed us a movie of the desktop. We actually got to see the bad guy for like 15 minutes using a browser to access their intranet. And they're like, well, they're showing us this movie. And he's like, well, what are they looking for? And I was just like, hold on. There it is. Because we saw everything. That was pretty interesting. Some of the stuff you can see if you just have the information available. So there you go. That's a web shell right there. So web shells are used to gain access to the infrastructure. Usually

the way that we see these used is there's a web server. could be a IIS, could be Apache, whatever the case may be, sitting on the perimeter. And it's got a vulnerability. Somebody pops a box, put a web shell on it, and many times what we'll see is they'll put a very basic web shell on to begin with just to get access, and then they'll use that to step up their access, put a more advanced web shell on it. Once they get that on, they get a greater level of control of the system, they'll start pushing other things in. You know, it depends on what kind of access they have available. For instance, if they've

got, if there's a SQL server and they can do SQL injection, that's always a great way to get stuff on the system to get in the infrastructure. But that's kind of a door, okay? And what I've seen more often than not is the boxes that are popped are the boxes that nobody knew were running a web server. In fact, in some cases, nobody could identify the system itself. Like, whose is this? So those things are sitting out there. They're accessible. Some of the things I've seen on a Windows server running Apache and WordPress, a GIF image that had special content added to it. And this is actually what was in it. So they would request the GIF image. And I would like to put log entries up here, but

they turned off logging on this web server. They had the error logs. But the interesting thing is in 15 years of doing incident response, one thing I found is that when the bad guys steal credentials and they use those credentials, they don't make mistakes. Your normal user will miss their password once or twice. That's kind of interesting. The bad guys don't make those mistakes. Can everybody see this? Is the podium in the way? IIS server with an ASPX page on the system. Very simple one-liner was just put in there. You can actually see this in the logs. It gets loaded on the system. Pretty effective. I took the bad words out because I didn't know who was going to be attending today. But sometimes they put easily identifiable

stuff in there. Really interesting thing about this stuff, is everybody familiar with the find string command on the Windows system? Kind of a grep. What's that? Everybody? Nobody? OK. Just really quick, if you just kick off from the root of the drive where all the web pages are and look for the word unsafe, you never know what you'll find. Okay, any questions about that? Usually these things go pretty much undetected. And this is a lot of times how, when they get into the infrastructure, this is a lot of times why they're at a command line level accessing the infrastructure. But the interesting thing is, and I've said this many times before, and I'll keep saying it for years to come, is these

guys, these people getting in, are smart enough and dedicated enough that they don't need any additional access to map your infrastructure. They know where your SQL servers are, they know where your Exchange servers are, and when I sit down with the CIO and ask where the critical assets are, oftentimes what we have to do is we have to go follow the bad guy's activity. Because the people that own the information often don't know. And I say often because many times they do, but often they don't know where the critical assets are. If it's in somebody's email, these guys will pull PSTs off of systems. They know where this stuff is, and they can map your

entire infrastructure just using the command line, so they don't need any more access beyond this. OK, you like my graphics? This is not a sales pitch over here. I just pulled the image off the web. It said Dell on it. I don't know about the zigzag. Does anybody know what those things in the middle are? Somebody told me it was a floppy drive. It was kind of a floppy drive. OK, so this was an interesting example of a web shell that I investigated. It was pretty fascinating. This loop arrow right here is intended to mean loopback. Basically what they did is they got on a system that had a web server and they wanted to move to a SQL server. So the

system was a web server, so they put a web shell on it. And they had RDP access to the system. So they opened up Internet Explorer and went to local host to load the web shell. Does that make sense? No, it doesn't at all. OK. And then they used the web shell to issue SQL injection commands using XP underscore command shell stored procedure to access the SQL server. OK. And we were able to put this all together with a lot of the indicators that we had on the system. Can you guess what some of those might be? Remember, they had RDP access. So they were basically on the desktop. Anybody have any thoughts on what we found, what we looked at? No. No.

Unfortunately, that would be my first guess too, but in this case, no. Browser history. And there's a place in the registry called, a key called, and the user's hive called, what? Typed URLs. We could see where they're accessing local host. So we did have logs, not event logs, but we did have logs from the web server on this system We also had, well, there wasn't a whole lot of logs on this system. But the most interesting aspect we had, so I kind of gave it away right here, access the web shell with local host, a.aspx. So even though they deleted the web shell itself, we had all these indicators left behind. So there was another file system indicator on IIS servers with ASPX web shells. The first time you

access it, the .NET framework creates a page called the name of the web shell, .compiled. It actually compiles it. So the bad guys can delete all that stuff, all their web shells. And I've actually seen bad guys come in, put a web shell on, delete it after use, come back a month later, put another web shell on, delete it after use, and keep doing this over and over again because we found these file system artifacts. OK, but interestingly enough, in this particular case, this bad guy, he had the IE open. He opened up, god, it had to be like 25 tabs in IE. And how did we know that? How did we know the tabs that were open? Say again? No, because this

is like after the system's been taken offline and imaged. So we didn't have processes. Good guess, though. Good guess. It's a setting, but not the actual indicator. Is anybody familiar with session restore capability in a browser? OK. He crashed the browser. When you crash the browser, there's these session restore files that are created, if you understand browser forensics. And it varies depending on the type of browser you're looking at. But for IE, it's in a particular place. And it's an OLE file. Does everybody remember OLE from the old version of Windows back when, well, back a long time ago? So these files were created. But the bad guy never came back and reinstantiated IE to get rid of those files. Because

that's all you have to do is reopen IE. And it says, do you want to load your previous session? And if you say yes, it loads it and then gets rid of the files. So he left them there. So we parsed through these session restore files. We found the commands he issued, which is how we knew this, how he created that he was creating a user. Now, why he did it this way, I've got no idea. It was very odd that they did it this way. But the fact of the matter is, when you take a step back and you look at the responses you guys had, That was my first approach as well. Oh,

he's doing this. I gotta look here and look here and look here. Well, no. They did something different that required us to look someplace else, but the indicators were still on the system. So we had those session restore files to look at and parse apart. We saw the exact commands, and we also had the username and password he was using to access the SQL server, which he'd gotten from the original web server he got into because it was in The username and password were actually in a config file. So any questions about that? Usually the biggest question when I finish with this slide is people go, well, why did they do that? I don't know. Maybe they had drink tickets and they started

early. I don't know. It could be it. I don't know. OK, scheduled tasks. Is everybody familiar with scheduled tasks? Anybody familiar with that? Who here uses at.exe to create scheduled tasks in their infrastructure? Okay, we got a couple people. Okay? The reason I ask is, I'm not gonna put a spotlight on anybody, but I did have a client that I was working with, and the suggestion that we had was, it had to do with AT.exe. And we made the suggestion to them of what to do, and they said, We're sorry, the suggestion was remediation. You need to go use the at.exe command and use the slash del switch to get rid of the scheduled tasks. And the reason we're doing it, the scheduled task is already run, but

if you use the delete command, it'll actually get rid of the indicators. It'll actually remove things like registry keys and things like that. Now it won't, it actually adds an event log entry, but it'll remove files and it will remove registry keys. So we said, just go ahead and get rid of these indicators and just run this command. And the response we got back three days later was, we can't run the command because we didn't install it. It was on the installation CD. The bad guys have used it. So this was another example of folks who really didn't understand their infrastructure, didn't understand their assets, their inventories, or their endpoints. And we had just given

them a history of what the bad guy was doing. And they said, well, we can't do anything. Our hands are tied because we didn't install this application. Does that make sense? No, it doesn't at all, does it? I'm glad you guys are awake enough to go, no, that doesn't make sense at all. So we got the scheduled task. You can see I stole the icon from clipart. So it requires admin privileges. AT.exe is most commonly used. And one of the things we use for classifications of threat groups is if they're using sched tasks instead. Now, the interesting thing is an incident responder, and not a lot of folks know this is that if you're doing

incident response and you're using AT.exe to collect a list of the scheduled tasks on a system during live response, you know, collecting that volatile data, the tasks that were created using AT.exe will not show up if you use sched tasks. OK? You can try that on your own system. Create one job using each one and then run the command to see the list of jobs. You have to run both commands to get the complete list. And that's something we run into every now and then. And be executed, need to be copied over first if they're not native. What that means is there's something that has to occur before at.exe is used. You have to copy those files over that you want to execute, which means there's

going to be other artifacts left. There's other indicators, which this is different from the use of psexec, because psexec, if you've ever used it, will copy those files over for you. OK, does that make sense? Maybe you guys are just, maybe you guys know all this stuff. Anybody want to come up here and give this presentation for me? Okay, scheduled task indicators, part one. Source system, again, like I said, there's a source system, there's a destination system. So the indicators of the activity on the source system are gonna be pretty much, if you're lucky, it's gonna be a workstation and you're gonna get a prefetch file. Why does it matter if it's a workstation versus a server? What's that? Exactly.

By default, Windows servers are not configured. They can, but they're not configured to do application prefetching. It's a registry setting. That's pretty much the answer to everything, isn't it? It's a registry setting. Yeah. OK. So the key to detection is process creation monitoring. There are several tools out there. I'll have a slide later. I'm not trying to do a sales pitch. We have a tool that we use that does process creation monitoring. And watching this stuff and lining up the process is absolutely fascinating because you not only get to see the command being run, you get to see all the other commands being run. Net time to check the time of the remote system. The bad guy checking to see if the task is completed. Reissuing the

task. The entire, like if they RAR data up, you get their passwords. It's all right there in the command line. It's pretty cool stuff. Okay, so I'll push the process creation monitoring over and over again. One thing, you can install Carbon Black. I've talked to some folks that have gone down.

entries are in the system event log just look for those some persistence any questions any thoughts about any of this this is all familiar to you right you've seen this before you do this all the time really so the sugar rush from the donuts is kind of crashing now okay

some key points for detection Okay, what you want to do is you want to look for clusters of indicators, not individual indicators. The problem that you have with individual indicators, and this is a problem I have with a lot of the threat intelligence that's out there now, is that most of threat intelligence looks for individual indicators. What you want to look for is TTPs or clusters of indicators. Because there's a lot of things that go on within an infrastructure that if you look at them in isolation of everything else, they could be threat actor activity. a lot of this stuff a lot of the stuff we see the threat actors doing is stuff that a

normal admin might do okay so um you want to look at the clusters of activity you want to look at if a process is created that seems suspicious when was it created what was the time yeah was it when somebody was at work or was it off hours You want to look at things like in the command line passwords that were used. You want to look for all these different clusters of indicators. So you want to go to different sources. You want to look at your event log. You want to look at your registry. You want to look at your file system. The list kind of goes on depending on what you're looking at. It's even more detailed if they have RDP

access to your infrastructure. OK, does anybody have any thoughts or any questions or comments about indicators? Yes, sir. Well, it's related to indicators.

It could be either one. What makes a difference from an attorney's perspective? Well, I don't normally deal with attorneys. Because the fact of the matter is, most of the organizations I work with, we go in and of course the first thing they want, the first thing, this is kind of a segue, but I'll go ahead and take it anyway. A lot of the folks I work with will say, I go in and I say, OK, sir, I'm here to help. What is it that you want? And what is the answer? No, it's even... No, it goes beyond that. The first answer I usually get from the C-level executives is, I want to know who did this. I want their head on a pike

out in front of the lunchroom so everybody can see it. That's what they want. So when you sit there and you say, so you want to prosecute and therefore make this part of the public record, they look at their attorney and the attorney's going like that, you know? So then you start to get to questions like, am I compromised? What did I lose? How much did I lose? That's when those questions start to come up. The emotion comes in first. The vast majority, I haven't done PCI exams for a long time, and I won't admit ever having done that outside this, oh crap, it's been recorded, hasn't it? Anyway, so I don't really deal, work in the area with attorneys. I'm there to help collect information, get answers, things

like that. And when I give my answers to my clients, I have data to back it up. It's not a swag. It's not a guess. I have hard data to back it up. So if we do deal with forensic images, that's great. There's stuff that others can go back and look at. But the other thing I have to deal with is the fact that if my client had the capability to do this, he wouldn't be asking me. My goal is to get the answers as quickly as possible so that we can start addressing things and move people beyond that emotional reaction, that panic reaction, into, OK, let's, we've got to get people beyond the, OK, I'm going to lose my job. OK?

I've been asked before, why don't you show up with a little jacket that says FBI on the back? And the answer is because you don't want people to know that I'm here. I want to blend in with your staff so I look like folks that are here, because you don't want people to know happening what's going on and it's not just from a public perception it's the i.t staff as well because if i'm there a lot of times i'm working not just with the cio i'm battling the i.t staff because they think that i'm there to take their job okay so my goal is to to do knowledge transfer to help the company out get

them back on their feet get them past that emotional and panic point and get them to where they can eat start answering the questions and and then when they can sit down and take stock of what's actually happened I've seen in the time that I've done incident response there's been a couple times where I've actually been able to recover the raw archives that were taken and we had the passwords because we had the process creation monitoring in place we got the password some of the stuff that I've seen taken I mean the actual most of 90% of the time that you'll see it in the papers it's a guess okay I've actually seen it this what's been taken it's pretty scary People put 15, 20 years into

developing something and it's gone. Somebody else has it. So to answer your question, I don't work with attorneys that much. I have worked with law enforcement on the side. I find stuff. I show them how I found it. They get the pleasure of dealing with attorneys. Any other thoughts, comments? Yes, sir.

Oh, yeah. Who doesn't? What? Can you talk about that for a second? DAVID MALANI-

What about it? I think you just did. All right. DAVID MALANI- No, hold on. So what's the question? What kind of . DAVID MALANI- There is the actual registry key being set. And if you have process creation monitoring in effect, like for instance, a great example is sticky keys. Does everybody know what sticky keys is? Yeah, there's a lot. So just a quick segue. One of the guys I work with found a tool that StickyKeys was pointed to as its debugger. And so instead of pointing to cmd.exe, it was a completely different binary. And he pulled the string no-no out of the binary. So he called the attack StickyNoNo. And I was like, please make this his nickname. And I really pushed it, but apparently it didn't stick.

But is everybody familiar with sticky keys, what the attack is? So basically the idea is there's a key in the registry called image file execution options with spaces between all the words. And the idea is that Microsoft left this in place so that you could add, well, you or anybody else could add debugging capability to binaries. So you can go in, for instance, and say anytime Notepad is launched, launch Word instead, or WordPad. So whoever tries to launch Notepad, boom, it'll launch WordPad instead. That's interesting capability. What the bad guys will do sometimes is they have remote access to the system. This plays right into the whole lateral movement thing, so we're just not off

topic here. You actually remotely connect to the system. You modify the key. It's pretty easy to do. There's a command line utility in all Windows systems called reg.exe that makes this really super easy to do. And what you do is you take the

There's a couple of accessibility tools, sethc.exe and utilman.exe. You can do either one. And you go in and you create a subkey, and you point the debugger value for that subkey to cmd.exe. And then the bad guy leaves. So if you change all the passwords in your infrastructure, and the bad guy can still access the infrastructure, all he has to do is RDP to that system. When it gets the screen up, instead of putting in credentials, He just hits the shift key five times. He gets a system level command prompt. Yeah, exactly. What? So you change your passwords. You think you've gotten rid of them. He's got a rat. He gets back in. And he's able to get access all over again because

he gets that system level command prompt. Remember, I said from the beginning, these guys use command line tools. They can do anything. Create users, change passwords, dump passwords. And what I don't see a lot of is actually making a lot of esoteric registry setting changes and things like that, other than sticky keys and a couple of things. So there's other things that you can do with this particular attack. I've seen the penetration testing guys will have a variation that they use. But basically, the whole idea of this particular key is whatever you point to as the debugger gets launched instead of that application. If you're using it with the sticky keys approach, because of how it's launched, that command prompt is now running at system level privileges.

So I'll let you guys look at that slide afterwards. OK. One of the last ones I've got is what I refer to as a sleeper agent. I actually did a blog post for the SecureWorks research blog. This is pretty interesting. The bad guy was already in and had moved laterally throughout the infrastructure using AT.exe to push his rat out to all these different systems. I think at one point we found on one day there was like nine different systems. This is like the month of April, beginning of April. And then we saw something interesting. Now follow me here. Two days later, he pushes out a copy of his RAT installer to another system, but he doesn't do it with AT. He mapped to the system, mapped a drive,

and copied it. So he did net use, mapped to the drive, and then he copied it. And where he copied it to was the setup folder for an admin user. So the admin profile had already been created. The admin had already logged in at one point sometime. So he reached out to this particular user profile,

and copied the installer into the startup folder. Now at that point, what did we have? What's that? Well, we had a file sitting there. That's all we had, because it's the installer just sitting there. But it's an indicator. What happened was during the incident response, and this is one of those things that's so simple, it's pure genius. During the incident response, this client had said, no, we don't use communal admin accounts. Every one of us has our own user account, and we have our own by name administrator account. Well, guess what? They used a communal admin account.

And they reached out to this system and logged in using this particular admin account. At that point, because they logged in and the installer was sitting up in the startup folder without any interaction from that user other than logging in the rat was installed it was sitting there for over eight months doing nothing until inaction occurred that and I got to tell you having been in the military when somebody comes up to me and says we never do this it would they're that emphatic you know they just did it

and they're gonna do it again in five minutes so So, and we had the evidence. I mean, we had the clear evidence. So we knew what was going on. So the point wasn't to like hammer them and say, okay, you use an admin account. We did put that in our recommendation, which was don't use communal admin accounts. But these guys put that installer. It sat out there for eight months. It had a different configuration. It had a different C2, which meant that all the MD5 hashes that people were using up until that point didn't detect this. Because you're all familiar with the Pyramid of Pain? You know where hashes sit on the Pyramid of Pain, right, on the very bottom? OK. So they were scanning the

infrastructure with these hashes. They weren't finding it. Because even though the file was a similar name and it did the same thing, it had a different C2. So computationally and mathematically, it was a different file. So the computer goes, nope, that's not it, and moves on. It sat there until somebody logged in, launched it, and then we started to see the indicators. When we went back and we looked at that system, the first thing we had to do was say, well, when was our endpoint agent installed? Oh, it was installed this day, and that's when we saw it. What happened was, when the admin logged into that system to install our endpoint agent, that login

caused the rat to be installed. So when our endpoint agent got on it and scanned it, it said, whoa, dude, there's a rat on this system. Like, OK, well, when did it get there? Why did we miss it before? Why didn't we catch any of the stuff in the network logs? It's because until that point, it was dormant. So imagine that. This is how forward thinking these guys were. They're going to leave something. a little special treat, and it sat there for eight months. And the reason they left it there, the best I can come up with is, we found all their other stuff. They are familiar with our incident response techniques and capabilities. They see all of us saying MD5 hashes are the greatest thing in the

world, and they took advantage of it. Is anybody scared at this point? Raise your hand if you're scared, because I'm scared. That stuff is. That was pretty fascinating.

You familiar with the movie? You know what that's from? Game over, man. You game over, man. Bill Paxton, the greatest B actor ever. What's that? Bruce Campbell. Well, Bruce Campbell's kind of a different genre, you know? But, you know, Bill Paxton, opening scenes to Terminator, right? When Schwarzenegger was walking naked down the street, Bill Paxton was there. And aliens, this is great because the cable system around my neighborhood, it's like the entire week I would turn it on and go through the movie channels. If I was getting ready to work out, I'd see alien aliens and alien 3 and then alien resurrection all throughout the day. So this is pretty good because you know what? For all intents and purposes,

it's pretty much game over. They're in your infrastructure. Is this correct? Is it game over? Is it?

What? No, it's not. Yeah, no comment. I'm going to wait to see what that guy says. No, it's not. We need to change the definition of success. What is success for the bad guy? What's that? Exfiltration. If you can detect them prior to exfiltration, and you can cut them off, even though they're inside your network, Do we win? Yeah. If they can't get what they want out, let me ask you something. The Great Wall of China, massive feat of engineering. What was the purpose? Keep up Mongols. What? Keep up Mongols. No. Keep people in. Because when you come in and loot a country like China, and you're trying to cart all that loot away, how are

you going to get it over the wall? Seriously. Go ahead. Come on in. Does prevention work? No. Prevention doesn't work. What do you need? We need detection and response, right? So detection is key. If we get that endpoint agent out there and monitor process creation on the endpoints, yeah, it's going to be a big database. But the fact of the matter is there's a lot of other uses for the data besides just security and instant response. It's a lot of very valuable data. And more importantly, once you start looking in those areas and you start looking at clusters of indicators, this stuff is easy to pick out.

It's like going to the supermarket, and there's a whole thing of apples, and there's a tomato in the middle. Yeah, it's a fruit, right? You eat it. You kind of put it in salad. People put apples in salads, right? It's very similar, but the fact of the matter, it's different enough that you can pick it out. The same thing with the bad guy's activity. Does anybody have any thoughts, questions? Am I completely wrong? Because I was in a cab this morning. I just made this presentation up on the way over. Any comments? I was going to leave like 20 minutes for questions. How are we doing on time, by the way? 15 minutes. 15 minutes. I left a lot of time. Well,

OK. So what are you doing for tablets? For tablets. Nothing right now, because they come and go. They're in the infrastructure. They're not. They're physically in the infrastructure, but they're not connected to the network. There are solutions out there that you can use. I don't know that Dell SecureWorks is doing anything specific for tablets. There's a gentleman down here you can ask. Don't want to put the spotlight on you. You said your action doesn't hurt, right? I did. Hold on. Let me get this slide up, because it's really cool. You like that?

I like that one a lot. Anybody know who that is? Kevin Smith. Right.

In the what instance? It depends on what you're calling prevention. What are you calling prevention at that point? So if you're going to do whitelisting, then obviously it wouldn't be a whitelisting software. So I guess it has a piece. If you wanted to explain that, I would have . DAVID MALAN- Wow. Yeah, sure, I guess. What's that? Well, yeah, you can change it. But if you whitelist applications, anything that's not on that whitelist isn't going to run. But it's not. It sort of goes back to the detection as well, because if you whitelist applications and you get an alert that something ran, yeah, it didn't run. But with moving the rat over, it's just an MSI. It's just sitting there. It's not trying to execute.

And so it's just sitting there. So I can see your point, if you have that capability. I think the thing is, is I don't ever see anybody with the capability. Because who's going to maintain the whitelist? It's a natural white list. No? Well, bit 9 has that capability, but that's not what carbon black's for. And then the device card, when it's coming out, which is going to be a lot of ? Yeah, honestly, I don't know. I don't know. Has it built-in app whitelisting? Yeah, but that goes back to my question, though. Who maintains that whitelist? Yeah? Well, you can do a whitelist, you can still inject a process into that. Yeah. You can still do a process injection.

So this is the way I see it. And please correct me if I'm wrong. individual company you know each individual private sector company having its own uh hash list who's going to maintain that you know you set up a help desk and you start to do application white listing what's going to happen to your help desk it's going to explode because even though you even though you send out the messages and you do all the training you say this is the authorized applications you know i'm just like you guys i'm a user i have to you know take the the training When I've got the training on, you know, I'm going through my annual training

that I have to go through and the little thing is talking to me. Wah wah wah wah wah wah. So imagine like in a company like the size of Oracle or size of GE, all of those users sit and doing the same thing and then they want to run an application that's not on the white list. But they're not thinking it's not on the white list. So do you maintain a white list by staffing out your help desk to answer those questions and do the help desk response? Or do you lock it down? It's just a mess. I have not yet met anybody that's done whitelisting. I'm not saying they're not out there. I'm not

saying that folks aren't successful doing it. I have yet to meet somebody because of the type of work that I do that actually has whitelisting or even blacklisting. I have never seen that. So the capabilities are out there, but the question I've got is how do they get maintained? THAT'S A GOOD QUESTION, THOUGH. IT'S ALL PART OF THE TECHNOLOGY THAT'S AVAILABLE TO US, SO WHY NOT USE IT, RIGHT? HERE'S ANOTHER WAY TO LOOK AT IT. WINDOWS EVENT LOGS WITH WINDOWS 7, HOW MANY WINDOWS EVENT LOGS ARE ON WINDOWS 7?

I DID A DIR AT ONE POINT ON A FRESH INSTALL. I HAD 141 OR 142 FILES. NOT ALL OF THEM POPULATED, OBVIOUSLY, BUT A LOT OF FILES. SO WE GOT ALL THESE LOGS OUT THERE. I have yet to see a place that does central correlation of all those logs from each individual workstation. Do you see people doing logging from workstations? I mean, I know people do core servers, critical servers, sometimes network devices. I saw one. Rarely workstations. Yeah. So the question was, for those of you in the back, does anybody do centralized logging on workstations? I saw one that did it. They only centralized the security event logs. And they had a GPO issue in their infrastructure so that all they had, they

had these massive security event logs that all had one event ID in it. That basically, all that, they were literally, the security event logs were full of one log entry that said that the GPO had been changed. There was an issue in the infrastructure. All of this stuff was being centrally logged. So the only time I've seen somebody do central logging from workstations at all, that was the setup. And they weren't aware of it. So going back to the white listing, Okay, who's going to maintain it? Who's going to monitor it? You have the same problem when you said who's going to maintain a whitelist. Who's going to look through your process creation logs? Think

about how many processes you create. Yeah, but that's easy. That's easy. You create watch lists. You have the computer do it for you. And that's where the part, having worked with organizations to set up incident response capability, It really comes down to this basic thing. The reason I gave this presentation today, and by the way, that's so cool. The reason I gave this presentation today is I constantly hear, well, what does that look like? What do I look for in my infrastructure? So if you go back a couple slides, I'll pass that one, to, well, look, here's a whole list. And here's another list. This is stuff to look for. You've got these. These are, you know, print these

slides out. So when you're doing incident response and people are saying, OK, it's great. We've got these logs. What do we look for? The first question I ask is, do you use AT.exe in your infrastructure? Nine times out of 10, it's, huh? What's that? OK, good. That's a no. So pretty much the bad guy is going to be the one using it. OK, because pretty much in most infrastructures, 10 minutes, OK. Most infrastructures, the only scheduled tasks they have are like Google updates and the Adobe software updater, right? And then, yeah, I guess if somebody puts iTunes in, there's the Apple updater. But beyond that, pretty much, I see very few organizations using AT for

anything. Wouldn't that be a prevention method? You just said, no, to that one is your Reddit? Well, it's a prevention, OK. And a great way to do the prevention, and this, you know, I've Go into that image file execution options registry key, create a subkey for at.exe and point it to a batch file that sends you an email. So anytime the bad guy tries to run at.exe, it doesn't run. And every time he tries, it sends you an email that it's being used. Okay, so that's a prevention and a detection technique right there. But if you haven't gotten that far yet, if you haven't gotten to that point, what can you look for in the infrastructure? Well, there's all these places you can go. One event

log file out of 140, and depending on if it's a domain controller, 160, 70 some odd event logs, there's one event log file that you need to go to to look for some of these indicators. And if you're not already including that in your Splunk instance, start doing it. Start collecting that information. OK? So there's a lot of detection stuff that you can do that's pretty easy if you know what to look for. Yes, sir?

Well, it really depends on the kind of lateral movement we're looking for. By default on Windows 7 systems, the event logs are turned on. I understand what you're saying about particular auditing, but the stuff that's included in default as of Windows 7 is almost sufficient. Just the stuff that's running there by default. Looking for lateral movement, my second favorite place is to go back to the first one and really, really wish people installed process creation monitoring of some kind. because that picks up the net use, that picks up the AT, it picks up the DIRs that are run. We've had a lot of bad guys, interestingly enough, leave their batch files sitting around. So when the client says, I want to know what systems were affected, like if

the bad guy pushed out Mimikatz, everybody familiar with Mimikatz? Okay, bad guy pushes out Mimikatz across systems, this guy did it with a batch file. So he used one executable to give me a list of all the systems in the infrastructure, And then he used the output of that tool to feed into a batch file. That batch file took every system name, mapped the drive, pushed out Mimikatz, launched Mimikatz, brought back the results, and then deleted the Mimikatz files. We had his entire script right there, literally and figuratively. So when the client said which systems were infected, I just, here's the bad guy's output. These were the affected systems. not everybody does that not everybody leaves their batch files and output file sitting behind some clean up

behind themselves some more than others in fact in fact with some of the incident response that i've done i was act i was able to see using a timeline which is a completely different presentation using a timeline i was able to see that one threat actor came on at 3am utc and then a completely different person started logging in at 11am utc And when I say completely different, completely different set of activities. First guy comes in, hits a system, first thing he does is open the PowerShell command prompt. We couldn't see what he did with it because we didn't have process creation monitoring, he moved on to other things. The guy that came on at

11 AM UTC had a completely different MO. He was all about the event logs, grabbing security event logs, clearing event logs, doing things like that.

So there's a lot of indicators. Going back to those annual reports, they're telling us something. What they're not telling us is what are those analysts looking at that allow them to determine a median time to detection or a dwell time of 205 days or eight years. What are they looking at? Because there's something there. Is anybody asking the question, how do you know? They're probably taking it on faith. But there's something there that's telling these guys, whether it's Mandiant FireEye, whether it's SecureWorks, whether it's Trustwave, whoever it is, there's something there that's telling them and giving them this information that they can base their statistics on. Okay? So hopefully this presentation will have some of those things in it for you.

Thank you very much, guys.

Green light. Sweet. That's red light.

John Davison. Thank you. Thanks.

All right. Automated detection strategies, custom tools and tactics for detection response containment. That's a long title. I want to start out by hopefully getting a show of hands from people in the crowd. Does anybody here work alerts as a part of their day job or at all during the week? Like you go through and you're clearing alerts. Okay. All right. Good. I've got a few. All right. So I'm going to take you guys back to 2011.

New guy on the job. I got hired to do, it was like a developer slash detect guy. Two weeks on the job, I decide, like there's just a ton of really awesome security tools at this place, and decide, you know what? I'm going to go find APT all by myself, right?

and we have a phrase for that now right we call that hunting I'm gonna go hunting for APT so I start digging and sure enough like almost right away I find something I was like alright so you do what you do when you get that initial thing right you get that thing I'm gonna attack guy I'm gonna take that thing I'm start digging on it I'm gonna look at it and I'll pull out little pieces of this and that I'm like okay I know what that is I'm gonna go over here I'm looking this tool you know and what I'm gonna do is I'm gonna try to build the case that what I'm looking at

is malicious And the more that I did that, the worse that it looked. It was pretty obvious to me that I was looking at something bad, possibly a compromise. So I went to my manager and I told him about it. I said, hey, I think I got something going on here. I told him what I had found. He said, go talk to the Intel team and see what they say about the things that you found. I said, all right. So I went over to the Intel team, showed them a few of the things that I was looking at, and they're like, dude, you found that? That's really bad. And I was like,

So go back to my desk and I'm new right? I'm two weeks into the job So I don't want to like throw this out into the air and be like look what I found and I know it turns out to be nothing like All right fine, so I dig and I dig and I dig and I spend about two hours on this and at one point that when I'm looking at the Data that I'm looking at I realized that the things that I'm looking at had just occurred and I was like oh my hands started shaking to get that real excited now I found something you know and and it was happening right then like

oh my god so I think okay I'm a detect guy I'm responding next step is escalate right yeah escalate the alert get people involved get people looking at it so I was like all right what do I do next all right who is it who's someone's clearly pwned so I go and I look and I dig out at this place where I was working at. They had numbers for IDs. So I pulled the number out and they had a tool that you could put the number in and it would show what that person did for the company. So I was kind of hoping and as a detect guy we have kind of a backwards view

on this. We kind of want it to be something crazy because you want to find that clever crazy hack. I was just kind of hoping it was like a scientist or something that was working on the next cool thing or maybe it was like some high up at the company, like CEO or something. Like two weeks into the job, I find APT on CEO's laptop. Oh man, what is this? So I get in it and I put it into this tool. And it turned out that I was investigating the traffic of myself. Investigate my own traffic. So of course all the things I was looking for, I was finding them. I was like, What's happening right now? Two hours of my life going down the

tube. That's how it goes. I am John. I work for a company called Ashland. We're an international chemical company. We've got about 10,000 people and about 15,000 assets. Ashland's a fun place to work at from a cybersecurity perspective. We kind of run the gamut of what we have to protect. So we've got scientists working on the next cool chemical thingamajiggy. we've got intellectual property, which is a target of APT. We've got a retail side, so we've got point of sale systems that we need to protect. We've got industrial PLC type stuff going on, which will probably be more important in the future. This entire presentation is gonna be about false positives. Probably the most boring thing you could think

of to have as a topic for a security presentation. And what happened was, I got to thinking about false positives, and I came across this idea. And once I had this argument with myself on whether or not my idea was even legitimate, that idea turned into a postulate, I think. And I could use that postulate then to... Oh, okay. Yeah, it's that one. somehow dragging across the clothing. Take it off that. Put it up here. And hopefully that one doesn't pick up like that. All right. Thank you. So I came up with an idea and it turned into a postulate. And based off that postulate, I've come up with a detection strategy. And as a part of that

strategy, come out with a tactic of which is to develop a custom security tool because that's what I do that's what I've always done everywhere I worked at I write new custom security tools so I wanna ask for a show of hands just kind of rhetorical question did you ever have that time as a detect guy and go back to when maybe you first started where you were working that alert a really long time, like maybe even hours, if not like the entire day, and you were so sure that you had the most clever hack you've ever seen in your life detected, only to have like the senior analyst or whatever come over and point out some really obscure artifact at

the very beginning of your analysis and be like, dude, that's a quality.

How'd that make you feel, right? I did that to somebody one time, and I feel really bad about it. So I started out thinking about what do false positives mean to me as an analyst, as a detect guy. And to me, false positives are just like a fact of life. I've got this long list of alerts that I need to go look through and look for the bad guy. And along the way, I'm gonna see a lot of crap that I'm just gonna toss out. And it's just like, that's how we do our job. But then I started out by asking, I tried to ask myself, what do false positives mean to management? that turned really into this question, how are

false positive metrics used to drive behavior where you're working at? So let me explain what I mean by that. Going back to previous job, what we would do is we would look at the false positive metrics always as a graph, and that graph would always be a line chart. And that line chart inevitably would always look something like this. It was always trending up every single time. And what we would do is we would all get into a room this chart up on the projector and we look at the data behind it and we'd say hey look it's trending up it can't keep doing that because eventually there's going to be so many alerts generated their analysts won't be able to keep up and we're going to

miss stuff we don't know our heads agreement and say yes and we go about trying to figure out some way to get that graph to go down and that's all logical and it makes sense right that's the right thing to do something always bug me about that not so much bug me but it felt like this graph was to tell me something else and I can never figure it out and that was that so fast forward until recently I was thinking about this and then something popped into my head and I was like it seems like going back and looking through all the detection all the alerts I've gone through in my lifetime it seems like a lot of them are false positives as a matter

of fact I might even say it seems like maybe even most of them are false positives maybe 99% were false positives? Maybe? I don't know. So, I was thinking about that. What I do when I problem solve is I'll argue with myself, essentially. That's why you see me talking to myself sometimes. So I get to do that on stage in front of you guys now to try to prove my point. I said, okay John, I think that 99% of all the alerts I'm looking at are false positives. That's interesting. And I thought, well, now, That's probably not right because think about all those IDS alerts that you got going on. There's so much stuff going on on the perimeter every single day.

I've got thousands if not tens of thousands, maybe hundreds of thousands depending on how big your business is. You got people running network scans, vulnerability scanning, SQL map through Tor exit nodes, how many WordPress vulnerability attacks. Just all kinds of things going on and I know that that's going to drop that 99% down to something else. There's just a ton of stuff. I said, yeah, that's probably, maybe my idea's crap. Then I said, you know what? If we're using a metric to drive behavior, then we need to make sure that that thing that we're measuring, we have a really good definition of. So then I asked myself a very simple question that I thought I'd have a quick answer to, and I didn't, and that surprised me.

I said, so what is a false positive? That's easy, man. A false positive is an alert. turns out to be crap right it's like yeah that's that's a horrible definition for something what is crap well it's an alert you know it turns out to be nothing what's nothing it's it's an alert that's not a false positive right all right how about this it's an alert that was a malicious it's it was malicious right okay all right let's think about that for a second let's think about some examples So let's say you get an alert and it comes from Snort or Cerakota or something like that and it was based on a regular expression, right? And you look at the signature of what the author

was trying to detect and you see, okay, I understand what he's trying to do here and you look at what you actually detected and the regular expression matched something that the guy obviously did not intend to detect, right? That's an easy one. Okay, false positive. Here's one. Somebody just tried to hack into your corporate network with a WordPress vulnerability attack. not running WordPress so I don't care but that was an attack right whether or not you meant to or not that was something that you were looking for and you found it there it is it's an attack it's a malicious attack if you have been running WordPress you'd have a problem right but I don't

care right so that's that paradox and that's why I can't answer the question what is a false positive I don't really know so I thought you know what okay I got it I got a definition for it as a detection guy I think every time I look at an alert every time an assessing alert I am always going to come to one of two conclusions every single time I'm either going to respond to this alert or I'm not right you guys agree with that all right so that's interesting I mean they're going to respond or I'm not gonna respond. Doesn't matter if it's malicious. As a tech guy, you come in and you attack the

network with a WordPress vulnerability attack and I'm not running WordPress, I really don't care. If I were an intel guy, I might care, right? I might put that source IP address in a list and track it or publish it or sell it. But

as a tech guy, I really don't care. I'm here to respond and if I'm not gonna respond, move on. So now I have a definition of what a false positive is. A false positive is an alert that I'm not going to respond to. And if you take that definition of what a false positive is and apply it to that idea I had, all of a sudden it makes sense. Because if I think back and I look at it, and I've looked at a lot of alerts, how many of those did I actually respond to? I bet it's 1%, if not less than that. Because I've looked at a lot of alerts. So with that definition

now, I think if I understand what a postulate is, I'd never even heard of the word before and I was like, hey, that makes sense. A postulate, I think I have one and that's this, that 99% of all the alerts are false positive. Now what I'm gonna do, because I'm old school, I'm gonna tape this up here because I'm gonna keep referring to it because I'm gonna make all these observations based on assuming that this is true. So I'm going to keep pointing at this. 99% of all alerts are false positive. What does that mean? If that's true, then I already know ahead of time, before I go to look at those alerts, that 99% of them are going to be false positive. I already

know that. When I'm looking at a alert, I know that I have a 99% probability that what I'm looking at, I'm probably not going to respond to. That's interesting. realized that and I accepted this as a postulate I started changing my thinking about how I'm detecting so what does it really mean does it mean that 99% of my intelligence sucks does it mean that 99% of my signatures are horrible does it mean at 99% of the time I'm actually wasting my time or more I go back to that graph I was talking about I was going up does the metric even matter like does a number of false positive even matter that

time I've gone through this slide or through this pitch or presentation I've had trouble transitioning from this thought to the next so bear with me on this does it matter how many false positives there are yes it absolutely does as a matter of fact it ends up being the most important thing in the strategy that I came up to do detection and consider this if you if 99% of all the alerts you're going to look at are going to be false positive if you were to chart your graph of all the alerts going to be 99% similar to your graph of the false positive alerts. So why is that important? That's important because only 99% of the alerts you're looking at are false positive. There is

actually 1% that is not. So there actually is something to find and we all know that. What you're looking for has to be in the form of an alert, right? Because that's what we do. We review alerts. If we're going to find something, it's going to have to be an alert that that we review. And that alert is gonna need to be in that list of alerts that you're going to review. And in order for that alert to even exist, the chance of that alert even happening depends entirely on your coverage of the attack surface. So follow my logic here, let me explain this one a little bit. I'm sure most of you guys already understand this. As the attacker comes in, he's interacting with

your systems, just like what Harlan just talked about. And as he touches all those systems, those are touch points, those are the attack surface Possibilities that you have if you had a tool to detect on that point to detect the attack Right so as you're going through like and there's a hundred points to detect on if you're only detecting on one of those points your chances of actually Detecting this attack are less than if you had maybe 50 of them covered which would be less than maybe if you had a hundred of them covered I think that logic makes sense the more coverage you have of the possible attack service, the better chance you have

of attacking that sophisticated cyber attack. And it turns out that the wider coverage that you have is going to turn into more alerts, of which 99% are false positives. We already know that now. So now, go back to that graph I was talking about, that graph that was going up. Based on what I just said, I'm thinking, I need that to go up. shouldn't go up as a matter of fact that should skyrocket if I want to detect the attack but we know that's not exactly true because my original logic is still sound if there's too many alerts there are analysts can't keep up with it and we need to tune it back down so I want to talk about this concept of hunting and tuning

and it's not new it's what we call it at Ashland and it's where

Hunt is where it's kind of like signature development I guess it's where you're gonna go and you're gonna look somewhere for that bad thing like I was talking about it's hunting for APT I'm gonna go look you know say I'm gonna go look under underneath this thing and I'm gonna come back and say hey I didn't find anything I say all right you're a computer so keep looking there all the time forever okay so you look and you look and you look and then you find something and hey I found this roll of tape all right care about rolls of tape. Tuning does not mean rolls of tape are stupid, that hunt is stupid, forget about it. Tuning means, hey, keep looking there and if

you find a roll of tape, don't tell me about it. Tell me about something else. That's what I mean by tuning. I'm sure all you guys are probably already on board with that. Your detection team, if it is a team, maybe it's just one person, can handle number of alerts in a given day or week or whatever, whatever time frame you want to talk about. I want to do a visual here with my hands like say that's one alert, two alerts, like X alerts, right? This is how many alerts your team can handle during a given day. If the number of alerts that get generated by your tools is larger than what your team can

handle, then you absolutely do. You need to tune it down, right? And get that below what their threshold is or at. If it becomes less then, and this is I think what people don't do, if it becomes less, Then you need to hunt and introduce more. Because remember, 99% of the alerts are false positives. It's okay if you're introducing false positives as long as you're hunting and tuning. This is important because every time you do this, when you grow it and then you shrink it back down, what you're doing is you're introducing more coverage. I'm going to go in here and I'm looking and I've tuned it. I've got the tape out. I've got the water bottle out. something pops up in here I'm gonna know about it

and now I've got that under control now I'm gonna go look over here too I've got these two points covered and I'm gonna do that over and over again every time I do I'm gonna introduce more coverage so the metrics of how many false positives starts to matter but in a different way because your graph of false positive changes right it's this it hunt and tune hunt and tune hunt and tune this they study this from here to here this is how much work your analysts do. Go back to where I was talking about, so what do false positives mean to me as an analyst? It's what I do, right? I need things to look through. If I don't have very many things to look through, give me more.

Go hunting. Go find new stuff. Fill it back up. It gets up here, 99% of these things are false positives. Tune it back down. the strategy that we're taking to try to increase our coverage and so as you tune and tune and tune and this is what the problem I was having with this graph this is really what it's trying to tell me this is your coverage of the attack surface as you hunt and you tune your chance when you're starting out your chance is low because you're not covering that much stuff and as you introduce more coverage then your chances of detecting which is what this thing really represents to me increases so

all well and good right 99% are false positive you want to increase your coverage to detect if you increase your coverage you increase your alerts of which 99% are false positives as I was doing this then I found another metric that I never even really thought of before and then the most important point this is the one that I really want to get through to everyone that's here the metric is not how many false positives are there the metric is if you already know that ninety nine percent of the alerts you're looking at a false positive then when you're looking at that alert how long did it take you to realize that what you're looking

at is a false positive why is that important why is that metric important I think it's important because detection in my opinion is really hard you might you might disagree you might think man I go through all these alerts really fast and you know the text is easy in my opinion detection is super freakin difficult and I was talking about this like this is how many alerts that your team can handle and I think that really this is probably like gonna be like this and I don't mean that to be insulting I mean because detection is hard how many times did you you know you get all these alerts and like I've seen this I've

seen this I've seen this fp fp fp you know you wear out your f8 key doesn't even say f8 anymore in here in here in but then you come to that one alert and you're like I have absolutely no idea what the heck I'm looking at It's a DNS request, the subdomain looks like it's base64 encoded, I decode it, it's just a binary blob, XR it, I don't know. And you spend maybe hours digging on this one alert when you've got all these other things to look through. In my opinion, it's pretty hard. This metric is the one to drive, right? And what I mean by that is this is what I need to improve. If I want to improve the detection capabilities of

Ashland, what I need to do is improve this metric. I need to take the amount of time it takes you to realize that you're wasting your time and bring it down to nothing because the smaller you can get that, the bigger you can get that. And the bigger you can get that, then the more alerts you can handle, the more alerts, the more false positives you can handle, coverage you can introduce. The more coverage you introduce the better chances you have of actually alerting in the first place. So what if I had a magical tool that somehow for every alert it gave the analyst enough information that he or she could quickly determine that it was a false positive. How would that affect things?

Like I just said you know that would ultimately increase your chances of detecting. So, end half of presentation. That's the strategy, right? This is my thought process. This is why I'm trying to do what I'm doing. Three key things. 99% of the alerts I'm not going to respond to. You've got to increase your coverage to get the attack surface. And the metric to drive is how long does it take you to realize you're wasting your time? Because there's a 99% chance that you probably are when you're reviewing that alert. So let's talk about alert triage. We're getting into more of the tactical part of it, right? How do we do alert triage as human beings? And then at the

end of this slide, I come to a conclusion that I knew all along, but I didn't realize that that's what it was. You get that initial alert, and you analyze it, and you're looking at it. And in that alert, you're observing things that you identify. Like you remember it, or you know it, and you're like, oh, that's an IP address, right? You know in your mind, you know what an IP address is, right? identify these things and then you take these things and you research these things and you build the case only now I'm not building the case to say that it's malicious I'm building the case to say that it's false positive up until

this point every time I sit down I'm looking at an alert I'm trying to figure out is this malicious is this malicious I've changed that way of thinking now that I've thought about all this because the much higher probability is that it's not going to be something I'm going to care about so that's the case that I'm trying to build Possibly as I'm doing this, I take that thing and I take it and I research it and then the output of the analysis of the research, I'm probably gonna find more things to dig on, right? Ad infinitum. And inevitably you're gonna end up falling down rabbit holes. We all do, where time just disappears. In the

end, you get, it's hard to describe, you get these impressions You're looking at these things, and you see these things, and you see it, and it imparts some impression on you, like a feeling. And the reason I bring that up is because I'll say detection is an art, right? And no, it actually really is. I think a lot of people use that to get out of doing something somebody asked them to do. But it really is because your assessment of an alert is your gut feeling about it. At the very end, when you've looked at the alert, and you've come to your conclusion, there's no math behind it. just feel that this is probably false positives probably nothing right or maybe you're super sure about it

or maybe you're like I don't know right there's no there's no math to it at all and that's a problem to solve it's a difficult problem because it turns out that feelings in computers don't really mix that well there's no way to impart that into Python code I couldn't figure it out or that that's what I'm trying to figure out so

paint the picture of an alert such that when the analyst gets it he can quickly figure out what he's looking at and honestly I don't have an answer to that so yeah this is what I'm trying to do so introduce oh well yeah automation is key forgot about that bullet point so introduce the tool that I'm working on this is we're calling an ACE which stands for analysis correlation engine I stole all this artwork. Any Netrunner players? Yes! Alright, there's three. So this is the custom piece of ice that I'm building for Ashland that's trying to solve the problem and this is what I'm using to drive the metric to get that time to disposition to FP down to zero. So automation is key.

When you're the next three slides I'm going to talk about the tactics of like this is basically how I'm designing this tool so I've already described like why I'm doing it we've talked about how alerts or how alert triage works right and we talked about at the end when you look at an alert it's a feeling so what kind of tool do I write in order to solve this problem so I'm going to say going forward that an alert when it comes in it has what I call observables An observable is something that has a type and it has a value and optionally it has a time at which it was observed. So an example of that would be an IP address, right? That's a type and the

value is 32.52.6.12 whatever and the time would be June 25th 2015 at 12.05.02 UTC something like that. That is an example of an observable. I observed this in this alert. Analysis is then performed observable I'm going to take that IP address and I'm going to do something with it what I do depends on the type right if it's an IP address you know as an analyst when I get an IP address I know you know there's things I can do with this I can go pull PCAP on it I can put it into Splunk's or do a Splunk search on the proxy logs do something like that and the analysis as I do the analysis on the observable that's going to generate more observables

so The things that perform the analysis represent something that was previously done manually. So in doing all this, what I've identified is that when you get an alert, if you're really gonna dig on it, there's a lot of time wasted because you're just like clicking on things on your computer. You're taking something, you're copy-pasting it into this other tool, look at the output. Alright, that's interesting, put this over here, Google it. Alright, that's interesting, back and forth, right? I'm all about automation. I say automate all the things. automate all the things that detect guys would manually do the menial tasks and have them already done so that when they get that alert all that work has already

been done all the analysis has already been done so all the modules that are running represent things that we do manually and the modules know what to do with observables of type X right so I'll walk you through an example of that this I have observable and it's a type IP address and say hey I am the PCAP extraction module guy I know what to do with an IP address I'm going to take that IP address and I'm going to go to all of our sensors that are running that sniff and G and I'm going to run TCP jump with a BPF filter set to this IP address and I'm going to select only the

files modified within like a two minute range and I'm going to pull all that PCAP back and here's an observable of type file and something says oh you're an observable type file I'm the file extraction from the pcap module right I'm gonna take that pcap on a run bro on it I got some code that extracts all these files and hey here's a freaking ton of files and oh you have a file and it's an executable I'm the sandbox analysis module guy I'm gonna go take this I'm gonna run this file in the sandbox and wait and hey look it reached out to us they try to reach out to an IP address oh you have an IP address I'm the that looks at IP addresses that did not

come with the original alert I'm gonna take it I'm gonna run it in Splunk looking for many proxy requests to this IP address and oh hey somebody actually did try to go this IP address and here's their user ID right it just goes on and on and on and all these things that you would do if you felt compelled to do it like if you look at that alert and you're like yeah you're not gonna go through all that trouble doing all that work so I'll just go ahead and do it for you in case there's something to find because there might be the modules can do anything that I want them to do. It's

all just a bunch of Python code so it's gonna be literally like whatever, whatever we can automate. And then recursion with limits for the win. I was talking about observable, I'm gonna analyze it, it's gonna generate more observables, I'm gonna analyze all these things, et cetera, et cetera. Until either I hit a limit or there's nothing else to observe. Now we get into more of the tricky part. As you're triaging an alert, you're visually seeing things and then you might be triggering memories on those things, right? You might see an IP address and you're like, ah, I know what that IP address is. That IP address is one of our Qualys scanners. Oh, that's something I definitely want to know when I'm looking at an

alert. If I don't happen to know that, then I'm gonna end up wasting my time if I spend a lot of time investigating this one alert. I want to know that at the very beginning. So what I'm gonna do is I'm gonna write a QALYS tagging module that's gonna go out and first it's gonna try to find a way to identify what all our QALYS machines are without depending on asset management. And then I'm going to take that list and for every IP address that we ever see, I'm gonna compare it to the list and if it's one of those, I'm gonna stamp it with QALYS and off it goes. So that when the analyst

gets the alert, all that analysis, there's gonna be a little tag pointing to something that says, hey, this is a QALYS IP. And immediately he's gonna start thinking, oh, look at that first right am I wasting my time here because there's a 99% chance that I am so the analysis and the observables can all be tagged in that way and the intention is to impart somehow that emotion of whatever it's supposed to be and then tags can then be assigned a priority and then the sum priority of the tags represent the priority in the entire alert so what I mean by that is you know okay qualus but hey this IP address this is a domain controller our core domain controller is involved in this alert somehow.

I'm gonna tag that as domain controller and I'm going to assign a priority of oh my god to that tag so that when the analyst gets it and it's sort by priority, it's at the top of the list, that's the alert he works and he already knows he's dealing with the domain controller.

And really this is the hardest part. So this is where you're getting into how do I impart that feeling to the user through a program. Somehow I've got to get this thing so that there's going to be a freaking ton of analysis. There's a ton of possible things that you can do. And what we're doing is this system is actually live in production. I'm a programmer who just starts typing it in and starts using it. I don't test. Well, I test in production, but whatever.

So what we're doing is that when we get an alert and when we end up spending a lot of time on that alert, we go back and we look and we think, all right, what were the things that we had to do to come to the conclusion of false positive or not? What are the things that we had to do to come to that conclusion? And those things that we had to do, we then look at and we say, all right, is there any way to automate that? And if we can't automate it, then that turns into a Python module inside the system. And then the next time that that situation arises, it's already done.

We don't have to do it again. So we're rinsing and repeating that process. We're introducing more modules. But the problem that I'm finding is that the more that we do that, the more data, like you just have a freaking data overload. There's so much analysis that you can do and there's so much output of this stuff. How do I impart, how do I take that and communicate to the user that this is what you're looking at? So I'm not sure what this is going to end up looking like at the end. know what it looks like now and it's not probably not what it's going to end up being but that is that is the goal so that's my presentation is anybody have any questions about this

yeah so ACE is basically it's not a sim but it kind of acts like one so we have all these other tools that we've written right we've got stuff that sends alerts to ACE We've got things that are listed on SMTP, things that tail the bro logs, just random stuff like that. And it's all feeding alerts into this one system. And the idea is that rather than having a sim where it says, hey, you need to have all these conditions before we alert, we're gonna take that one single condition and we're gonna start spidering out through our analysis of observables and hopefully hit all those points, all those on the attack surface, and then somehow

present that to the user in such a way, like what Harlan was saying, clusters, right? This is everything that happened and this is how they're all related. What do you think? That's the goal of it. Sure. Are you

relating to other events or alerts that you saw? Yes. Yeah. Yeah. That turned out to be, because we were doing it, I was like, hey, that would be a great idea. And that turned out to be an analysis module. Here's the alert. Here's all the observables. Here's all the alerts that were related to all these observables of these types. How big's your team?

Three, three, six, six, seven? Is that right? Seven, yeah. So developer Nate Hossroth does Intel. And we got another guy here that's doing the last presentation on DLP. So he's our architect. And then we got three guys in Lexington to do like all the Splunk type stuff. tech guy and like we got a guy that's been working there for 25 years so you can give him an IP address and he's like oh yeah that's the router in Sydney. Cool dude thanks.

Yeah? So how do you avoid, so with a lot of the automation, how do you avoid you know a crappy rule in there tossing your entire load? I don't. It just happens and I deal with it. So I mean that's a you introduce these rules that generate like tons of false positives. I've built tools before that did that, so I kind of have some experience with how do you handle like tons and tons and tons of alerts. So that was an important aspect in designing the tool, but it still happens, and when it happens, you just have to deal with it. Cool, anybody else? I see.

Not yet. Sounds like I should.

Have you looked at that?

your name? Justin Swisher. Okay, I'll talk to you after this. Cool.

Yeah, yeah. That's come up a few times. Definitely. Like internally, it's all just JSON type stuff. So I was thinking about how do I export this stuff and graph it, maybe different types of graphs. So yeah, I'll probably explore that kind of stuff. Definitely in trying to figure out how do I present it.

Sometimes it ended up being infinite. So it was really just trial and error. Trying to figure out what types of relationships end up just recursing forever and putting just integer limits on those. But there's also resource limits. So I can't do everything I want to do all the time because I don't have infinite resources. So there's limits that I have to put here and there. But yeah. I got another question. Sure. Is it open source? Yeah, we're working on that. We want it to be because I've written, I wrote a tool before at previous job that they let me open source it and then someone else took that and like ran with it and now we're using his tool based on my tool to actually

do detection. So in my opinion open source is freaking awesome and we're working on getting the business to give us the check mark and go forward with it. But then I have to separate out all of our, a lot of the stuff that I'm writing is going to be custom specific for Ashland. But if I can get that separated and get it out there, I definitely would like people to help me work on it. Sure. Yep. Cool. Yeah. Ben. Hey.

definitely because how long does it take you to work the alert depends on okay when did you start and did you take a break and when you're working on it did you get sidetracked like you always do you know how do you measure that and really it's been so far

people analyze basically the output of that herd and really dive deep. Is that how your herd is set up or do you guys, everybody learn? So the answer is yes. The way it's set up is that we've got basically one or two guys that go through the alerts during the day and they disposition them all or as much as they can. And then right now it's just myself. I'll go back through all the alerts just so I can gain an understanding of what's going on. It's all kind of concern of mine. You got the tools that you go through and you disposition the alert and then that alert's gone. So that guy's only got to

ever know about that. Even if it wasn't something that was not, especially if it was something that was actually an intention. I don't know what it

would be, but that was my intention when I was designing the tool. How can we all get an understanding of what's actually going on and still be able to not overstep

about recursive analysis. You get that thing, and you look at it, and you analyze it, and you get that output, and you're like, oh, I can analyze that too, right? And it's just that rinse or repeat. That's really interesting. It's like we built tools at previous day job. And I think the technique is a super powerful technique, so you can automate it. You can just go ahead and have it all done. Why not, right? You got the CPU resource to use it. Yeah.

Every tool comes with its own console and alerting and stuff like that. Can you talk a little bit about the infrastructure challenges you have to get all this together? It's not an issue for us. So it normally would be an issue with a lot of companies because you have to get a lot of buy-in. So we're going out and we're connecting to all kinds of different systems trying to pull all this information that people would normally have to log into a console to get, right? hate that. At Ashland, they actually take a little bit of security. So we have a lot of people that say we've been pushing, but we actually have some authority to

say, hey, we need to do this. And then the IT teams have been awesome. They're like, yeah, here, here you go. Yeah, all right. So definitely, like, has been a part of the success of it. And then as far as this, like, infrastructure type stuff, there really hasn't been a whole lot of issues with that.

Does that answer your question? I don't know if I got sidetracked on that. Yeah, that's good. Okay. Does

the tools go out and pull these different data sources and pull them together? All the logging that we do at Ashland is a part of the detection part. So we've got all these other tools that I haven't even talked about that go out and pull all those things in and look for all the different kinds of things. We've got a lot of automation with We got all these tools and it's all looking for all the things. We got hunts that we do, I was talking about. We go out, we do the hunt. It's awesome. Okay, automate it. Run it over and over again. And it's all easy in all those logs and stuff. And all that stuff. Only the alerts go into A's, not logs.

What are your favorites? What are the most valuable, best signal to noise ratio? That's a good question. What are the most valuable? I would say definitely the network, all the network stuff that we have. We have network sensors in place where we're running SMART and BRO and SFMG to report traffic. Those are super valuable, especially as part of the analysis to go out and pull all the PCAP. you go and you already have all the PCAP associated with those IP addresses at the time you were, was created, like that by itself is freaking awesome. And it helps me, like, oh yeah. I think the other thing is . So there isn't . So getting all the logs into

one place where we can query them is super powerful. that I'd create a point to.

Sorry? For the most part, I'm not going to jump around the stage or anything.

Alright. Alright, hello everyone. My name is Jesse Lanz and I'm going to be discussing some PowerShell for instant responders and security professionals. I chose the good, the bad, and the ugly because well, for one, I'm a big fan of spaghetti westerns. And I mean, they're Westerns, but what, good, bad, and the ugly is on the top list of Westerns in the universe, so can't complain about that. So our agenda for today, first we're going to cover the history of PowerShell, how it's going to be used in the environment, the specific reason I'm covering the history will be because you're going to run in from the basics of PowerShell, PowerShell version 1, up until they're about to release PowerShell version 5. And it's going to be

important because PowerShell version 5 has stuff like SSH, which is important. I kind of feel that you need to be aware of this. A lot of...

This one? Which one's not mine? I'm not on? Yeah, I don't think it's on. I'm talking too loud. Do we have new batteries for this? I didn't break it, I promise. I'm going to take this out front and see. Okay. Just yell really loud. You guys can hear me back there? Alright, so it'll just be really, it'll be different. So the dangers of PowerShell, so I'll discuss some of the actual attacks that have used PowerShell in the past. Specifically some of the things that we've seen and some of the other big names within the security community I've documented. And then also go over, this is where I'll go over the actual uses for PowerShell within your environment. And the reason that I actually chose

this is because two specific reasons talking about PowerShell is a lot of people don't even know that they're vulnerable to PowerShell and PowerShell attacks. And a lot of people don't even know that they have PowerShell as a tool for them to use. A lot of network administrators or system administrators used PowerShell in the past, but they, as a security professional... Alright, we're good? Alright, but security professionals don't really think of it as a tool. You know, everybody thinks when a scripting language comes out, oh, you know, it's Python, it's Perl, it's Ruby. And granted, PowerShell really isn't much of a scripting language, so to speak, but it's on every Windows system. Windows, what?

version 2 or service pack 1 I believe it was and a bit higher it is by default installed. So a little bit about myself. So I work for RSA. I work in the professional services field. That basically means that I technically am supposed to do a lot of security analytics and ECAT work, installs, stuff like that.

evaluations, use case scenarios, working with the actual member for them. I've worked for some Fortune 100 companies, I've worked for federal law enforcement agencies, and like a lot of you I've got a full complement of alphabet soup that I could put behind my name which you really don't care about. And then you can find me on Twitter where I bitch about customers a lot.

Alright, yeah, so I've been asked, I was told I was gonna get, somebody was gonna get me drunk at lunch so I could tell you customer stories. So I'll start off with, so talking to a CISO recently, posture, now this is not a CISO of maybe a 20 company, I wanna say was around 40,000 users.

We were discussing different ways that he could do things to improve his posture. He was talking about some fish that had come in recently and one of them had a index.php extension. His brilliant idea was at this point you need to block all PHP extensions.

PHP is just an executable just like an EXE and it's just evil any time you see it come across the internet. So that's his words of advice.

I don't know. I did what I was told. He was telling me about his mentoring an intern. And this intern approached him and said, well I just finished a college class, C++. very good but you know I don't want to focus on C++ but I need to learn a language which language would you recommend? So the individual being the brilliant incident responder he is without you know basically blinking says well you need to learn HTML because it's just like C++ and it's a great programming language. So we could keep going on like this because you I'm sure some of my colleagues will tell you that I could tell you about some really really stupid people.

So what we won't cover, this isn't a beginner's guide to PowerShell. I'm not going to be teaching you what security is. I won't make you rich and famous with PowerShell and I won't help you find the one, the person you're love mate in life unless you meet them next to you and then you know that do what you want but do it somewhere else so what we will cover so we're gonna go over PowerShell or overview we're gonna go over the dangers of PowerShell in your environment PowerShell and PowerShell as a security tool so let's start

let's talk about the history of PowerShell. So PowerShell was basically first developed in beta format called Monad. I don't know if I'm pronouncing that correctly and if some of you old ancient guys can let me know if I am. In 2005. In 2006 they renamed it to PowerShell and then the iterations are basically 2002, I'm sorry 2009,

PowerShell 2 came out and was integrated into Windows 7 and Windows 2008 R2. Older system support was added at 2009. PowerShell 3, 2012 September, and PowerShell 4 is October 2014. Currently PowerShell 5 is in development. Got a pre-release April 2015. Class is introduced. They're going to the Zip module, PowerShell gallery, which everyone's not familiar with that, it's going to be kind of like a almost an app repository that you'll be able to sync up and say, I want this module, which I need to go back and talk about something. Do symbolic links and then a lot of people are all excited about OpenSSH. But opinion, what could that possibly be bad for? I mean, you know. Exactly.

So going back. So with PowerShell, the initial PowerShell, it had what was called plugins and modules. With PowerShell 2, they went away from plugins and now it's modules, but supposedly there's still support for plugins. Alright, let's talk about the dangers. The popular statement, and there's quite a few advocates for PowerShell, and I'm sure you've all heard about other presentations about PowerShell,

in a security environment, and there's a lot of people that are completely, PowerShell is no more dangerous than anything else in the environment. Which, in a sense, is true, but if you were listening to Harlan Carvey talk earlier, This gives you the capability to have command line access to a Windows machine. Most of specific advanced attackers are going to be using command line access because that's the easiest thing to do. It gives you the capability to basically just pull in modules that you want to use. You don't like it? Well, just download the module, especially if it gives you an app repository to be able to specifically, you don't have this app on there or module on there. to bring it down, give it to me.

There you go, you're good to go. It's not going away. Microsoft has went all in on PowerShell. It's not specifically just PowerShell, but they've basically said the future is PowerShell. I think they were tired of Linux, in a sense, kicking their ass as far as command line access and things like that, so they wanted to be able to do that. You can't disable it. I've went into customer environments and handed them PowerShell scripts and say, hey, this is a great tool. You can use this script, help you do a lot of different things. They'll say, well, PowerShell's disabled. You can't use PowerShell. And then I do a couple commands, and PowerShell's instantly working. Like, you hacked our system. No, no, you're just an

idiot. You don't know that PowerShell's still enabled.

And then it gives you the keys to the kingdom. Now granted, you've already got the keys to the kingdom. To use PowerShell, you've kind of sort of already got access to the system. So that's probably a misnomer. It's probably a bad saying to say that. But how about it basically, instead of having to use a lock pick to get into the system, now you've actually got a physical key. I mean, you've still got access, you've just got it easier.

So this is basically a simple command and there's half a dozen, a dozen different commands to bypass any policy put on that said, nope, you can't have policy, you can't have PowerShell, we're not allowing PowerShell scripts. Simple PowerShell execution policy bypass. Wow, that's secure, isn't it? You got an execution policy, well let's just get rid of that, we don't want to use it. And that's literally it. You can do the first one for scripts, the second one after you run PowerShell at a command line. And it gives you full access to PowerShell. So there's some different exploits that have been talked about. And one of them as far as that is, can strange spaces save you? And I put a bunch of links in here and I'll

share these slides afterwards. I was a bad boy when I was getting my slides ready. I didn't actually give them to anybody beforehand. But this is a really good article written in 2004 by, I honestly can't remember the guy's name, Climber. But he's very good in community, very good with PowerShell. This basically, what this does is you if constrained spaces are supposed to be you can only run a certain amount of modules in PowerShell and that's the way I understand it if anybody else has got a different understanding you know speak up and tell me I'm an idiot I'm I got thick skin what this does is just a little bit of a bypass and it gets you back and gives you full

access to all the modules that are currently available in PowerShell so again it goes back to the whole it's not disabled there's There's really no way around it. If you got it on there, someone has local admin access, they have access, period. So, talking about some of the PowerShell attacks in the news. Anybody have a venture to guess? Okay, so PowerShell is used on your system specifically by a lot of sys admins. it's going to look like normal use. Specifically, if you use something to install program, PowerShell to install programs, PowerShell to some admins that are really savvy will use it to do different Active Directory items and things like that. So what you end up basically doing is you've basically

sort through the chaff. You've got to look for the needle in the needle stack. So which specific PowerShell actions could have actually been malicious. So I'm going to go into this, but first I'll tell you a story. So I went to another customer, and it's somewhat related because it'll go into my next way, into my next stuff. Went to a customer and they came to me and they said, we've got this Linux system. recently brought it in from outside the DMZ. So I'm like, what do you mean you recently brought it in? They said, well, it wasn't covered by a firewall. Well, it's Linux, so it should be fine. But it wasn't covered by a firewall. And we recently brought

it into the corporate network. We just need you to make sure it's not compromised. And I'm like, well, OK, it's compromised. No, no, no, we need you to look at it. No, it's compromised. Just put a thermite grenade on it and burn it to the ground. No. So they bring it in, I start taking a look at it. And they're like, well, check the memory, check to see what's going on. And this is where it comes into what I'm about to talk about. They're like, well, let me look at the uptime first. The uptime, seven days. What do you mean, seven days? It's a Linux system, why is it up in seven days? When did you bring it in? We brought it in six

months ago. So what are you doing? Well, he reboots it every week. Does he reboot a Linux system every week? I'm thinking, oh, maybe he updated it recently. No. So it's a Red Hat Linux from, it was built in 2007. So it had been sitting on the network or outside of the network since 2007. Memory was useless. Logging was set to roll over every five days. Great, yeah, I know, it's outstanding stuff. And on top of that, the... admin who had been using it, had been using it as his own personal FTP server. So he was using it to store whatever, let's say inappropriate videos that he likes. So, yeah, at that point, again, my recommendation was here's a thermite grenade, just burn it to the

ground. They don't listen to me. All right, so what does PowerShell look like in your system? in your system, in your network, on your devices. So you've basically got your standard, what can you look at in the network or on your device. You've got your memory, your network. What other kind of things can we look at? Logs, anything else?

We can look at the disk. That's really about it. It's not going to put anything in registry because PowerShell is already in registry. It's already there working.

So if we run a memory capture on it, again we're going to see different things. We're going to see logs. We're going to see network. What would you see? What kind of things would you actually see in the logs though? All interactive and stuff. You guys just lunch. They gave me the shitty spot, huh? Right after lunch we were all asleep. Okay, so you go see basically your installs, your different things. Basically it looks like standard usage. You're going to look like you know what an average user would do. But you can see the programs that run. Specifically what I'm going to cover is if you use memory. So we're going to actually take a snapshot of memory and look at

different things, you know, what would happen. So we pulled out and we're going to use Volatility Command Scan. Everybody familiar with Volatility? Yes, thank you, good. All right, so looking at it, it's going to show our PowerShell. Anybody see the problem here with this? We had Harlan talk about this earlier.

Say again? Execute through command line, not PowerShell. Exactly. So PowerShell got executed, and then it brought its own console up. I think that's what you said, right? Good, I'm deaf. So we didn't actually see anything. So we go to consoles and that might help a little bit more. So then we can see all kinds of goodies in the console from application powershell.exe.

But then we go back to the commands and it's still just powershell. So you have the capability to look into the powershell console and see what different commands you may run. But you command.scan isn't going to help you much. But that's really the only way you're going to see exactly what the attacker did on your actual system with PowerShell. As far as could you do it through full packet capture on your network, that would depend. I mean, you're literally going to have to look at was your attacker nice enough to use plain text

you know connection did he you know did he use a web shell which if he used a web shell yes you'd be able to see it if he used you know Encrypted RDP what good did it do you didn't do anything so

So with that being said that that's what you're gonna have to worry about in your environment now should you want to use the these actual tools within your environment which reasoning behind my logic for this was going into one of my infamous customers. They asked me to work with them in their incident response role. So I'm working with the company and they come in and they hand me a whole bunch of registry keys. They say, okay, this is what three letter agency gave me and said these are evil, find them on your network because if you find them it's bad. I said okay so what host based scanning tool do you have in place for

me to use? What? What's host based scanning? No really, what do you have? Well we got semantic. Oh great, thanks that helps. So my idea for them was to use a simple PowerShell script to be able to read specific text file, so to speak, pull in that and scan, go through each registry. Had to be able to run it from the domain controllers, had to be able to push it down, and it ended up, that got tanked just because, well, they didn't want to put any effort into it and they would rather actually just pay my contract out for me to sit and watch cats on YouTube, I guess, I don't know. So,

you need to do as far as PowerShell goes, you need to be able to find out what version. This goes back to what I was telling you before because in certain versions you're going to be able to instantly check different things and we'll talk about that in a little bit but like PowerShell 4 has the capability to do MD5 some on a single file. Before that you had to go through an elaborate script to just get a MD5 and it would involve a bunch of math and different things like that that I'm not gonna go into, because I suck at math. But the simplest command is at a PowerShell prompt, PowerShell.exe, command prompt, gives you the ps, blah blah blah, run psversion

table.psversion. And that'll tell you which version, which subversion, which minor patch to said subversion you've got. So if you want to then use your your PowerShell guru, your Kung Fu, that you're now, I'm going to get all excited and use PowerShell in my environment to fix the world. Good luck. So then you can do different things. Just for maintenance, you can check your patching. This should be an admin job, but I know you guys have never had to do your admin's roles for them at all, and that should never happen, I'm sure, right? So you've never had to check your patching. You've never had to check your user's privileges. never had to check your use modif, you know, modify user. I'm sure that happens like

never, right? They're all telling you that. So patching. This is a simple example. I stole from this guy. Well, it's actually from Microsoft, but somebody submitted it. And you can't read it, but I'll, like I said, if you email me, which if anybody needs my email address, then let me know. But I can provide you slides. There's an actual script out there. module, so to speak. That's one thing that I like about PowerShell is you have the capability that if you wanted to do anything, you know, it's kind of like what rule 32 or whatever, you know, you think about it long enough and somebody's actually written it for you. Rule 32 is probably a bad example, but. So then you can

list all the patches that were installed in the last two days. So, you know, you got the patch Tuesday came out and hey you know did my admins patch this server or did you know did I get pwned so to speak. I actually used this a week ago so we had a phish come in at a customer I was working at and said customer wanted handed me a list of 40 email addresses said hey I need the user names for all these 40 email addresses. It's like well you know do you have something like quick scripted search you can look at? No, no, no, just use the active directory MMC console and look up each individual user. Took three minutes to write the script.

Just basically path to the line, line out, give me your email address, and then give me the SAM user account. But that five minute script as an incident responder saved me probably about half hour or hour and a half maybe I don't know I'm slow of clicking a mouse which I hate clicking a mouse so so I talked about this a little bit these are just examples what the whole process in my logic for for a lot of this stuff is I go into a lot of incident response teams and they don't really understand what they can do with PowerShell and that why I wanted to bring this up because I've been in environments where they don't think outside the box. They don't think of let's solve

the issue with the scripts that are readily available. Well, I've had people say, well can't we just install Python or Perl on these Windows systems? Why would we do that when we've got PowerShell already installed? Granted, it's different. You might actually have to learn something new. God forbid you learn something new, but it's available. probably going to peak everybody's interest in here because you know everybody wants to be a pen tester. Powersploit. If you haven't seen Powersploit it's pretty much what if you wanted to go exploring what your attackers would use against you go look at Powersploit. The pivoting lateral movement like Harlan talked about earlier adding users downloading payloads antivirus bypass.

I'm not even going to go into it too much because I'm not a pen tester, but I know it's going to interest a lot of you individually. And then what Harlan again was talking about earlier, PowerShell and Mimikatz. Simple script, invoke Mimikatz, and it goes out, downloads Mimikatz, and launches it on your system. Gives you all kinds of goodness. Forensics.

ago I was working with a couple smart individuals in here, smarter than myself, and we had a live response script that we ran. And everybody's familiar with live response script? Right? Yes, I think you all are. And some of these individuals went to a little conference just a little south of here called DerbyCon, and they were all excited about PowerShell to point. And

named Nick Hoffman came back. I don't know, you guys know Nick? Is he in here? Did he ever show up? Little turd.

So he came back and he said, hey, you know, this is really awesome. Him and another little guy named Nate said, we're going to write LiveResponse in PowerShell. And lo and behold, somebody already done it. So, well, late but then what I was talking about earlier was indicator scanner and memory collection for memory tools. Now I tried the memory collection and it works only on specific versions so I'm going to tweak with the tool a little bit but that's an individual and he's on my last slide as people to pay attention to. He wrote Powersploit is a freaking genius when it comes to PowerShell. If anything he's got, read it. He's the one that wrote, there's the link to the memory capture. I was going

to give you an example of it, but like I said, I could never really get it to work right. So there's the live response. Excellent paper at sans.org on live response with PowerShell. Individual wrote this up, and then On CodePlex, there's a sample of the code. For those not familiar with live response, it basically, it's, I know there was an individual talked earlier about if you had to deal with lawyers and things like that. This is more for your intrusion. I'm never gonna prosecute, we're never gonna do anything beyond find out what they stole. So you collect your pertinent information, memory, volatile, stuff like that. logs, different things like that. Put it into a nice package where you can download

it and take a look at it on another system. But I recommend if you get a chance to take a look at this guy's code. It's really good, fairly neat, does a lot of what you need. So if you don't have anything in place, this would be a good item to get. That being said, you don't really have to install anything else. You've got PowerShell. It's on your systems. If you're interested in live response, there are some live response for Mac and for Linux out there. And I have worked on a few of them. But yeah, well, that's where the ugly part comes in. Anything that I've coded is the ugly part. Indicator scanning. So here's kind of a snip of what the tool

was that I was working toward. Specifically I cleaned it up a little bit so it does a regex to look for a specific IP or I'm sorry all the IPs so it'll search the HKCU and It'll specifically look in each registry key and does it see an IP address and then give you a response and that's all it does my actual script what it had done was It's supposed to be its own. I guess I missed the slide. I must have the old one. I apologize. But my script would go through, read a specific file, and give you different registry keys, and then go into each system remotely. Well, you would push it down remotely and

scan the file. Then use something else I'll show you in a little bit that I've gotten in a more cleaner script to email me the results.

Alright, so we found evil. Like I said, I'm not giving you a how-to power shell. So I apologize if everybody's, you know, there's somebody disappointed. I got people walking out on me. This is terrible. No, I'm just kidding. So

when I was working as an incident responder for a local company, I hated users. I'll be honest with you. I hate them. And some individuals in here can attest to that. I would interact with users and I would send them a message on our little instant messenger and say, hey,

you've got detection. We detected this piece of malware on your system. We just need to clean it up. The amount of things that would happen suddenly to the laptop you tell them, hey, you've got something on your computer we need to investigate, clean up, would amaze you. Or maybe not. You would send them a message, hey, you've got something on your computer. Suddenly the computer would go offline. Three hours later they'd come back and say, oh, really? Did you talk to me? I just had my system re-imaged. I didn't know you were talking to me. Hard drives were shipped but never arrived. Oh, the mail must have taken it. And then, you know, of course you've got, well, the whole,

you know, the expression from dual cores music where he's like, oh, I just, hold on while I run Antivirus 2009. And, you know, it happens. It happens so much. I literally asked for permission to waterboard some of our users, and I was denied. I did put the request in multiple times. I was still denied. But, yeah. I do work for RSA not NSA so I guess that I didn't get the proper training for waterboarding. So but I wrote a script because I hated users that we would run remotely for an mCert malware cleanup. We were having a big problem with Zeus so to speak and so I would work run this script remotely on the systems it would download mCert the system, send me a

nice email with their logs. We can use it for user removal, computer removal, and then file scanning. Now this is what I was talking about if you know where we go with the remediation. I don't know why I put this here instead of detection but I don't know, one too many beers, whatever. Alright so my cleanup script which I posted on the internet nice and clean.

Actually, this is the older version. I've got a newer version, little tweaks and stuff like that. But again, it's PowerShell, run it, PowerShell, execution bypass, and your whatever you want to call it. It goes out, you modify it to your proxy and different things like that. It goes out nicely in the background, runs mCert, which everybody knows what mCert is, right? Microsoft's malware removal tool. Downloads it on the system. it up, sends you an email with all the goodness that it's done, throws a dollar in the Microsoft bucket from your Bitcoins, and then cleans itself off the system. Removes the PowerShell script, removes the msert. Like I said, this is the old one because I cleaned it up a little bit. So

that's kind of my example of my best script, which is pretty pathetic, I know. But that's what I use it a lot for. I'll go out and write some different things. And most of what I write is on the fly, but that's where I was trying to get the point across to you all is you need to be able to have these tools in place where you already have them. You don't need to worry about going out and installing Python or going out and finding a new tool to clean up what already exists because Microsoft was nice enough to give you a tool that you and China can use to exploit your system. Alright, so these are three individuals that I would recommend

highly if you could read anything from them. It's worth reading. That being said, Jeffrey Hicks and Don Jones are Microsoft

And Don Jones is the one that says there's nothing in PowerShell that is malicious, it's just how you use it. Well, there's nothing malicious in anything that they seem to exploit on Windows, but it's there. And then Matt Graber, he's the guy that once you get the basics of PowerShell, read everything that man has wrote. He wrote Powersploit, he wrote the memory tool, he's got a blog, he's well worth reading everything about.

And I got about 20 minutes, I kinda went a little bit shorter. So let me see, what other stories can I throw out for you? Let's see, I'm trying to think what else. Yes sir? Have you ever heard of PoshSec? PoshSec, yes I have.

Right, so that's the PoshSec and it's like he said, he was talking about and I'll go into a little bit because I haven't really looked into it a lot but if you get the opportunity I think it would be very beneficial. Have you used it? Yeah, a little bit. Ben 10, one of the guys that helped develop PoshSec was actually at a conference a couple weeks ago and he was talking about it, running through some scenarios and it's basically a framework for storing a lot of these scripts. Right. Yeah, in GUI format. So if you don't know which one to explore, read, determine if that's what you want, then you can go through and select

them on the Graphic User Interface. And it does everything from baselining all the way up to lessons learned, the entire incident response chain, if you will. And it's actually really good. Yeah, see, I stumbled across it a couple weeks ago and then never got around to adding it. But I'm glad you brought it up because I completely forgot about it. So, PoshSec is, like you said, it's a bunch of modules that some very smart individuals wrote for specifically for this type of scenario that PowerShell for security professionals. I think that's actually what the title is or something along that lines. Yeah. Yeah. It tends to be for people that are just starting out to use the tool until they learn PowerShell. Okay. So, you know that term.

No, no, especially if you build a tool that's just for newbies. Once you gave them the easy button, why are they ever going to use something else? So yeah, we got that. All right, so we got any questions?

Yes, sir. Where do we start with learning PowerShell? OK, so if you don't know PowerShell, There are some really good videos YouTube videos honestly There's two books that I recommend actually by Jeffrey Hicks and John Jones one is the PowerShell in like a month of lunches or whatever and the other one is the PowerShell handbook or maybe I don't know it's got some it's from Manning it's got some guy in a Weird-looking dress or something on the cover. I don't know what it is, but it's really good Did I bring my copy with me? not. So those are two books but you don't need to really buy anything. Like he was saying, go to the PoshSec, look at those modules.

Microsoft does have a good beginner's course of videos on PowerShell. We had an intern recently at another customer and he asked me the same question and and that was basically what I recommended. I gave him, there's a couple YouTube links, and I don't have them off the top of my head, and then a Microsoft video series on basic PowerShell, some of it being taught by Don Jones.

Any other questions? I got like 15 minutes. I'm sure you guys probably want to just go wake up or something. Yes, sir.

So that's going to go back with your execution bypass and I apologize for not including that. what happens is with a lot of, you can't physically disable PowerShell within your environment at all. But you can set it so strict that only signed modules can be run or signed, you know, it can only be used by signed processes. Or it can be, you can set it to, you know, it'd be signed, unsigned, or just, like I said, bypass or no restriction whatsoever. As far as be set to only run specific files, like download an executable, and if that signature is there on that executable, I think it goes back to the whole thing. And I don't know, so if anybody knows, correct me if I'm wrong. It goes back to,

it only does the signature checking that is set at the GPO. And if you've just disabled that for the session or bypassed for that session, it's not going to matter because it's gone anyway. As far as MD5 signature, that's only going to happen at, that's only a module that has physically been added by Microsoft at version four. There is a way to do an MD5 signature prior to that, but like I said, it involves a bunch of math calculations. Any other questions? Yes, sir.

I didn't even know it was possible to log specific PowerShell commands. In what way? I saw a recent post where you can kind of hack the PowerShell from a little bit so that it logs it just like you would the command prompt commands for the event logs. Of course, you have to actually get event logs set to that level first, which is a challenge, but I didn't know if you'd ever actually seen that. I have not. What I found in most environments that actually do have that level of technical expertise to be able to do things along that lines, they either have one of two problems. Well, they don't care because they just don't care or they don't have the storage that would be

necessary to maintain that amount of logs. Right now, just in many of the environments I consult in, the of storage that they do to maintain a specific time frame and just a month or do a cold storage. We're talking in the hundreds of terabytes and just try and get the execs to get a decision on what actually do you want to maintain for a certain length of time. I can guarantee that command line is not going to be one of those options. You know, great, I would love to see it, and I think it would be awesome, specifically as an incident responder, to be able to see logs from command line. You know, some CEO somewhere is looking and going, well, can I fly my personal jet

to my next golf outing, or should I get some more logging? And what do you think they're gonna take? Obviously the logging, right? So,

any other questions? Yes, sir?

do not I rely on what the customer wants so the question was do I rely specific on PowerShell for my all my LR analysis I do not I went into environments where fortunately we've had some level of you know disposed intelligence that may or may not have come with my original contribution seven years ago. But, and that's just by coincidence, most environments that I go to either, like I was just saying, either have the expertise in place that can already do some type of analysis as far as live response or they just don't care because they are to two people deep and I'm just giving them enough scripts or enough items to basically get their

security manager, security director from firing them. Any other questions? So in most environments, it's like you were saying, or the question you were asking, so I do a lot of consulting. flew in from Chicago Thursday night. I'm flying back to Chicago tomorrow. And I have to go with what the customer wants. I can make all kinds of recommendations, but I like to default to PowerShell for specific reasons in that it's already there. I've already got a canned set of scripts that I hand to the customer and say, hey, I'm making your life easier because well you suck right now so I got I'm doing my due diligence there's no warranty there's no guarantee there's no nothing here read this read this book read this

you know watch this video just learn some something do something because you're not gonna these customers may not have the capability to spend a million dollars on RSA's ECAT or you know something along that lines you They don't have that. Their technology is not there. In the incident response field, a lot of people call them the low-hanging fruit. There's a reason there's a low-hanging fruit. It's as I was saying to a couple individuals here recently. Cincinnati is surprisingly, you wouldn't believe it with the smaller community it is, a lot of intelligent people here. It gets a lot thinner base of intelligence once you go out to different areas. These people are getting... Yeah, I know, that's saying a lot,

isn't it? You get these areas, these companies that are really... Because we're talking about an individual, a co-worker from past, and I that went to a company and was ready to quit after a week because the individuals that he was working with, you know, to put it bluntly, couldn't fight them their ass with both hands. And it's just you get to that point where you're going to a company where there's 35,000 individuals and they're willing to put two positions toward incident response. And you're like, wait, wait, wait, okay, so where's your other security team? Oh, well, that's you. You can do SecOps too. You can do architecture and so on and so forth. Like, so how

much time do you actually spend for incident response? Well, I checked logs every Monday morning. I was like, oh, great. Well, good, bye. I'm sure they won't do anything before then. Any other questions or questions?

I have. I'd have to refresh my memory what I've specifically done, but I've also heard rumor and someone else can confirm or deny. I do not know. I do understand that they're in the works to replace some of the SysInternals tools specifically with PowerShell scripts.

Anything else? Okay.

So I'll talk about one last customer and then I want to leave you on a good note otherwise you guys are gonna leave here and you know go he just didn't really tell us anything but well I didn't but I told you about a lot of customers that are idiots. So just recently had an argument with a customer. I know we talked about volatility a little while ago and I'm gonna fill out my eight minutes don't worry about that. I got seven minutes yet. So, had a conversation with a customer. Customer was all excited, thought he was the super guru of everything that was memory analysis. And for those of you not familiar with memory analysis, I apologize because it might go over your heads. But this individual stripped

it out to take a memory capture from a box he thought was infected. And he ran volatility and ripped out all the EXEs out of the memory capture and then had a script that would then submit all those memory capture MD5s to VirusTotal and was all excited because every time he ran it he'd come back with no positive hits so he his system must be infected with APT because you know Lord knows APT doesn't have anything on VirusTotal and everybody knows what the problem with that is right Well, I guess you don't. But when you pull out an EXE out of memory, the odds of you actually getting a MD5 of the actual correct file are about, what, like one in 10 billion or something

like that? So he checked each time and roughly would pull out about 70 EXEs, and none of them were on virus total. So he was creating all these incidents with all this new malware, and he wanted to send it up for all these He had a contract with an IR team,

specifically from some AV vendor, and he would submit all this malware and want a big detailed report as to what malware these attackers were specifically using on his system. And I had to pull out the art of memory forensics and show him specifically in the book where it says, this does not work. And he stopped talking to me for two weeks, but you know. for me.

Alright, so that's all I got. I hope you guys got something out of it and you know if you got any questions or concerns or you want to just tell me stuff with the damn stories, feel free.

No, that's Josh's.

I'll put up my first slide here. If I remember... Oh.

Oh, that's right. Hang on. I have to go to settings, displays, this thing, secondary. And then, I don't know. Apply. Let's see what happens.

I'm gonna bring this back up. What resolution should it?

Oh, mirror's not the way it's done here. And I don't want to mirror it up. Put this thing up. What? I do not believe that's possible on this system. Hang on a second. Oh, it was just, yeah. I just had to play resolution roulette.

That's what I'm going to do. F5s? Perfect. Well, what? Oh. So it's like, hang on a second. I might be able to fix that. Give me a second. Do I have that as an option? 7, 6, 8? Yeah, I do. Okay. Did that change up here? Okay. Okay.

How's that look?

How much of the top?

Can you get enough of it there? I just changed. I can look at what I got. I got the adjustments on mine, but I think I got most of it. I'm going to start recording now and do some adjustments. All right. I'm going to deposit my phone up here.

She's doing all right. Well, actually you tell me. She's a nice lady. Yeah, that's what I hear. Ladies and gentlemen, from General Lockerbie Fulman. Yeah. All right. All right, everyone. So I'm from GE Aviation. I've been working there since 2011. Again, some of the other folks in this room either work with me or have worked with me in the past. And over that time, so over the past four years, I've put together the cyber intelligence program at GE. More recently, I've become a program manager responsible for both the cyber intelligence and the detect operations side. So trying to merge those two programs together to try and help us from say common practices. In addition to that, I also am enrolled at the University of Cincinnati as a

PhD candidate in their cyber operations track for computer science engineering at the University of Cincinnati. I think I said that. My advisor, Dr. Franco, is actually here sitting in the audience. If anybody has any questions about UC's cyber program, he would definitely be a good person to talk to. Also, I believe Mark Stockman is here as well. I'm not sure if any of the other faculty are here from the IT department. He can talk to you about the cyber security track that's over in the IT school as well. I also graduated from University of Cincinnati, Bachelor of Science in Computer Engineering.

I've been a frequent contributor to the CRITS project, which we'll actually go into later. I've been a FreeBSD community member and contributor since 2001. Some of you may or may not be familiar with that. It's very similar to Linux. It's just another Unix operating system that's open source. I've got five years experience in IT security, and I found that this is actually very common. You have some. So like Harlan, earlier, has a huge amount of experience in cybersecurity, but I found that a lot of people in this space able to get up to speed really quickly in a very short period of time so I've also got a lot of experience in just kind of general computer science and IT I'm a Cincinnati native so I grew up

here I didn't live here my entire life but I lived here for most of my life and I like running mainly because I like cake and other sugary things and also because after a long day of hunting down incidents and dealing with that type of stuff and actually seeing what you know attackers were able to get away with I to like to work that out of my system. I have a five-year-old daughter who goes to school here in Cincinnati as well. I'll actually be hanging out with her after the close of this conference. I love coffee as many of my co-workers will attest. It's probably the only thing that keeps me going most days, especially

with all the other stuff I've got at the top half of the slide. And then a common running theme among us Cincinnati InfoSec professionals is the cat fancying

all seem to enjoy. Amazingly enough, I managed to get this presentation together with memes but no cats, which is surprising. Sorry to disappoint. I'm getting the heckling from the crowd. Come on now. I promise that next year there will be a cat in every slide. All right. So getting started, what I'm going to do is I'm going to introduce some terminology that I'm going to use repeatedly. Cyber's on here, so for those of you who are following along in the dining area, I feel sorry for your livers because cyber will be brought up multiple times throughout this conference and through the rest of my talk as it is on cyber intelligence. So what I like to describe in

kind of what you're building cyber intelligence off of, and I'm actually gonna work from the bottom up on this slide because I really just put these together recently and I didn't practice. So cyber intelligence is what I call the analogy So it's knowing these two things, I believe John Davison gave a great presentation earlier about false positives. And false positives end up being a lot of times when you detect something and the alert shown to you is actually describing legitimate user activity. So it's describing activity that you don't want to stop. But you also, when you built that detection, did not predict that your users would actually be doing that. You've actually just learned something about what I call your attack surface, which is your user base, the

elements within your network that I call within your control or at least within your influence, network design, user behavior, site geography, security tools, things like that stuff where you get a say, you get some input, Sometimes you even get direct control and as you've gathered from a lot of the discussions here, you had Jesse talked about it from the service provider side where typically they make recommendations to clients and they describe findings to clients and then you had John talking about how they're able to pretty much have a huge amount of discretion over what they do in their environment. It's a huge varying control over the attack surface. So that's basically, you know, to use the military terminology or whatever, that's kind of

your territory, that's what you're defending against the attacker, that's what you're trying to keep the attackers out of. Your threat landscape on the other hand, that happens to be all of the elements out there on the internet which are trying to attack you or pose threats to you. Some of those might be adversaries trying to pose threats. In other cases they might be like Microsoft implementing some sort of new feature in Windows that helps you administer systems, but didn't necessarily go through a huge amount of, say, security review, or maybe just introduced a lot of functionality. So like the PowerShell talk was brought up earlier. I thought one of the very interesting things about that talk was the request for privilege

elevation within within PowerShell is very similar to when I write a Java applet and I want to request elevated privileges from within the browser as well when it's running a Java applet. That's a feature that definitely has some very good use as far as system administration goes or software development goes, but is also very easy and very valuable for an attacker to exploit. That's the type of thing I call the threat landscape because typically those things pop up need to defend yourself against them or you need to be building your infrastructure to detect those types of things. So in a nutshell for cyber intelligence, this is what I like to image I like to depict it

with is intelligence beats size. You don't have infinite resources at your disposal. who gave that presentation that was one of the big things that he said was he doesn't have infinite resources he can't analyze all the things he still needs to be selective about the things he analyzes what his strategy is is he's putting together some tactics to help him make analyzing what he's looking at more efficient regularly so that he can increase the scope he can broaden the scope of what he gets to look at with the same number of resources so in this case everybody can see over the bar, but the smallest kid can see over the bar the farthest because he happened to figure out the best way to look over

the bar in the first place. So one of the things, just from the intelligence point of view, and this will be a slight divergence from the more tactical topics that we'll talk about, but one of the ones I like to drill into people's heads is the concept of certainty. So in this case, I have this quote up here, and I said it was quoted by Abraham Lincoln. Do any of you actually know if it was quoted by Abraham Lincoln? No, probably not. With this type of thing, you can be pretty certain that he said this. Somebody recorded it. They don't have any relationship to me from when he was alive, so they wouldn't be trying to pollute the water for the

purpose of me giving this presentation a lie to you 200 years later or whatever, right? 150 years later. So if you can find this published somewhere, then he did say it. If you can't find it published somewhere, that does not necessarily refute that he may have possibly said it. That just means that you have not been able to find any evidence that he said it yet. So you cannot necessarily be certain that he did or did not say it. So to use the Wikipedia terminology citation needed. certain that he didn't say this because I just kind of made it up and I would be pretty right on the mark if you know it was actually something that came into

my head and then I was like well it would be kind of funny if I just blame Abraham Lincoln for it so Abraham Lincoln also known as the person who says a lot of things on the internet that he didn't say so

yes so the reason I brought that up is because you can't really be certain of anything with sufficient amount of evidence. You can be confident of it with a certain amount of evidence, I should say. But you can't really ever be certain of anything unless you perceive it, unless you experience it happening. And so much of the attack surface, threat landscape, all that stuff, your view of it as a security professional, what you're trying to defend against, what your interpretation of, say, the attacker's actions, is all based on your perception. And your perception versus reality is almost always not overlapping. You, as a security professional, are trying to make it so that it is very closely overlapping, but you do not necessarily know. So for instance, when Harlan

Carvey was up here earlier, he described that attack where they put a web shell hosted on local host on a web browser, or on a web server, and they used the user's web browser to go to local host and execute that shell. But they weren't certain as to why that approach was being taken. There might be very many reasons why it was taken. you do not know exactly what's going on inside of the attacker's head at the time. And I have a couple of opinions on that. Maybe I'll bring those up later on if I can get through this fast enough. So I put two examples of environment is constantly in flux, which basically means that if I was to learn exactly what my company's environment looks

like today, a week from today, my attack surface is going to look different from what my picture of it was today. The same's gonna be true for your attackers as well when they're in. They're gonna suffer from the same fog of war type of approach. In this case, what I gave an example of is, a lot of you probably saw the news recently, Lockheed Martin's buying Sikorsky from United Technology Corporation. Many of you have probably seen the UTC, United Technology Corporation vans driving around. They also own Sikorsky as well as a number of other companies.

selling this division, Sikorsky makes helicopters a lot of times with a predominantly military focus. What this basically means is that Sikorsky, which needs to keep making aircraft, needs to keep making helicopters as well as everything else they make, during this change is going to be transitioned from UTC who built their network originally or possibly altered their network since they bought them because I don't remember all the history of it. move that thing to Lockheed Martin which can only really determine what that network is comprised of what are they inheriting based upon what they can perceive from running through as many assessments and things like that that they can afford so they can only become relatively confident in the security

posture of that network they can really only become confident through a level of uncertainty with how integration is going to impact the security posture of the rest of the Lockheed Martin Corporation and it is their job as Lockheed Martin who may want to enable deals like this in the future this type of thing to make sure that they are building a attack surface that can support a lot of those uncertainties and that's where you get to the point of a lot of implementing a lot of detection solutions, trying to come up with new ways to implement detection solutions, and I think kind of was the underlying argument with the comment made earlier about does prevention work? And the right

answer was no, right? And it isn't necessarily that prevention fails, it's that the attacker's job is to get around prevention within your network, and on a long enough timeline, prevention is eventually going to be overcome by the attacker. So I think the example that was brought up earlier was prevention would have prevented the execution of that rogue binary in the startup menu earlier. Well, they probably already had prevention installed and prevention didn't work. They prevented a lot of things, but you can't prevent infinity. And I think that was the basic crux of the argument between whitelisting and blacklisting and everything that came up is that at some point you are selecting what you're preventing you're also selecting what you're permitting because business needs dictate it because

Lockheed Martin and Sikorsky and General Electric and Ashland and others are not primarily security companies. They're industrial companies and they're trying to enable industrial users to do their work. So the other thing that I'll add up here that also introduces uncertainty is the threat landscape. So again that's the that you don't have control over necessarily, you don't have a huge amount of visibility into. In this case, I picked on the OPM breach, both because it's current, both because it might be near and dear to the hearts of many of you. So in that case, I'm highly confident that the risk of misuse of OPM information about employees is likelier now than it was in 2013, before that information was alleged to have been leaked.

I'm not certain that it will be misused, but I'm highly confident that it will be misused. I will now want to take measures to change my attack surface, possibly by implementing focus detection on those people. I don't know. Come up with a new plan on how I'm going to change my attack surface to respond to the fact that I just received a large bucket of information that the threat landscape changed. two weeks from now, some other crazy thing's gonna happen out there in the threat landscape that's gonna dictate to me that I need to change my attack surfaces in some manner. At the basic level, this is kinda what intelligence is. I learn what's going on out there, and then I take measures internally which might

be deploying indicators to a whitelist, or might be deploying signatures to proxy blocks, or might be deploying to Snort and Suricata tools and things like that. Basically try and change my attack surface or change the visibility I have in my attack surface, re-instrument my attack surface to respond to what I learned that's going on out there. Like hey, I just learned all this PowerShell stuff and a whole bunch of information was talked about in here about the vulnerabilities in PowerShell, how PowerShell gets used. Maybe I didn't know a lot of this stuff ahead of time of this conference, I could take a lot of that information back and share it to the people who, for instance, run a live response script or run some of the

host intrusion detection systems that we may have on our systems or the insider threat program, things like that, that might actually have instrumentation but are not leveraging it sufficiently because they did not know what activity to look for in the past. So, oh, and then here's the perception versus reality. I thought this was a really good slide that kind of explains to you the pitfalls of assuming that what you're seeing is actually real. You can get yourself at least into embarrassing, if not troublesome situations, if you make this mistake.

So, is there bad feedback over there? Okay, you just turned your head and then it changed.

All right, so I listed out some factors that are impacting confidence and I'm not going to dwell on this slide for a long time but some of the big ones that I would add is some information sources more reliable than others. I don't think that comes as a surprise. That's true both in cyber world as well as real world, right? Some people you can listen to their gossip and it makes sense. Some people are always just trying to game you that type of thing.

investigation can increase confidence but also increases delay to reporting the more time you spend looking at something the the longer it is to get that information over to somebody. Age of evidence I learned something two years ago that was true two years ago not necessarily true today even a lot of the geopolitical changes recently might suggest that the reports that came out for 2012 may not necessarily be describing predictions that apply to 2016 because of all of the other factors that go on in the world. So now we'll get back to like the technical side kind of what I put together I just kind of wanted to frame that stuff as a like cyber intelligence talks

really good place to to put that type of information that type of discussion that type of mindset because trying to educate and encourage thinking that way that what I'm looking at is not telling me fact it's just suggesting things or reinforcing conclusions or helping to support conclusions you want to look at a lot of your analysis from kind of a I should say you want to look at a lot more of your alerting and a lot more your investigations from a scientific perspective from the what are the possible outcomes based upon what I'm seeing For instance, the talk earlier by John that talked about the false positives. He gave an example of where a false positive was detecting bad activity that was

designed to look for, but it was considered false positive because they didn't use the vulnerable software. But that's one type of false positive. In that case, have one classification that really can be bucketed into different types of classifications. So trying to support that type of analysis, trying to be able to define what your conclusions and outcomes are likely to be, and then the actions to be taken from those is very important. So from the cyber intelligence programs components, I put a cyber intelligence programs components, not the cyber intelligence programs components. This isn't verbatim, even what we use, although this is a model that very closely approximates it at GE. This is one that I'd recommend,

feel free to steal this or whatever, use it, build from it. I basically break it up into really like three stages, four stages, however you want to call it. Collection is basically getting all the information, that's when you receive it. A lot of people joke about feeds. So what feeds you get from versus what feeds you don't. That's your collection plan as well as other information sources. Some of you might be subscribed to mailing lists. For instance, that all fits into collection. You're not subscribed to, nobody in this room is subscribed to every security list, right? If you are, feel free to raise your hand. I will call you a liar, but feel free to raise your hand because your inbox is not gonna handle that. You made some

conscious choice as to what is in scope and what is not in scope. That's part of the collection. Processing, how do you turn the fact that there are 200 teams out there on the internet writing reports, writing spreadsheets, writing JSON dumps, putting together zip files full of things. How do you convert that all into a kind of common object model that works for your team that multiple people within your company, within your security team can actually leverage for arbitrary purposes that apply to them.

Knowledge management, and that's basically after you've, what I call, normalized the intelligence, after you've normalized it to a picture that makes sense for your team, then you store it in basically a big knowledge management system. Do I have a question? No.

So once it's in the knowledge management system, that's when you get to the retrieval stage where it's available for everyone to retrieve it, everyone to use it. That's where you might deploy it to your proxies. That's where you might want to start pulling it out and do deeper analysis on it. That's where you might want to start pulling some of it out even further and putting together reports that go to your senior leadership that tell them who's been phishing you or who you think has been phishing you this week based upon what evidence. Or even better, who at your company has been receiving phish. Things like that. And that's what I call your consumers. Pretty

much everyone outside, usually divide it up into consumers and then the detect prevent event analysis incident response teams so you have your internal customers and then you have your intel your intelligence reporting consumers is everyone with me so far or does anyone have any questions yes

so the question was about How much time of my day do I spend on any one of these? Do you have an answer for me, Krista? We spend probably, most of our time is spent doing collection and processing, and it all happens at one. I would say that that's probably between 16 and 17 percent, another 20 percent is analysis, and the smallest due to reporting.

Yeah, so about two-thirds of the time, Krista said, is spent on the top two buckets. about 20% of the time is that right was spent on

yeah yep one of the interesting key points that was brought up and I love that John's talk went before mine because I didn't cover this model in my in my presentation at all so it lines up nicely John gave a really good description what I call your analyst bandwidth right it's basically what Once I've managed to make my alerting, my alert response more efficient through tuning or through other means, then I've created more space to basically cast a wider net. We like to look at our intelligence program through the same sort of lens where we like to try and start collecting what we consider to be the prioritized set of intelligence, try and get this entire process to be really efficient for that,

then we want to, once we make that time spent a lot less, then we have more time to broaden the net and collect from more sources. And that's the only time we recollect from more sources. So we just think of what sources we'd like to collect from next, or possibly what analysis and reporting we'd like to generate next. And the key thing about that, and you'll notice I got the two arrows that go out of those buckets and back into the system. I will definitely dive deeper on those later, but we consider our own internal team supplying more intelligence to our collection plan as well so we collect from ourselves for lack of a better term and we gain a huge amount of both similarity or simplification

we gain a huge amount of simplification out of that and we also get to basically eat our own dog food if something is broken on this left hand side of the chart here we're gonna learn it and we're gonna figure out what was done by the Intel the Intel producers to cause it to happen so we're not necessarily running two parallel processes where we get a whole bunch of complaints every time we process a fire-eye blog that we never see elsewhere we'll actually get to see that and so we try and consolidate this operation as much as possible and I think that is trying to identify consolidation opportunities is one of the biggest like efficiency wins you can get so in the end the goal is

with trying to step up that process of automating and improving the fidelity of intelligence that we're collecting, the goal is collect all the things. And I believe somebody mentioned earlier that that was going to be my tagline, so I very hastily grabbed this and threw it in the slide. So as you can see, I put the pro in procrastination. What? Not a cat meme. I promise you. So getting back to that chart, this talks about that first slide. excuse me, it talks about that first box that was listed up there. So the collection. And I break this up into collection. So the act of actually getting it from not your network and bringing it into your network. And also collection plan. And collection plan is

what I call defining not only what you will be bringing into your network, but also what aspects of those reports you care about. So number one, you select what PDFs or what CSVs or what JSON documents or what malware samples. going to collect because you can't download all of them out over the internet that's like saying you're going to collect you know all the PHP documents off the internet or whatever right just like EXEs so

your collection plan is what you use to try and filter that information down and I put in some good example like kind of thought processes that might go into this snort and that's a bit that statements a bit dated do you snort or suricata because I've heard that suricata also supports the same or similar language if you do not if you not use either of those in your network if you have no way to store a snort rule in anything then does it really make sense to have a collection plan which includes gathering a whole bunch of snort signatures and I would say no it does not maybe if you have infinite time or infinite resources it does eventually but going back to the you want to limit

it down to what is most critical, what is most applicable, what is most actionable, doesn't make sense. So you take that off of your collection plan until, for instance, your infrastructure team builds sensors that use that technology. And then I also grabbed a, and I'll save this for the end, if I have time I actually can do a little bit of live demo, live examples, I have them up here on my web browser, but I'll give a shout out to Liam Randall who's one of the local InfoSec community as well who's the owner of CriticalStack. They provide an Intel framework for Bro and they provide both a command line and a graphical web-based user interface for managing Intel and consuming, collecting Intel

in the form of Bro's Intel framework. This will all be shared out and those links are actually hyperlinked. So you'll be able to get this stuff off of the website or at least off of our GitHub.

So, processing, what I call processing is basically using the industrial terminology or using the manufacturing terminology. So the idea is that you get one thing, you get something that's raw, and then you process it into something that fits more what you want, what you can actually use. So in this case, movement of data or material towards a known goal, result by passing it through a series of stages or a sequence of actions so in the end convert our raw material which is the Intel I'm getting from 20 sources that looks 20 different ways that's in a whole bunch of different formats and even when it's all when a bunch of it's in PDF the PDFs are formatted differently or may use different character sets or I'm

sure all of you are familiar with the variance that's you know that exists in you know consuming Intel reports that type of thing through processes to normalize the information so that then when we store it into our knowledge management system, and I'll just do spoiler alert, our knowledge management system is primarily crits and then for unstructured content we actually use a highly modified media wiki instance. And I bring those up for two reasons. Number one, because I any questions you can actually ask me about those but number two they're actually like really basic open source tools that you can build a very effective program off of you don't necessarily need to run out and buy a four million dollar you know security management platform

if you want to be effective so what I did here was I gave a I gave like a kind of sequence of steps that someone may use to process information so this might be one version of a process we actually have different processes defined according to what the input intelligence is the goal being that the outputs end up being highly consistent and highly predictable in the end so in this case you might get a PDF report right PDF report documents a in this case I'm actually gonna just I'll just do this do this one live so I actually happen to have a FireEye report of some sort over here. So shout out to Mike Reeves who will be speaking later. FireEye blog.

So they gave this operation Russian doll Adobe and Windows zero day exploits likely leverage by APT 28. So I'm not gonna get into description about all that. What I will say is this is an example of an intelligence report we get. They go and talk about which Metasploit module was used so they give you the source code that the attackers may have used for the attack so you can go and you can try and build attacks yourself to collect more information out of that would be say the analysis stage they also go through here and they have some very specific string data like this video data rate stuff like that so they tell you what to look for that you know that's creating it they tell you what

versions of Adobe Flash Player so all these things actionable pieces of intelligence that you would excerpt from this document and then you you know base you know base the information base the metadata that it's leveraged in for according to what the narrative is so in this case this is talking about the buffer overflow vulnerability talks about the exploit code here so getting to the to the kill chain phase or the kill chain distribution that was brought up by Harlan Carvey earlier, there are many different stages there where you can break the attack up into. One of those stages is called the exploit phase. And that's where you break the target software to try and misuse it to get it to run, in this case, arbitrary

code. So what you have right here is a whole bunch of information about what the attacker would be throwing into your network or onto that system or whatever at that phase in the kill chain. you store this information in here you know not just what to look for but you can actually narrow down when to look for it or where to look for it you're not necessarily going to look for this stuff in a you know an SQL attempt or something like that you're gonna look for this stuff where it makes sense to look for flash documents for instance if you can watch all users traffic that's transferred over the network that's downloaded from the internet you might only look for the

that they describe here in the case where it is a downloaded shockwave flash and it is being downloaded from the internet so that's what I mean by processing the intelligence is extracting those important pieces based upon what you can look for so let me get back to this so they say read the narrative document relationships between significant data points contextual clues. Those are examples of contextual clues. I also gave some other ones here which might lead to intent. So actions on objectives being the end goal of the kill chain. I believe stealing all your data was described as the action on objective. But actually a lot of attackers have, there are a lot of different objectives that are desired. So I gave a

few examples here. Is it used in data theft like brought up earlier today? Maybe just used in ransom. Maybe they want to steal information or say do like CryptoLocker thing where they want to encrypt your data public key encryption so that they only have the private key to decrypt it, and they want you to pay them to decrypt your own data for you because you feel that those wedding photos are really important or something like that. Or is it being leveraged for a DDoS attack? All those things, having that associated with what IP address is being used or what exploits are being used, that type of stuff, can be very beneficial in controlling what gets

deployed to what tool so that, for instance, you don't run into the problem of blowing up the sensors, which I think was also another example of a fail state that was brought up earlier. So the other thing, too, that I'll add, and there was a piece of unintuitive wisdom that I just kind of came upon as this last point down here, which is the spreadsheet CSV as

data management formats can actually be surprisingly effective and efficient. You wouldn't think it. You'd think that, oh, I need to put together a web form so they enter in all the fields. What I found is that spreadsheets are probably one of the most efficient, dense, or the densest data management layouts known to humankind at this point so far. So we've found, don't try to be overzealous in automating things don't try to automate things that are easy if they only take up say like a half hour every two days of the week or something like that for one person to get through like 20 reports or something maybe you have the most efficient thing there and this is where it kind

of gets back to it's really good to record data and it's really good to do to use data to drive decision making and reach conclusions based on data such as conclusions as to what should you automate next rather than necessarily just trying to always automate the easy things. So that's what I added and also just as a reminder don't think that you are a, you know don't fall into the trap of feeling like you're a immature security organization if you do use a lot of CSVs, you do use a lot of Excel to move around to do data management. There's a reason why like a lot of the database management systems pretty much use that universally for a lot of the table representations.

So knowledge management, so this is that, you know, the disk shaped box that was in the previous one, so it's what I call your program's intelligence in its final form. This is what you're trying to coalesce all the information to be. This is going to be typically the representation that most of your intelligence customers are gonna use, and that includes your detection tool, your responders or event analysts who are trying to respond to incidents trying to navigate the alerts to figure out why they care about something and what they should do next and what else they should be looking for it's all gonna be stored in the system and it's important for them to not have to in the middle of running down why commands are

being remotely executed on a box try and figure out I need to interpret what given to me differently if it came from a FireEye report versus coming from the RSA shell crew paper or something like that. So your team's job from the intel perspective is to try and make sure that no matter what the source of the information was, to the best of your ability, or I should say within reason, the information provided to the event analyst that actually causes them to take different actions is reported to them in a very consistent and predictable manner. call normalization so the other thing I'll add is once you have organized it it actually becomes very much easier to reorganize and I say this because I don't

I have seen so many times where people have fallen into the trap of what I call analysis paralysis trying to come up with the perfect database design the perfect data model to represent the attacks and never actually delivering anything year after year and it It can drive me crazy sometimes to see it talked about, say on the internet and so forth. Just come up with one way to do it. It doesn't have to be comprehensive, doesn't have to be complete, it just has to be something that you can start doing as soon as possible. You can start giving to your analysts. Because you, typically the intelligence team, is not necessarily the team that is having to suffer from the consequences of your good or bad analysis.

That's typically the event analysts. as you get something usable by them they can tell you what's broken about it and you don't have to sit around a huge table debating about what may or may not be the best next step they'll tell you what's broken and what's causing them the most pain that's your next step to solve so and actually that pretty much it hits us like last two points there you know basically minimize pivots external information by event analysts try and them to be the ones who are telling you how to do your workflow how to build your data organization what what needs to be reported to them and in what format and then finally you know what I also like to say is if you build it

in a database where everything's queryable then you can actually generate signature deployment based upon static queries where the information is fed into the database in such a way so that every time you run that query, you get a new picture of the data. You get a new picture of what right now looks like in the threat landscape. So here's just a quick pitch on crits. Freely available on GitHub. It's actually a project that's put together by MITRE. Apologize, Ashland, I didn't actually get your, okay, ahead of time to throw you in here. But I know that Nate and have both been talking on their GitHub issues and everything as well as others at GE. So we're starting, you know, it's a big community driven platform.

It's becoming very popular. It's only been open sourced for maybe a year, maybe a year and a half, something like that. And it's definitely a good first step. It's not perfect. We actually have a heavily modified, so heavily locally modified crits version that we use internally. And I'm sure we're not the only ones. Again, it's modeling those tools. So the benefit to this is I have a tool that's open source that I can mold to fit my attack surface, which doesn't look like any of your attack surface at all. It's, you know, organically grown to what it is today based upon our business needs. Because again, we're not first and foremost a security company, we're

an industrial company. And we, and a financial company and a lot of other things. So. the end of that sentence were something different right and that's kind of the joke so this is a really good something to build from this is what I would recommend my personal advice is your next step if you don't already have something for this

so intelligence analysis so this was on the right hand side of that graph I had two boxes one was analysis and the other was reporting. And the only real difference was that reporting went to consumers and analysis just feeds back into your program. So analysis is when you are researching your pile of intelligence, which you've made a lot more accessible by trying to store it in a system like that, so that you can create intelligence that can be further leveraged by your program. And just some examples, because it's kind of like a really nebulous, kind of high level description.

I've got a whole bunch of network indicators say IP addresses or something like that so I want to dump a whole bunch of those out to a document and then I want to write a script that goes through that document and tells me you know can you can you tell me which of these were associated with core cow malware some of you feel free to search for it on the internet very fun stuff it's a really good malware you know family to investigate tell me

subnets, so what class C networks maybe, or what class B networks, have five or more distinct IP addresses within them that have also been used for command and control in CoreCOW networks. So that requires you to first collect all the documents you can about CoreCOW, dump them into your system, run them through processing, then you'll have a bunch of raw IPs. Then you can learn more about it from that huge intelligence mountain to say, well, I've determined for whatever reason, my conclusion is that if a subnet contains five or more distinct IP addresses for core calc and CNC, then I'm just gonna blacklist that entire subnet or I want to alert on that entire subnet or something like that. Maybe that's a bad strategy, I

don't know, but just hypothetically, that's kind of what I mean by analysis. So I'm gonna take those subnets now and themselves are gonna be say indicators or signatures or something like that, that I can go back into my Intel database. And now I don't just have a list of IPs, I also have a list of just low reputation subnets as well that are in there. So I've just fed my system back with more intelligence. Another example, just to use that report that I was up earlier, gather domains associated with APT28 campaign. So there's like 10 domain names on there, right? What do you need to set up a domain name? to fill out the form

to buy the domain with your own information, your own contact info and usually your own credit card and all that. Contact info is retrievable via public database, the WHOIS database. So you can go and mine that to determine what WHOIS terms are common across those domains, that type of thing. So what I typically do is I'll actually have this, the and the analysis is published in a report that's then just fed back into the system again eat your own dog food or run this through your same process don't create some parallel process where the authors of this document or I should say the researchers of this work may not even be documenting it in one central location and then are responsible for following a completely different process

to get it into crits then say if you got it off the APT 28 report yourself so and then in terms Experience reporting is basically that, but it's the explain it like I'm five version of that. So the idea is your audience now is no longer your security operations team, your audience is actually say your executives or say the public. So if you're a commercial security firm or someone like that, if you're FireEye, amount of the FireEye report actually goes into the background. So it sources information from other reports about APT28 to try and frame the story ahead of time. Because the audience of that report may not already be familiar with all of the prior reporting from FireEye and others

on that particular threat group. However, FireEye's internal team probably is very much familiar with that. And again, what we like to do run into that type of stuff when we have to publish reports is we actually will document them just like analysis reports and it gives a lot of us to do peer review so all of us can throw different our own different edits in there our own different changes that we feel might be better wording for that type of high-level audience but it also causes forces us to follow the exact same processing the exact same collection and processing steps which on our side one of the one of the questions I didn't predict getting I wish I would

have known I would have gotten it is this last one which is why have I never heard about this? Why was this never published to the company? Why did this never go out? And the answer was it did. It's just one of the million internal spams that you get at your company. So being able to say, oh no this went out. And I can give you exactly what it looked like it went out and everything like that by just asking crits to tell me where my version of the report was published based upon how it overlaps with the indicators that say caused the question, this last question to come up in the first place.

That ends up being very, very helpful because then it helps you customize your reporting process and your reporting formats based upon

upon excuse me helps you customize reporting formats based upon feedback from your leadership which again you can't necessarily predict what's going to be useful to them and a lot of times what you're asked what they're asking for may not necessarily be what they are actually need and I think that was also another comment that was brought up at least twice earlier in some of the earlier presentations so in deployment and I'm gonna rush through this because I actually covered a lot of this earlier but this is basically deploying intelligence directly to tools so putting it in a database you can create static queries that basically will select for certain intelligence to go into places where it belongs so your

intelligence team can make conscious decisions when the processing intelligence to say, well, this is an IP address. But this report is talking about all these IP addresses as being DDoS IP addresses. So if they go into the database with DDoS metadata on them, then any tool that actually cares to make a decision of whether to deploy or not deploy can actually use that data point to filter those out of their deployment set. So my favorite is this one. Process list of IP addresses from known DDoS attacks. on your squeal console from Snort or Suricata might not be the best approach on your perimeter. There may actually be a better way to manage that data such as if you have Bro deployed as well and

it's logging metadata to a central system like Splunk was brought up earlier. If you can't afford Splunk, a competitor to it, I'm not sure how good it is, but the goal is to make it better is called Elasticsearch. have right here worth looking into it's actually very easy to integrate with basically maybe you want to deploy those things to elastic search queries instead so you can search that data set for all the DDoS stuff and manage it differently than managing say your command and control traffic so micro is this for you so

and then here's some examples that I figured I'd throw up here. These are links again, slides will be published. And then finally, thanks to everyone for coming out. I know we had like over 100 attendees this year, that's really great. And then I figured I'd put some links up here. I gave a shout out to UC's cyber program, but also the Security 513 crew who helped organize this FIND event here. And I'm very happy to be a member of that. besides since he paid. So any questions? Because I got like six minutes left or comments or jeers.

All right, yeah.

So from our perspective, ones that have known to have targeted us in the past, like just whether they have targeted us in the past versus not, so whether we've been attacked by that group or not, we generally consider the group that has attacked us before to be a greater threat because they are the ones who we feel likely to have any information about our network that they might be able to leverage for future, for learning for future attacks. So they're more likely to be the ones that come up with the next sophisticated attack against us because they've learned from failure. Because they learn from failure just like I do and I fail a lot. And so do they. So there's that and then we can further break it down

from what was being targeted. Were they just trying to DDoS a web server versus it appeared that they were trying to strategically fish a specific group of people that are responsible for a specific defense program that has specifically been in the news in the past two months or something like that. So we'll select based on what their targeting is, whether they've targeted us. We also have some relationships with some others in the industry, both in the aviation and the defense industry, where we can do some information sharing. generally what our peers are seeing, so we may partner with people on that type of thing. That information is also information that, I should say those groups are also considered to be higher priority threats than others.

Question?

Yeah, that's a great question. One of the ones that I may do, or I should say, one of the approaches that we do take is we will try and record in the database what tools it was deployed to, and we'll give our event analysts, and some of them I think are here and might be able to talk to you some more about some of the mechanics and decision making behind it. We give our event analysts that if they, for instance, determine something as a false positive, so if they decide that they're gonna close out an alert because they don't think it's warranted to look into it, even feel that we shouldn't be alerting on that

indicator, they'll actually disable that by changing a field in CRITs. We can go back later on and then measure how many of those are turned off in various tools in CRITs. So if we have 10 detection tools that we can deploy an indicator to and it's turned off for five of them and it's left on for five others, we can actually across all of our consumption and since we're relating those indicators not only to what tool they're deployed to but also what information source they came from, we can rack and stack our sources that way. And then we can even choose to take a source that is a common, useless source of information out of our collection plan. Yeah? Yeah.

That's kind of a hard one because we found some groups do like to camp on infrastructure for long periods of time. Generally I do consider that when we're responding to something and it says this was a very significant indicator of attack two years ago and it comes up again, it becomes recent, current again and we're looking into it, usually the decision is made at that point in time to try and discount a lot of what our base assumptions are. So if it says that it's being alerted because it's related to a particular campaign, maybe we don't initially assume that it is actually activity from that campaign. So that's kind of it. A lot of the tools that we've built don't necessarily suffer from,

or I should say aging isn't the best way to solve that problem. We actually use scoping and our reputation to do that instead, and that ends up getting the job for us a lot better.

so feel free to talk to me later and there's like contact info and stuff on these slots. All right, thank you everyone.

All right, how did it, did it make sense?

that I fall into the trap of talking to myself. Being my own audience. All right. How did it... Did I make sense?

Yeah, I'm sure they were all shocked by that.

Adrian, you getting good signal from me?

There it was.

Hear me? Anybody? Good? Gravy? Alright.

Yeah, throw popcorn at me. Alright, so we ran a little over on the last one until we get going here. I want to introduce Justin Holley, he's one of our organizers and he's going to be talking to you guys about response for any infrastructure. Hi. Hi, everybody. Hello. Everybody hear me all right? Good? Good? All's well? Awesome. Thank you so much for coming to B-Sides. We're very happy to have you. Hopefully, y'all are enjoying the day and the venue and this wonderful Cincinnati weather and yada, yada, yada. Yeah, it's been really exciting so far. I've been excited to kind of give this talk for a little bit. I work for CBTS, Cincinnati Bell. I've been there for a while. Been a consultant for companies

large and small. Did a few years at GE with some of this crew. If you care about SANS certs, I've got some SANS certs. I've done some SANS teaching. I might reference some SANS stuff a little bit here and there in this talk. So that might do something for you, might not. We'll see. I'm gonna tell you a story. In 2013, a guy come join CBTS as a vice president and start a new security team. And when he did, I went to go work for him. And the first thing he told me was, Justin, I want you to build me a network from scratch. And I hadn't been a sysadmin or a network admin since the early

2000s. And technology had changed quite a bit since then. I was like, all right, cool. What are you doing? Everything's all virtual and I didn't know anything about any of it. It was wonderful. There was like new Cisco switches that I'm like, what does this thing do? I'm used to this old catalyst like this ancient. I was way out of my league. What he told me was, I want you to build me a network like the classified network I had at my old company so that we can store all of the intellectual property and the code that we create and all the products. And also, I want my guys to work in it. You want

your people to work in a classified network. Do you know what a classified network looks like? How it operates? How it's restricted? Yeah, well find a way for them to work in it. And we kind of went back and forth a little bit on requirements. Ultimately we decided that this network was to be purpose-built to be resilient to attack because the types of adversaries that we were going to be investigating and uncovering things about that would target us we knew they would target us We would need to be able to be resilient to those types of attacks those tactics that they use and And so it was a very interesting Challenge and I want you to sit back and think if you

were to ask to build a network and I'm talking green it was Greenfield it was you know that literally there was nothing there there was an empty spot in a data center and spot in the data center with gear and stuff and make it a network and make it do this thing and y