Mike Reeves - Taking a Distributed Computing Approach to Network Detection with Bro and the cloud

[Music]

all right well welcome back so we had uh we had a great presentation uh that some of you may have missed in the first time block this morning and our next presenter is a good friend of mine Mike G known for many many years I'm a big fan of Mike and the work that he does uh so years ago uh only has question first so how many security onion users do we have in the crow good number of folks awesome you guys are awesome uh so you're awesome oh stop

it uh so for for folks using security onions so anybody running onion salt couple of folks I know so uh the guy who wrote onion salt this guy right here Mike G uh so Mike wrote onion salt I don't want to see entire talk but out of his experience running all of the massive NSM sensors for General Electric we're talking about an organization with 500 sensors and like five pyes of full packet capture just crazy numbers like that right so this is a guy who knows his infrastructure he knows how to make things scale uh very good friend of mine so please join me in welcoming Mr Mike Breeze

so my name is mikee G uh L balcony I've been do security for a while um I Ro fire eye I love Road and uh I like dou I have a lot of experience in large sensor appointments and high manual stuff um my current largest customer has I think 1,500 sensors deployed with uh probably going to be around 7 pedabytes of data so um G is kind of smaller now I thought I had problems with 500 sensors and it's nothing right so I'm big time into RC stuff I'm really into um to drone racing right now it's a picture of one of my mini quads that I have I have a couple um it's very intense it's a very

fun hopping um hopefully the people that keep flying DRS in the buildings won't ruin the rest of us but it's good time like dou said I'm a security so we have any Mr Robot bands in here Rich Man Okay so we're going to talk about a company we're going to call it eor [Music] and this company eorp um has an interesting story this is this is fictional well some um so but before we get into this we need to talk about the Pyramid of pain so has everybody seen the Pyramid of pain for I bian dollar for showing the slide um so the Pyramid of pain talks about how difficult it is to change T we're going to talk a lot

about TVP we're going to talk about how um how the adversary Chang changes his ttps when you you know when you change your TTP so you're changing ways to look for them they're changing ways to own you right so um so keep this just in the back of your mind as we as we talk about it so let's so let's talk about uh this DT so at this DT we had a lot of um threat groups you want to call them that um interested in what we did so when we first started looking we said you know what let's take a look at the alpha let's look let's go after that C2 right because that to um you know

they're sitting at their desk in some foreign country probably and they're just clicking buttons because they've got C2 they don't need to do anything special um said oh I want this file and grab so we went after out we we really pushed and pushed and got this and when we did that we uh we were we got to see a lot of great stuff right we've got to see um different ways the thread actors were getting into our environment um a lot of the things they were doing so we got an idea of their ttps right so we started building that pyramid we've got lots of information so we used that and we we uh we we sort of knocked them out

right so we we made it harder for them right you're not going to stop these people your your goals to make it harder for them in for it to be more expensive and expend more resources so then they said you know what we see we see what you did there you know we see what you did there with the internet so we're going to go ahead we're going to attack your VPN so then they went after VPN right and there's the mult mud of ways they went after that but you know so what we did we put number of sensors in front on the bpn and we were able to shut them out so they said okay all right I see that so

what we're going to do is we know you guys are CED up a bunch of different eort businesses right so at e Court they um you know some businesses had things they wanted but they would go after other businesses that didn't really have that kind of stuff so they would go after them and they would say you know what I know these got Security in the back but they're all connected right so if I go where the security not as strong I so we went really hardcore after movement well then they're like okay you guys catching us right you're catching us quickly so then they're like third party so it got to the point where um

eor would do a a um PR we're going to buy this company right we we got really good at doing Network detection but we're going to buy this company so what they would do is they WR the press release right and they would go and own that company and they would sit there for 16 18 24 months doing and just they would own and just wait because they knew once it was integrated that they would get up to the network right so as soon as that I mean it was like the second that connection gets open a beacon goes out and says hey we can see what we want and then they got work right so um you know it was

very it was very Advanced from that perspective right so when you you talk about persistence you know they had a game plan that they were ready to execute as soon as that be came back and as soon as it did they were in it and then what we had was um the DMC right so then they're like okay well we know you're catching on now we're just going to start um owning your your external infrastructure this is a little bit more difficult this takes zero days this is you know this is causing the change in add new tools to change the tools um does anybody have cusion servers in their environment so cold fusion is if you don't know that

and you put up sensors and you find them um they will know where your cold fusion servers are for you so that's you know a so one of the things I want to talk about is they know your network better than you right so um you know we hear a lot of things from hostb folks that say you know hey um we got a sweet agent put it on there they know the machines that that sweet agent isn't on right they know that some guy in um some building said you know what the internet's kind of slow I'm just going to get a DSL line put in here and you know I want to go

get it from my desk so I'm just going to plug into the right so but but they find it right so a lot of the Rogue infrastructure come from our buddies finding said Rogue infrastructure for us so um the other thing they know and this is you know I've talked about this a lot lately is they know the typical value proposition for your network they know I got to go get budget I've got to do all these things right and they know that what makes the most sense for me to put protection is in my main data center right but there's a problem that with that right um you still have critical data at your Edge right people do work

you know what I mean like that's where your data is not maybe not all of it have sweet B thing that put all your goodies in and but at some time that's got to come out that they have to be able to do their job you can't have an engineer already K into a box doing crazy AutoCAD stuff right they got to be doing it live and they're going to hit you where you're weak is so again think about that value Pro when you you have your company you say okay well we're going to H and smoke everything we'll put sensors in here and you know we you we'll put this huge box in and we'll we'll catch

everything but the reason we want to do that with protection is because it's a so packets for the most part they don't lie right so you get things you um m you're dead you get um you know ideas of connections you see activity after that in so if you have like an a alert or Hood Space Alert um you know a lot of times if they do something to that host space tool then you're not going to see that information um it's also a quick way to get something some sort of visibility quickly right so if you want to roll something out to desktops let's say you have I don't know 500,000 desktops it's a little bit of mature to get something

out there quickly to 500,000 desktops so by uh by putting Network detection you get something quick and it's not as interruptive as saying okay we're going to throw this down on machines we're going to run some scan that's going to crush everybody's machine while it's running you don't have to do anything can get some visibility you're not going to catch everything right it's really important to still do the host space stuff it's really important to do all the different things right um to detect all the things uh so you know is very important right um but but doing this you know gets this and again what we talked about hostb stuff hostb stuff is great on the stuff that you know that

you own that if you think that you have this host based up on every one of your machines you don't because they know where they are I've heard it a million times oh we're all patched we got everything you know and then somebody's machine they bought but there are problems with network detection and one being a INR is routing so does anybody know why a ining routed is bad you're not allow to answer right so you really only see half of the conversation right so to do anything effective when it comes to snort or torot or bro is you need to see both sides of the conversation um encryption is pretty obvious um and then the bad guys they

know your network and they know where your detection is right so when we go back to that eor example we would put in we would change our ttps right we top here and we would change it we say okay we we got you here and so what they had to do is they had to reformulate a new plan change their tools right because how you attack a is different then how you're attacking um you know doing a fishing campaign um so you know it's it's you know the they they know your network they they know what's going on um and if you don't think they do they it it'll be surprised when if you get some Advanced

adversaries and you see what they start doing NLS is another problem um wi optimization is is a problem we'll get into these a little bit deeper and this one's really important so the Network detection the evolution of networking are in direct conflict so what that means is there's BR device there's uh all this Cloud stuff right I've got this Cloud machine I got all these different types of things um the you know the the regional data center kind of thing where I I hit the NS network instead of going all the way from California to New York to go out the internet I just go out the nearest AT&T or Sprint pop in the NLS

Cloud right and I had some slides on that and it was really cool and then I don't know what happened to so um HD rout so this is BP for the most part right so if you have um a uh two large JS in close proximity and they each have really big internet types you're going to see um AC tring the only real way to to do the put that together is to is to use some sort of um load balancing device you gave on to transport it across there to get that uh no asynchronous you're going to run into this no matter what if you have a large Network you've got two data centers the

whole point of HP is to you know to to remove some of that ingestion and things like that um but you should be doing it to INE performance on an eff or reducing cost right let's say you have two internet connections you have one to let's say AT&T and then You' got one to coach right and your Co was really cheap and your AT&T one is really expensive so what you do is you use bgp weting you wait that coaching connection to be heavier right so you want you want everything to go out or come in for your as on that well the challenge there is to bop on the spr at& so if you're on the AT&T network and

you go to our site you're going to you're going to come across AT&T connection but if you're on Comcast there's a chance that if you come in on that AT&T that hey it doesn't really matter so I'm going to use waiting I'll go out the Cent or I might come in the Cent and it it can change but it's important um from how the internet works it sucks though when you're trying to do detection so how good budy a Chan um I won't spend to some on this slide but there's not much you can do um there's a lot of things so what we would do is a lot of times we would do uh um we would

pre- terminate the the encryption like on F5s and then we put our sensors behind the F5s and then we have our web server on right so there are some things you can do there's some man in the middle stuff um lots of people probably work at large companies they you're probably being in in the middle um so whenever you uh go to your bank account or things like that they're seeing that stuff um so let's talk about NS team so NLS wants to use use the right we still BM about so the um the challenge with um with APS detection is let's say I have let's say site one in this example is California and site two is Arizona and

site three is my headquarters in New York so what n is supposed to do is say you come out and say oh well you know what it's I'm in California and it's faster for me just to go directly to site too because you know hey that's where you know that's the closest path but what we do is we say okay well we need to get everything coming centrally to our G so we can put this big sensor on there and monitor everything going on so now we Crea this laty so to get the site 2 from site one you have to go through like three well that's great from a modern perspective but from a network

performance perspective s right because that's what the network team gets me measured on they've got some app that requires low leny but you have your detection requirements and you got to see what you got to see so that's a problem for what we do um way op compensation does does anybody know why um we optimization makes your your traffic or messes with your um ability to monitor

traffic does all kinds of dat duplication you have traffic that disappear right exactly so so what happens in in we optimization let's say you have 100 Gig file that bunch of engineering want or something and you want to copy that from from New York to California so what happens is that first time it gets copied every bit of that data goes across there um the next time they go so then guy I'm sitting next to the guy down I'm download that to you well you click it and all that stuff is in the sort of the cash that's in both sides and then it sends a few packs and says hey I've recognized this guess what

you don't have to send all that traffic Pap it'll send some stuff but it's not the whole thing well if I determined later on that there's some malicious stuff in that 100 Big File you know we we analyzed said this is bad we want to know anytime we see this file transferred across there the problem is my my monitoring at my at my Gateway doesn't see that traffic ever again because it's been optimized so what you have to do to put sensors on the unoptimized side so that's even more detected right now I got this big whizbang sensor on my my Gateway and then I've got this way up farm so I got to put sensors over there

so it starts getting really um complicated so I've been telling you uh why the stuff why you know why detection is hard so what what we need to do is sort of um sort of flip this on its head so to speak so I'm going to talk about whiz Banger what I call whiz Banger is we've got this tool we've got this something or other that you can put in and everything's going to be awesome right well we talked about Showp points before that how they're you know again I want to go from California to Arizona I don't want to come through anyork but the problem is I'm I do that um I have indicators on my sensors right you talk

about Intel Frameworks all these kinds of things if that information is good enough for my sensors why is it good enough for everything else that I have right and flexibility the reason I got young here and for for the young here that's a like an old clation type carto that he could stretch so um but anyway so that's when I when I design um security or monor people it's all about flexibility you don't know what the bad guys going to do tomorrow I don't know what the bad guys going do Tom if I did I'd be sitting in Silicon Val my mansion and just you know take them back BR Mar or something right so nobody knows

what's going to happen so you have to have a change on a DI you have to be able to say they're doing this we're losing money I know a way to detect them I need to get it out there now and you know that's very very important so there's no shiny CR so all the M here there's no shiny CR T there's nothing out there that's going to stop everything or so how do we fix this right we need to go where the users are right cuz that's what the bad guys do I'm not going to sit there you know I mean if you look at a lot of the attacks they fishing attacks they're attacks on

people and those people have data you know think of it from a you know a lot of people talk about I saw 800 terabytes fly out of my environment so I know something bad happened well think about it from a sales guy perspective let's say you're a sales guy and or a saleswoman and you're you're working on a on a quote or some let's say I don't know some Water Reactor kind of thing just something something horribly expensive right so some country wants to buy this from you they're putting out bids well in in America we have corporate companies that do this they say hey we will build this whizbang crazy thing for you and it'll

cost this much and then we have other countries that are this this companies are nationally sponsored they're they're nation state companies so they use their nation state resources to attack let's say Bob Smith who's the sales guy right and now they have Bob Smith who's running the show right they they went to the website found out who Bob Smith was that he was the sales guy right and they went and they say you know I want to see what they're offering so I can go under that right and that's a word about or an Excel spreadsheet right so that's it's not something that's obvious but there's real you know tangle dollars associated with that thing you lost that deal

because of that state or maybe corporate Espionage or something like that said you know what I want to know what they're doing um so what this means is we have to distribute our detection right we've got to go closer to the users the the days of we put everything all in one spot we put this 400 gig box that can do all these things there we need to spread out so um we're going to steal some stuff that we do with this Distributing Computing so introducing the double decker C so um use sensors for the resources right so think low power devices think um you know think a lot of things pool is much detection you should

never have you should never need ever again an atomic indicator on your sensor your sensor should be generating the data because you want to centralize your data not your sensors right so you you say okay I have my horsepower in my data center I can easily does anybody ever try to get a server into Brazil well you guys don't count so it takes months to get a certain so let's say you have a site Brazil um I'm sure it has arm guards because that's just how they do it down there and um you need some more detection right they upgrade their bandwidth or something going on you need to get some it's going to take you two

or three months to get stuff down there if I want more resources in my data center I call somebody up or you're Us in the cloud and C some buttons I need to have those resources so um and these things also need to be Expendable we'll talk about that a little bit but you should be able to run on whatever you have so in in the case of say Brazil you have an extra workstation down there right maybe they don't have a big pipe you should be able to use that in a pinch right you should be like I need to use this right you're not going to get a vendor solution there quickly um in Brazil and it should be

man managed as a single device why why do I have to treat these things um like complicated devices and this is when we talk in scal this is how um we're able to scale to 1500 sensors and Beyond so again we want to do this by getting our sensors done and we do that by taking as much off possible as much off you still have to do your deack inspection on there so you're going to have some of that stuff right there but again IP indicators um you domain names all these kind are very expensive rules on a sensor I know that the IP reputation stuff has come out and they say oh it's a lot better but when

you have a raspberry pie right that those things change from the standpoint of what you're trying to send so um and again what we talked about before is that if I know ip12 that what I want to see that in my fire laws I want to see that in my mail laws I want to see you know I want to see that just not there so I can pull that back right um and they're there to provide the data right and we you know use power devices so how does this look when you want to do this on scale so I prefer to use Sal there's puppet there's Chef there's anel there's all kinds of different uh tools

out there to do these things um all Source um the way I've chosen to deploy it is uh putting all the code in GitHub right so when we manage our sensors we manage them out of GitHub we don't we don't necessarily have to log into these devices to do anything and the good part about GitHub is you have your PCI you have your chain control if somebody rolls out something to a sensor and break something you know what that thing was and you know who to troll right so those are the you know the importance of having a comp then you have a master and this is where the scalability happens so you have these minion Masters so I can

have a thousand minion Masters talking to a thous minions each because that's how it scales it scales completely horizontally um and this actually works right so like I said I'm doing about, 1500 sensors right now with using that but they're there to do work for their master so the minions um you can devel a time in which they check in so they check in they check for new work so do I have new roow policy I need to do oh there's a new package I need to update oh you know that's what it does you can say you know say every 15 minutes check in and do this and this allows you to have a

single configuration for thousand prices so this is just a um a video of how a sensor gets built so on the right side is the um and I know it's kind of small the right right side is a uh a minion and the left side is a master what I'm doing right now is I'm just checking it in saying hey I'm here um it's uses a key based authentication so I have to accept that so I think that's what I'm doing yeah so I see it in my keys I'm going to go ahead and set set that key I did this so I do mess something up so um so I'm going over to the the the Min

I'm going to show you that hey bros not installed um I'm going have check in and uh so yeah's not running's not installed um packages in there but the the important part is is that I can do this from you know I can do this from a a a remote access place where I I do this over nmi or something like that so I went and check it in and now was saying I went from a a a minimal install Centos and now it's going to make it a sensor in my environment all all at once so that's basically What's Happening Here I won't um you know let the demo go through all of it but um you know it

just goes and builds the censor so it's important is that you it makes these devices thrill right so I can if I'm troubleshooting machine and I spend longer than 15 minutes just rebuild it right CU all I got all the information already I don't need to figure out why this is happening that's happening I've got 1,800 1,700 1900 sensors I I don't have time to spend three hours trying to rebuild this box in the state just rebuild it and it's back right um so I talked about pulling all detection off and all that's great right that's neat but then well how do I detect stuff if I don't have any Atomic indicators on the boxes and so what we do is we break this

into a service based architecture and that makes us able to scale these things so um I worked at a company where we had a time where it was we want to we pay all these different Intel sources all this money we want to detect all the things right so we've got 5 million Intel indicators we need to we need to check all these IPS well it doesn't uh doesn't scale on a sensor think you think a machine that's pretty much overloaded and uh you know it's not going to be a to do that again we for the data instead of devices put the we needed so we do Pub sub so it's the pub sub to

the rescue um so we ship our prologs um we you know we make subscribers process those log files and uh that gives us the the ability to lose um time instead of losing packets right so when you throw a bunch of rules on something it's going to crash the sensor you're going to lose traffic you're going to lose packet um but when you uh when you have a service based architecture it just gets backed up so here's just the sample architecture so we have log we have rabbit Cube so we use what's called a fan out in that environment um and we break it to different services so rule service an indexing service and an Aral

service so what I said before is like you know if it's good enough for the sensors why is it good enough for everything else so we can use it for logs too um so we get I've get not just in the broke onog but we get it in proxy logs and Fire Etc um we get uh URLs and proxy laws and we get domains and domain laws and there's a big great pitch on this from besides Augusta 2013 from Dave Bianca so I I recommend checking it out he talks about Enterprise secur so what do we approved we talked about all these different things we talked about how there's different problems etc etc well now we can we can use us the

way was ended because we've got sensors at the for the users are so now when California wants to talk to Arizona I see everything coming out of californ and I see everything coming in out of Arizona I don't need to bring it ship it all the way back up I don't need to do this kind of um H is not as bad of a problem now because typically at your site you're not going to have multiple W connections you might have an active in the back up but you're not going to an active active where you're doing things in the different rou and things like that there to expensive you can get it before it's

optimized and um it gives you more eyes and more places to detect collateral movement so um you know that's very important so um before I get to questions you know one one thing that's important when you think about um you know how we have to do this and how we have to react is that we need to you know stay with the flexibility one we have to be able to be flexible and by having things out of the users we can be that we can get that flexibility so um that's really what I wanted to leave you with is that um there's always kind of crazy kinds of tools there's always CR cat of things that you can do um but um

you know by having that flexibility it gives you the gives you the the ability to roll off detection so is there any [Music] questions um so one of the problems that that I'm seeing is we have more and more virtualized Mach networks um and that's even going down to the desktops being P what's what's the strategy to get all this traffic so there there are some virtual switching now that they're they're offering some ways of Port Maring things like that um the challenge is a lot of um the challenge is there's a lot of you know doing NSM if you're doing full NM takes a lot of IO takes a lot of resources away from your virtual server

um but there are ways to do it so we we do have cases where we do install on um again because it's an ISO we install on Virtual machines like a bware box and we're able to get feeds from um you can put uh some of the networks in promiscuous so um so that's one way to get it but it is a challenge right especially when you talk to cloud and things like that where you don't get the traffic so again there there's still you're going to still need host based tools and things of that nature so there's a question here so um one of the problems I face is that uh you decentralized sensors um and

those type of solutions they they still ask for a log aggregation to occur before analysis so you you take all those logs you pull them back and then you and then you look through them um but you know what are what are some suggestions on maybe a centralized analysis or query model um that can be used with a decentralized sensor I don't know yeah so what I talk about that so there's um a lot of people use Cabana and elastic search right so and log stash the El stack um that's an option um you know it depends on how many events it gets a little crazy when you're talking lots of events but you know the El stack Works um Elsa um you

know you can create um different tools to bring this in a stream so you bring those events in as a stream and as you bring those events in you can analyze and take care of so um my suggestion would be get log stash and Elk going and then um you can put like Reddit you can put all kinds of things there's plugins all over that you can grab that information and uh do something with it right so you can that F I was talking about that means I I sent a copy of multiple services so you could have a rule service and something doing indexing you know side by side Sor follow that still is

aggregating right those alerts before you well to Lear it's aggregating the data right so I'm I'm asking as an analyst how do I think of a conceptualized query and have that go out to the sensors and only you know right so we're not querying the sensors we're going to query the data that's already been pulled back from the sensors so the sensors are just sending data they're not they're not really doing any detection so to speak for the most part on the sensors they sending the data to where you have your centralized in electral you know which for IPs and your your your rules and things like that so you know you want to use

compression and things like that to pull those blocks back but that's the you know that's sort of the G is that I want to centralize that data and then do stuff to it instead of asking these sensors to do all these different things when they should just be sensing and S any other questions how do you deal with the uh when you go to you talk about us source youal that uh that hasn't been too much of a problem um lately uh you know everybody knows that there's a lot of different open source tools um as long as you have the legal stuff worked out with your um your legal team as far as you know we don't make any

modifications to any of the software that we roll out uh it's just you know the standard um stuff so work with your legal team though before you do things like that but you know from a customer they're very receptive they're you know they're really when you put one of these devices out there and they start getting the data and seeing it they don't care if it's open source or whatever it is they're getting they want that information they get addicted to that information so you know it's once you have that disability you don't you don't want to lose it again I'm not very familiar with this

sting type

AR so I mean Ty yeah so we typically use log is the transport method um and then compressing that there's I mean there's a million ways to do it um I mean I think that's the best way I can put it that there's there's just uh you know it's so green field but but the elk stack has a lot of things that are plugins that you can use to to grab that um to get that into that centralized database and then if you use things like Reddit or RAB and Q you can fan out to other services that can you know you can run a python thing that Fe that grabs off there and does you know

your PC your IP lookups and things like that For Those sensors is are they also a tap or you require a tap for every single one of well no so that they're not a tap so these aren't inline devices these are just boxes right so you're going to need taps or spans or whatever you want to use to grab that trap these are just the devices that get the stuff so um yeah that's a whole different thing right so a company work that we had like 900 TS and it's pretty you know pretty cumbersome but but yeah you can use spans although I don't recommend spans but they work so it's better it's better than

nothing any other questions awesome well thank you very much