BSidesAugusta 2014 Mike Reeves - Scaling Security Onion to the Enterprise

all right folks if you want to go ahead and grab a seat we're going to move on to the next Talk of the day uh next up we have Mr Mike Reeves and uh Mike formerly was at a fortune five company one of the largest companies in the world including the largest networks in the world and he helped to run one of the largest squeal deployments one of the largest NSM deployments in the entire world so he's certainly an expert at knowing how to scale these Technologies and and give this ability to very large networks and very large Enterprises so please join me in welcoming Mr Mike Reeves

um I've been doing this for a while my main focus has been on IDs and uh Unix um I work for FireEye which I was actually working for Mania and sorry I bought us um like Doug said I spent 12 years on the fortune 5 and I created something called onion salt for security onions um just a little bit of background when I started security at this place we had one of the first squeal installs and then um I actually came over to our corporate security team where we started rolling out sensors when I joined the team we had I think 12 12 or 14 sensors uh when I left we had 460 something sensors we had 800 Taps so it's quite a

big uh challenge so to speak to um to cover all that area so what I like to do is uh when I give this talk is talk about what is innocent right A lot of people ask questions like um you know what are what is NSN right we do IDs or we do IPS but what's this innocent stuff I keep hearing about I like to use the Richard baitlyn definition which is the collection analysis and escalation indications of warning detect and respond to intrusion so another way to put that is um it arms the analysts of everything they need to make a quick decision in order to say is this something I need to escalate is this something I need to

move forward with um you know giving them that that information and why is this information important well we can take the typical IDs versus NSM scenario right so if you know back in the day but um you know you would you would look at an alert come in like um you know like the ISS console or something back then um you look at alert come in and say um I see this signature go off right so you look at it you say well yeah the signature is bad let's say it's a first stage um somebody exploiting a flashlight so we see that alert command say okay this person has flashed it importantly so I put in a request the it guys and

say hey this machine one two three four five it's m right or it could be owned can you take a look at it so what do they do they go over there and they fire up antivirus and they scan and say oh it says it's clean we're good close the ticket what are you why are you scaring people making us run around right these guys look horrible right because the job's doing so you take that same um scenario and you apply it in a sense so that alert comes in um you you look at and say okay well it's the first stage they they owned uh the flash flying on there um you pull up the trains uh the

transaction log or the um escaping my mind but you pull the transcript there it is you pull the transcript up for it and you see that it was uh um it saw you saw the first stage and then the second stage you see it go down you see it connect and grab a another file from a website an exe file that that probably got executed thereby looking at that transcript you can also see okay well what else did this thing connect to well right now you see that it has a C2 connection to the Ukrainian turkey you don't know if it's a C2 but if you're a company in America and this person shouldn't really be talking there

and they've got open connections today that's a good idea that hey there's something wrong so you open that same request with IG support they go out there and they scan it again they say hey this thing there's nothing wrong with it anybody says it's cool well the great thing about it is you could pull that so you know if they download it so you can pull that binary out of the pcap and now you have so you can throw that in cuckoo you can throw that in anything um you can throw it into virus total and then you find out that hey this thing that they said you know you're crazy this is there's nothing

wrong with this um you now know that the antivirus vendor that you use doesn't recognize the virus but 20 other ones do right so that gives you a situation where um you know you you actually you know from security perspective versus like T you win right so um but it also knocks the other way around so let's say you could say we see the first stage right and instead of putting a ticket and going crazy and getting everybody riled up you saw that it got a 404 and I wanted to get the second stage so you know that that machine needs to be patched which you do know that when it went to grab the evil

so to speak um that uh it didn't work right so that now you've saved yourself time so you know it gives it again it's arming the analyst with what they mean and I I did a blog post um about this uh scenario it was a real scenario so what are the challenges of bringing this into the Enterprise so the first thing is always convincing your management that hey I need some money um the network teams hey I need to put these boxes for any routers and switches um where do I put them once they say okay that's cool what do I do when I've got lots of bandwidth uh lawyer people what do I do with those

people because they're gonna be like oh you're sniffing the network uh you're doing stuff um now I got all these devices these new devices and I don't want the the admins the manage them and then dealing with all that data so let's get compliance out of the way because it's the most boring um when you do a global fund so the company I worked at was had a very huge Global presence obviously with 500 something devices we have lots of stuff in France and Germany where they have really strict privacy concerns right so you need to check with your work councils or legal departments to make sure that hey if I put full pcap out

here am I going to run into some some issues right um so check with your legal folks right um you always want to protect the data right so a lot of times the targets of your um incidents are your administrators right so I always use the windows admin thing where you have cash credentials on your Windows Server your windows service owned from some sort of SQL injection attack now they have that person's credentials right or it could be a key larger situation or whatever because once you you know as an address there if you go after those people it's a lot easier to get one of them and use that finish all the way around so you don't

want them to have access to your systems that um have all your indicators on it right you know what the bad guys knowing what you're looking for so that you know because we have to address our ttps to get around that um learn bpfs bpfs are very important so um you know this company that I worked at we had a financial sector and there was lots of PCI and the medical sector so there's HIPAA um but you couldn't you weren't allowed to to pcap certain traffic right so if it had to do with credit cards or something like that or had certain personal information so we wanted to look at snort we wanted to see the

connection info but we just didn't want to see happen so an End security I mean there's uh you know there's ways for Individual Services to put pcaps or BPS in or just say net snip NG in this case so what's the best way to convince your manager find an old box that you have sitting around that has you know eight gig of RAM or so drop it talk to your network team get a never expanded port or a tap put in and just throw it out there and you'll be surprised if you've never had any visibility on your network before I'm sure you will find some um I I've gone into lots of small and

medium medium-sized companies who said well there's nothing wrong we're awesome our security is great like there's this one admin guy who like everything's perfect um and it's really good when you put something up and you find their machine is one of the one says and and doing things so um and then just some stuff is you know you can say hey you know the median is 243 days it shouldn't be average convenient um okay you know they're on the network before being detected so the never team in tax they're afraid of tax a lot of times um so they uh yeah for some reason um if especially if you're talking with dump tabs so a dump tap is basically a

it's basically a networking that's why I was explaining them so if you're using a dump that now if you're using something with smarts in it it can be aggregation all that kind of stuff you got a little bit different story because those are the only tabs that I've seen have problems where you get connected but a normal tap especially five so a fire attack is just mirrors right so the only way to if it when you plug it in and it works it's always going to unless somebody drops it on the ground and breaks something or realize them so there's no way that like a fiber attack is going to have a problem I mean I'm

sure and there could be a problem but I've never seen one fail um copper caps they are wired so that yeah even without power so when we would install these tabs what we would do is we would install them with power off so it had two we use Netgear Taps and they had two connected Connections in the back um and we would plug them in with them off so that hey look see everything works they don't send anything out of the monitoring ports um but they work uh you know the traffic is back and forth so you plug this in um and the reason I'm talking so much about taxes there are issues with Spain so spans

um there's there's a couple different things that I I could tweet about some of the articles that you don't get everything from Spain query you get most of them so all you have is a spam port go with the Spanish word right um or if you need it quickly while you're ordering Taps are going through your procurement process or anything the span will work but there are there are limitations to expand so if you have let's say you have a big back-end background switch let's say it's like a 6509 Cisco switch I think there's like a 12 million packet per second limit on its way so if you say take a VLAN and I don't even think there's an easy way to

tell um that this VLAN is pumping you know let's say 5 million packs per second can you mirror that thing now you have 10 million packets per second right because you take every single packet that that switch sees which is not designed really to do and you're duplicating it out that port um the other bad thing about uh spam courses never teams like to take them well we're having a problem with the exchange servers we need to run the sniffer on there right uh well then they forget to take it and get back to you right so all of a sudden um two days later you're like man I haven't seen any alerts from this sensor

in a while what's going on here um or I've seen any data and then you figure out oh we're not seeing any traffic on this this fan

yeah we'll get to that in a second so um because they're designed for that but like your standard Cisco switch is not we've we've melted down some good size switches um and labeling interfaces that have a tap and always house so I have a tap right here the giveaway it's a dual com tap it's a really really nice one um so I have some trivia questions um or a trivia question this is star working so all my questions are going to be Movie base so get your idv's ready um how did Han Solo meet Chewbacca

so I talked about placement before so why why why is placement important well placement means everything right so what I like to do is I like to get the true source and the true destination why is that important I wanted to use machines on them and I want to know where they were going um sometimes when you have proxies involved with something like that you can use X4 there are ways to yank that out of there but uh you know typically what I like to do so let's say you're um proxies on your DMZ I like to put a tap on both sides so I see that yeah it's a little bit of extra stuff in there

they're different sessions from night from a from a session perspective right because you know your client's going to your proxy and the proxy turn around going out and doing something so you're not gonna have too much stepping on each other and it's it'll be easy to weave those two together so that you can quickly um quickly put those things together so um so that's very important um you also want to check the inside of your third-party routers anybody know why you want to go like like on your VPN while you want to go on the inside not only that but there's another reason why if you're doing VPN that you want to be on the inside there you go right so

it's not it's no use putting it on the outside so this even though your weekend traffic is coming through this town it's not really doing any good here because you don't see anything oh there's an ESP going by that's great you know you don't really know what that is so um that is uh you know another another good thing um you always want to put in front of your like Secret Sauce Networks um and that's where we talk about you know we'll talk about high speed and stuff like that um and then try to avoid asynchronous routing this is really hard on the internet side let's say you have two locations that are that are close

there's not much you can do um but let's say you in the situation you had two routers um and you had two tabs you would want to bring this into the same box so that you could you could bring those sessions together because if you go out one ISP and come back the other one and that breaks with every tool so feel the need the Need for Speed um so from an Enterprise perspective we're seeing lots of speeds I was at the very conference I don't know three or four weeks ago I could do what it was um anyway we're talking a lot of people so the most I've ever insurance is 40 gig and now a lot

of these universities there's big hundred connections to the I2 and stuff and they're like what do we do right it's the solution is the same for a 10 gig or 100 or 40 gig as 100 Gig the problem with 100 Gig is just the interfaces cost like 40 Grand change so it's going to be really hard um to monitor this um generally a single sensor that I that I normally that you can do um generally I can see about two gigs of traffic I can maintain two years of traffic but that would be disclaimer there there's lots of factors and we'll talk about that a little bit um if you put specialized gear in um it costs a lot of money so

um there's Nicks that cost you know fifteen twenty thousand dollars just for the network card and I can build several systems that can do way more than with that just one system with that network card for the price of the network card um but you need something like a flow-based load balancer or a special Matrix switch or very uncommon so what is flow-based load balancing so flow-based load balancing takes multiple taps or multiple flows takes them in and balances them across multiple outputs so what that means is you take a 10 gig connection and now you're splitting it in by three on a flow basis so you this isn't the the perfect solution because it's not

like saying okay this session I'm doing two gigs or I'm doing uh you know I've got it it counts them right so it says you know because one one two two right it sends them across but if that first connection if somebody's downloading um a bunch of stuff from the streaming a movie from Netflix and it's going at you know 100 megabits versus the other person who's checking the gmail which is so you know so there's it does pretty good about balancing the traffic but it's face on Flow so you can have a single flow that can that can roast one of these sensors in there but it doesn't happen very often but it's technically

possible that's good enough so um bro's really good in this scenario from a clustering perspective so if you're running a bro cluster down here um they know about other things so one thing it you know from a security gun perspective if these are all running sort um this guy doesn't know what this guy's seeing whereas when Pro they all know who's seen one right so you can you can rule say if somebody sends out this many gns packets within this time period it's not just on this flow it's from the you know the entire cluster group um it also lets you share with others I mean you take that Network team example where the network team needs to look at

the exchange services contains to reject up um you can give them a fee off this load balancer if they somebody wants to test some new tool you can give them a feed um and get them off your back right so it's cool um some of the advantage the example vendors are like a Resa recent sells one they give you the tap add mode there's the CPAC at Sea View series there's the gigamons um I think uh net Optics has a their director series does this um there's a lot of different ways you can do it so what does that look like though in a large Enterprise so when you have um lots and lots of bandwidth so in this

scenario we have multiple 10-day connections to the internet uh 40 gig on the back end side and then 40 gig to our secret sauce Network and what this enables us to do um is to look at these links and do it from a commodity level for cheap sort of and uh and do it effectively but the downside of doing this way is as you can see there's lots of these things called sensors on here so managing all those boxes becomes a real pain so now we've got all these sensors what have we want so now we need to have this new specialized architecture people say oh you put the slide box on my network it's

scary you're gonna get tons of pushback right because you know I want to put these in there um you don't want this stuff supported by your typical support team so you need automation right you don't want to be sitting around you want to be looking for bad guys you don't want to be adding user accounts all day and changing stuff so um I live in Kentucky I'm not from Kentucky but I live in Kentucky so um put a little things in here so Hardware sizes this is the most uh the biggest question I get asked all the time I've got this is my family with which side box do I need um it's a hard question to answer so I'm

taking a shot at it if somebody has better recommendations um all ears um the track traffic profiles things that you know things of that nature they all play into this as far as what you need so I've had sensors that have been on seeing two gigs and using like 25 of the box but it was like mostly CIS traffic so I didn't have a lot of snorkels and it was like whatever but then I've had boxes on like a three or four hundred connection that were getting roasted because they were like a web and they had tons of users and lots of things going on but typically you can use the fact that the more course that you have the more

traffic you're gonna be able to see you need lots of ram so um whatever you can put Ram in that's what you need to do so keep on my theme here um for a hundred Meg connection um you can get away with the four core hyper threaded processor 16 gigs of RAM I mean it could probably go a little bit lower but like I said I would put 16 in there and you can use software raid um and multiple Nicks obviously you want to have a monitoring interface and all that kind of stuff now to get up to two gig it's a little bit different so um the uh you want to go with at least two

six core hyper threads we were talking at dinner last night um actually 10 core processors would come down a lot in price um so you could probably even do more than two days now with a quick twin um 10 core processors you want to look at 100 Trade gigs of RAM you want to have um Hardware refile as many distance as possible the reason you want lots of discs is you want more specific trays so that you can write data fast um and you want PC Express next quad Knicks are great for this because you can Bond them together so the other question I get asked is okay I I've got this sensors on high

speed connection what can I do to make this thing go faster um so these first two have to do with the way Linux handles this cache so in Linux all free memory is assigned at this cache as it needs it it pulls it out of there um the problem is now that you know all this Ram you have a situation where you're not right so you get great write speed for 35 45 50 seconds and after they did the full timeout team I think three minutes so you have great write speed and then when it gets to the point it says okay I'm timing out now I'm going to go right now you're you're you're it's pretty affordable so you

have to pay for that speed at some point so what you want to do is you want to balance that so that you're almost constantly right so you'll see a process called PD flush and when PD flush is running that means you're writing to disk so you want to make sure that you can get PD flush running office costly so you got to tweak these things um I think that the background ratio is the default 10 so when 10 of the right cash gets filled up it starts writing um some some kernels that only let you go down to five percent but play with that and then the center seconds which is you can use that to uh to determine

what the best way to get the point there's not really a number I can give you because I just I mean this is a per sensor thing especially when you're talking high speed um the PF ring num slots thing again this I have this I think my the phone's like 4096 it goes up to 65 535 I believe um and this is just the the Ring Side the number of slots or yeah basically your safety buffer uh on PF rank that you can use so that's something that's Ram dependent so you want to be careful so if you have if you don't have a lot of ram um you can't take that number same thing

with the net Sniff and she ring size this is something uh you know from Full packet caption perspective the same kind of thing it's just this is your buffer it goes from The Ring to this buffer and from that buffer to disk so you know depending on you and that helps you in the burst situation so if you have a short burst you're going to fill up the cache and then you gotta write that but the more cash obviously the more burst you can handle um and you want to pin your processes to real quarters so if you look at cat if you can't proc uh CPU info you'll see that um you'll see a processor and you see

like a child processor I forget what it's called but you always want to put it on real processors um when you pay your processes they have to do with the disk cache on Wednesday so like if you have 256 speeds of ram you have a lot of disk cash right so if you're you know if you're going at two gigs per second and you have um you know 25 gigs of of um write cash and your writing already it's right in cash and then when it goes to write then all of a sudden you're in the ioa situation so you want to get it to constantly right versus that big it is well hopefully you have enough Rand that

you shouldn't be swapping so you don't ever really want to swap um but the ratio yes no no so um okay so when you take I think when you take this cache um it's always seen as free so when you do like a hello I'm geek here sorry about this we had some technical difficulties and the section of the software that takes care of the extreme cap u3 seems to have freezing up and uh well we didn't get any video of the speaker or sound here so for the next minute or so you're gonna have absolutely nothing should start back up after that with sound though we did lose a small section sorry about that

gotta triage these I'm seeing something about some sort of RDP session but it's coming from VPN we don't like to see that across VPN right and you can tell that by naming using a naming convention with your sensor saying you know this is a VPN sensor we're going to call it VPN right so we put VPN in or something or you know this is our our internal sensors we want internal or an external sensor for traffic going outbound um you want to uh you know because you're if you have a student we're using the flow based load bouncer you can get a copy of the traffic and it's a great place to test testing on real quick

the hardest thing to do one of the hardest things to do is test um because you need real traffic because I can put a rule set on this device and on this device and the results are completely different so um you you really always want to have a test sensor on something that's like uh unreal product so if you have floaties loadouts you can do the cut you can do that and you want a stage of rollouts so even though you did test it on this roll it out to a couple test sensors in each group of rule sets that you have to make sure that it's not going to bring everything down so then

if you impact something you're only going to impact a couple sensors and set up the whole thing when we first started doing this we you know somebody would have a typo and a rule it would take two hours to put the rules out there and so it's not we're good to start up on that new rule said it's like oh crap like there's something wrong with this I'm not starting well now it takes another two hours to do it again so um when we get to the Salt stuff that's why I um you know turn to something like that to do that so let's talk about security I I assume everybody knows about security Union

here I this guy probably doesn't know but um so it smells yeah it smells it smells really bad um so uh security onion is awesome I'm a huge fan um but we won't go too into there but there are some challenges with it when you come to like an Enterprise level deployment so when you have a lot of sensors and then like I mean even even now when I help people if they're even putting two sensors I haven't set the onions also because it's like look you're gonna add more and it's just easier to do it this way um room management so uh you know I was helping a friend I was trying to tune

his rules he had 15 sensors I wrote a shell script and special users that could do things um so that I could start restarting the rules but then it would take a bunch of time and I said man this is this is a pain because by default security onion just downloads it once a night right so there's a Cron job that runs grabs it from the pulled pork stuff and then the Cron job runs on the sensors and it waits five minutes and then it goes um can you grabs those from there there's lots of tools that you might not need um and even though it's simple there are some learning for free there's a lot of

knobs you can turn to make it work to your advantage so a couple tips always set up your innocent partition before you do so setup I've done this many times where I've uh created a sensor that around us to set up and I'm like oh crap I didn't Point NSM to the partition that I have on my face on and I got to move it there's five commissions a lot of times I'll just say screw it I'd set up the partition agreement what's the setup because I just didn't want to deal with it I said I've done that a lot of times um create your virtual interfaces if you're going to run so let's say you had

a tap it has two outputs right for your TX and your Rx um you're going to want to bond those interfaces together to bridge them I think either will work but you want to create that interface beforehand so that you can point all your tools to that interface and in my previous life we use that a lot right because we have boxes that have up to you know three Taps on them and that would have six interfaces well they were all the same type of traffic I wanted to look at this one I didn't have you know a bunch of different configurations running so it made it so I can have a configuration across all

um turn up all your stuff that you don't need um and use it every day this is the most important one on the slide because the way squeal works you need to be in there and categorize some events because if you leave it alone for a couple days or a couple weeks then you go and load it you're gonna get that gray screen of death waiting for stuff to come out um so I created onion salt because um I wanted to be lazy and I didn't want to do a lot of stuff that I had to do um mostly I didn't see either count so every time somebody said hey I need to use it I hate creating music

um so what love lets you do is it is manage it once the sensors like one right and make a Conformity across those devices right you need to standardized thing so my previous line we had 500 sensors they were all the same Hardware platform they all had the same size um management partitions that as time went on and drives got cheaper they all had different size stuff we had ways to detect that but in all you could take the config you can lose the hardware bring in bring a new hardware and use the same configuration on it and be up and running again it makes the user handshake simple because nobody wants to do that it

changes the way the rule Management's gone so anytime a rule changes it sees it and makes it in a resources hard so what this does if you're tuning something you don't have to go to each sensor or run a script or anything like that it says you know if I've got a set of 15 minutes I know in 15 minutes everybody's gonna be running this rule set or if it's a situation where I want to get now I can tell them hey do it now I have full control and that's the GitHub of it there's some stuff in there so users are now managed essentially if you're doing it this way they're quite default for granite pseudo access

um they're creating with no passwords so it's all Key Management based you drop your user accounts into this file and I have an example file I'll show next and then you drop a key into that directory and the public key lots of people try to send you the private key so our users file this is where we um this is where we uh put our user accounts and it's really simple it's a ginger format um so these are spaces you can't use tabs or they'll air out on you but you just put the name of the user account person's full name um and then the group two and that's it you drop the key in there you save that

next time my sensor checks in it will say I need to create this uh user account and I need to um added to that group so now they're on there the rules um what we're selling it here is the file.recourse means installed is everything in this directory sync it so I'm saying take everything in the sensor rules directory on the master and put it in this directory up here and then here I'm saying okay I want you to wait and if you see any changes to this folder run this and that's that's it so what it's doing is it's saying okay I'm going to watch the rules file directory rule file rule changes I run this and hey it changed

restart so you're good to go um before I get the Bruins full framework I got some PSN stuff to get giveaway I'm a huge fan of PSNS the first one is the USPS and CSV file um does it have malware on it I don't know giveaway so if you can guess what movie this quote is from you win the key fob they mostly come at night mostly you got it right there

all right now this is a big one for this t-shirt and the sticker I think it's a t-shirt yeah where does this what room does this quote came from hey sorry you know how I said to us I died there you go

I don't want him again it's all about this stuff so one thing I've added is the Bro Intel framework um let me check my time here before I go to this so the Brew until framework um allows you to do things in this case it's the scene module snort's really bad at uh doing large lists of ips because it doesn't have a pre-processor they have the the new IP um reputation protocol but it's it doesn't give you context you can't say this is you know this it just says hey I found a bad IP right um well that doesn't help when you have 10 000 IPS in your list you want to be able to attribute that to somebody that

you know is it for some Zoo site or is this like you know ABT 600 or something um so simply here what we're doing is we're using something called file block replace and what this allows us to do is have a section in the file that no matter what happens to it salt will always put it that way so you can have all your custom scripts in your local Pro it won't touch any of that it would just make sure everything that's in between this line and this line equals this so that's really um it's really all it does um so it loads the scene mod or scene policy which um and then it does the the

Intel notice framework and then it reads this file we're called evil not Intel so what does the inside of evil not Intel inside that we have um there's there's several different groups you can look on the pro documentation there's stuff you can go to icmb5 so like that just didn't look good on the slide if I did it um but you can do emails you can do um domain names and the cool part is you can say okay I would you know if I see this and I know that this is you know some evil group uh and I can point it to a URL so an analyst you harm them says okay I saw a

bad IP what is this well it's associated with this bad group I need to go look at it and then oh I click on here and now I've got all the details about you know what are the tgps of this actor what do they do stuff like that so it gives you you know the ability to do that um so a couple ticks the tips and tricks with onion salt so I talked about before how you could run commands on sensors well that's the command run so when we talk about naming convention and everything like that you can use wildcard so you can say if I start everything with sensor you can say every sensor upgraded packages right so that's

that sensor command applicate upgrade um you can tell it to check in which is saltcall state DOT High State and I'll I'll put together a cheat sheet of stuff and I'll I'll tweet it out of some of the things that you can do there but it gives you the ability to do things um from one spot so if you want to put UW and G on all your boxes you don't have to log into every box you have to install you just you type this and boom everybody's got it um so that that's very very helpful when it comes to managing one so I type it on One sensor while the master in this case

and now everybody's got it so I manage that whole grid of tons of computers as one computer um and it you know because it Checks In it uh you know and make sure whatever I want to be uh you know from performing perspective would be that um so there were some things I got coming up with uninstall um I'm going to centralize Elsa sorry Martin um so that you know you're not you don't have contention versus your pcap and your low data you might want to keep flow data for like a year um versus the peak if you want to keep as long as you can that could be like a couple days on your sensor multiple rule

set support where we talk about different sensor types um I want to have that in there so that you have different directories for different types and you can assign since there's different rule groups and I'm going to retire so set up for sensors that that onion salt does everything for you just fill up file um and it makes it happen so any questions any more questions I guess I think I may have asked this questions and answers your presentation yesterday how many sensors can a manager manage I guess what if you stagger them that that depends um with salt I don't know so I used I use puppet in my previous life um and you're obviously able to do close

to 500. um with salt I'm not sure that's a good question I mean I think the most I've seen on it is like uh like 40 or 50. um you know anybody that's running 140 or 50. well I know that there are some security on you deployments that are larger than 50. um and I can tell you for result itself I know that it scales to very high numbers there's folks in the devops community that you can solve up to thousands right so I think for security Union specifically I think you probably run into a limitation yeah because you got to tickle so it's a 1024 Amendment on squeal so um which I think you've got two

connections per instance of start running so that there's math involved there and that's just the so if you have if you have a sensor that's running four instances of snort that's what six connections or eight connections back to the the main server and uh you know so that factors it's a TCL review limitation because we we try to overcome that my other company and we actually ended up breaking different Services out um into different things because of that so sure

well even if you run surikata if you're talking about squeal you still have to have these snort agents to allow the story alerts to get back to school and assuming you still want to run the whole pcap then you still have to have the pcapp pager so it's it's the agent communication back to school need of consuming the science yeah it's a socket limitation

each instance runs a sense smart agent I believe right so that's that eats up a socket in PCS 10 24 and you can't recompile it we tried I've tried tons of different things it's TCL yeah any other questions all right great thank you very much

great job Mike thank you very much uh so folks we have lunch coming up