Cleveland BSides 2011 - Talk6 SteveJ

christiev jawarski and his the talk of his or the title of his talk is where is the flow Steve ja risk is an associate with booz allen hamilton supporting a super-secret DoD client i add to the super-secret thought a cooler that's fine yeah not too many in Cleveland exactly he's an information secure professional more in 12 + years experience in enterprise IT some of his security interests our network flow analysis log analysis and wireless security steve has a bachelor's of science and information systems from ball and Wallace College as you can see here post your child exactly when not sniffing packets Steve presents on various security topics in the Northeast Ohio area Steve's is a sans mentor and

maintains 55 GX certifications g-sec GCF a GCF WGCI H and G CIA my company paid for that's right so ladies and gentlemen Steve jawarski thanks thanks guys all right I'd like to comment on the last presentation real quick it was pretty good right yeah I thought it was pretty good what really disgusted me about it was how clean that code looked didn't it you know for a you know where Steve is even in here to get oh he took off huh yeah he's not feeling well so anyway it was pretty clean pretty disgusting so anyway but pretty cool toll so to tool sorry about that so for the next 45 minutes you guys will be disappointed

now so here we go so where's the flow well we need to ask the all-knowing Wikipedia right so essentially talks about net flow and s flow and I won't read it verbatim but essentially it's just some long definition so essentially what's a flow in my opinion a flow is you know what's going across your network source destination what is it what's it doing how long is it what ports it's running on what protocol that's a flow the question is is what can you do with that flow what's the security take on the flow so how many folks here use net flow in their environments a few nes flow people s flow yes alright so I like us flow better the net flow

but I'm bias so some of the popular flow technologies net flow that's the biggest one there's now an open standard IP fix anybody playing with IP fix today netflow v9 is IP fix jay phlo never really done too much with juniper networks flow and then s flow I'm a huge fan of s flow and I'll tell you why they have an old RFC but there hasn't been any updates to it but there are new versions of s flow just no updated RFC so net flow versus a s flow well one thing cool about net flow is it captures the whole conversation so if you configure it properly you'll see the whole duration you'll see the whole

communication you get a lot of information about it but one of the other things is the data is sent after the conversation is complete so the question is is do you need the conversation while it's starting or after it's already ended just something to think about s flow is only sample s flow is a sampling technology so s flow is it only captures every so many bytes so the default or the fastest it could do is one and 128 x 128 packets but it captures the first 128 bytes of a packet you get the whole first 128 bytes of a packet that's a lot of information net flow is just the session and we're going

to look at some of the packets so just to take a step back essentially I just want to educate you guys a little bit about net flow and s flow and then we're going to show how we could do it was security other thing cool about s floats hardware-based there's actually a chip in routers and switches and wireless access points that the s flow code runs in and it's free so what's in the nest flow packet well like I said hold I'm sorry first hundred twenty bytes of data so you get the full ethernet header the full IP header very cool stuff and then it clucks right off the wire so whatever the protocol is

it doesn't care it just captures it and sends it off you just need a collector that could decode it very cool for those who know me a kind of hate Cisco I've kind of learned to hate Cisco over the years so I'm a huge foundry networks fan now brocade so essentially that networking equipment is probably going to go go to the wayside sorry that's a personal opinion no no not sent brocade so anyway that's the s flow packet so what is a pretty esflow packet look like well in a simple tcp or a Wireshark dump you know essentially it just breaks it out you know Wireshark already has a predefined filter in it to break out the

ice floe packet and one of the cool things it's hard to see but you get the like I said you get the whole frame so we get the source destination TCP packets or TCP flags essentially so it's pretty cool and then what's in a net net flow packet same stuff source destination protocol the duration of flows what ports TCP flags all that stuff one of the cool things net flow has that s flow does not have is the next router hop that's kind of interesting you know where's this pack in destin next might be useful and then version 5 same stuff one of the biggest things i like about s flow over net flow as i get the mac address of this source

so in a corporate network you know lay an environment whatever you want to call it we have a couple thousand hosts well along with that IP address you get the source MAC you have the mac address you know exactly who that person is you could find them a lot quicker than if you only had the IP s flow too along with net flow though they give you the obviously the device that collected the packet what port it found it on and it sends that information of the packet so that helps but when you have the mac makes all the difference i came from academia so mac address was the ultimate right we go bus students

for illegal file-sharing because you know the RIAA said we had to and we had no money to fight the RIAA so we had to comply so student would say well that's not my IP address was like well we don't care we got your Mac so we knew what it was all we do is just match the time stamps so f flow agents or exporters whatever you want to call them well the 800-pound gorilla we all know who that is Cisco brocade and you know not going to list them all but there's everybody has some sort of flow technology in their system you know I found s flow I got it sounding really redundant here apologize but it's all in there layer 2

devices HP actually has the first vendor because they kind of co-created the protocol with a company called in line to actually put in wireless access points so to start getting flow data from access points really useful plenty of collectors out there iron view which is foundry networks Lancope I really love Lancope I think it's kind of the excuse me though what you want to call it packet flow Cadillac a packet flow collector or Cadillac of packet flow collectors licks er is kind of cool i just tested it one of the cool things about sir is that it just runs on a windows box the land cope stealthWatch is a whole appliance style system you had a collector you got a

management console they're all separate boxes so if you're doing netflow s flow and all kinds of different things you can have like three or four boxes but Pulitzer is kind of cool and then arbor networks is pretty popular I think with ISPs but I really haven't used it and then in mine the original creator of s flow they have a thing called traffic Sentinel but not so much of a security peace I think you see with Lancope and pollux are more security usages for flow data and then s flowed orgs a great site for all different kinds of collectors if any collector collects esflow it also collects net flow some free stuff right free is good free

is good so anyway the s flute toolkit runs on linux really easy to use what's kind of cool about that you could collect your s flow data and being in Linux just pipe it over to TCP dump pipe it over to snort wherever you want it whatever you want to do with it and ha everybody's kind of probably heard of men top and then netflow there's a thing called flow tools written by a guy out of Ohio State part mar 4 that's correct thank you Mark Fullmer I haven't really played with too many of the net flow tools but you know lmgtfy com net flow you know anybody ever used that site with their users yes okay we're like

that let me google that for you there's thousands of net flow collectors out there so some free flow generators when I was putting this presentation together one thing I did not have access to any more was free flow data working in an academic environment going on the student network I had tons of free flow data well now that i left that environment i don't have so much of that data and who I work for would be very upset with me if I took some data so anyway there's some pretty cool traffic generators out there that just create random netflow packets well the one I really liked was this net flow traffic generator so it it does version 5 for

free and you could really customize it you could specify what ports you want in the in the net flow packet what protocols everything anything you could put in a net flow packet you customize autonomous system numbers source destinations whatever you want so kind of got me thinking when you have a net flow collector running most of the most of the time these guys just collect whatever data is sent to them on UDP port 5202 or whatever it is or 5,000 to whatever netflow runs on well if you're collector is not restricting what IP address is collecting Don you can send some pretty cool stuff to somebody's collector so something to think about with your net flow and I'll

just say your flow collectors restrict what what hosts are allowed to send data to it because with these little flow collectors you could bombard those collectors and put in whatever data you want it's kind of fun a flow liser that's not misspelled it's from sir it's not bad it's ok not as customizable and then there's just another free one just net flow generator but if you google for those they're really cool tools like I said the one I really like e uses a net flow nine it's just that you have to pay for it and we like free things so old school versus are still school whatever you want to call it so here's what we're trying to

get here's what I'm trying to get at with this little slide right you need to monitor your whole network ok well one of the things you could do is you could throw sensors everywhere whether they're little you know TCP dump sensors and you're doing port mirroring and your switch infrastructure got something sitting off your router arm you know arm of your router and you just got a little collector going on well that's going to get expensive and if not tedious to maintain so that's where flows come in right you know send flows to a centralized collector or maybe a few collectors collect that data now here's what's interesting who thinks they need to see every packet on their network to

do good analysis anybody anybody in that shot that hey you know we need to see every packet ya think somebody argue with me you're going to art yeah absolutely you need to see every pic well you're wrong alright so so anyway no one thing I learned is I always struggled with well what if I don't see every packet well you know what who cares right you're not going to see them all it's just there's not enough time in the day the year your life to find every packet that crosses your network so that's kind of where flows come into play and that's what I like about it let's just get a snapshot of what's going on

if I find something interesting then I'm going to drill down more so it's always good to maybe have a TCP dump box handy I tell you that little box Steve Bryden would be perfect right you could you just go throw that out in your network somewhere span a port or if you got extra money in actual tap right and plug it in if you need more data so that's kind of old school new school right most of your networking devices already have a net flow ur s flow technology built into it Cisco Cisco gurus is their net flow and their layer 2 switches yet anybody know that because I'm having a hard time finding it so that's one of

the things I don't like about Cisco is that it seems a lot of it's still in the router technology we need to see that stuff in the switch that's where a lot of action is happening and you get to see deeper in your network right if you could send all your data from your switches or your wireless access points to a central location at no extra cost except for the cost of the collector you're going to see a lot more and supposedly the it's a lower cost until you meet with vendors and they give you the quote for the collector right and you're like oh okay it's per device or purport right that's that's where they

get you so simple flow design all right it's pretty simple I stole this picture I don't care come get me I gave you copyright credit at the bottom but anyway you know it's essentially all your networking devices no matter where they sit there collecting data and they send them off to my little awesome Vizio vizio s fellow collector that i added a copy and paste pretty cool you guys should try it so security right we're here to talk about security not a history lesson so DNS right mr. Douglas I'll address him as mr douglas because i don't know him very well but he's pointed out you know dns servers who who's dns server should you be using

well yours nobody else's so one of the coolest easiest things you could deal with flow data is find out who's using dns so you know because i think you know you guys are going to take all these IP addresses i had to knock them out but especially this this is from iron view networks management tool iron view network manager and essentially i have a bunch of hosts and i just told it to say hey tell me what's going on port 53 and show me all the destinations well it's pretty interesting some for doubt what evers and some eight dots and some other stuff so here's what's cool so we dig into that data a little deeper and we

find out I have some users on my network using google dns and then I have some users going to this really odd IP address turns out to be some dns server in the Ukraine well I'm glad that Ukraine is serving DNS to my my employees and students but I think we could get a little closer so I don't know what else they what their other motives are except to be good net citizens so and then the other thing is private addressing right nag ademia what's fun is you get to point I'd miss it I've only been gone from academia for like three months and still having withdraws so you have to forgive me but you got to two worlds right you got the

student network which is free and easy and all kinds of viruses and good stuff and then you have the business side where you actually have to behave yourself and you know install antivirus and patch and lie to your auditors that you really did do that so anyway right so moving on so this is being recorded oh shoot forgot so anyway private addressing right that's somebody who had their system hard-coded or somebody decided to put a linksys wireless router on the network because the wireless signal I provided was not strong enough and they forgot that way n goes to not them but they obviously they plug my way important to the network or the land port into the network thinking land goes

to land and start serving everybody else IP addresses so pretty easy find right that's pretty easy right your net flow and sflow packets they all tell you a destination really easy query so smtp who's spamming one of the things we dealt with all the time was users getting infected with the virus and they'd start spamming ok well in academia you do not block stuff on the firewall no egress rules right no egress so when you don't have any egress rules well we need a way to figure out who's spamming so here's a nice good clean smt smtp flow on the will say the left the left column our source addresses on the bottom row the y-axis our destination

smtp servers and all the squares and diamonds mean is the bigger the square or diamond the more data is being transferred and if it's a diamond the connection is currently active and if it's a square the connection is finished now this is with the land cope stealthWatch tool ok so this is one of the custom reports i could create and so it's pretty easy to understand right you know we have a bunch of hosts connecting to a few smtp servers looks pretty clean now when we get some hosts again our source hosts or on the left or the x-axis eighth grade algebra teacher would be very happy with me why access our destination host so it's pretty easy

to see who the spammers are right i got a lot of freaking email accounts that they have to check every day it's a lot of work but no it's pretty pretty easy stuff right again this is the land cope tool so it's pretty easy to find these simple things with uh with flow packets and again you didn't need every packet right if somebody spamming or sending thousands of packets per second you only need a handful to figure that out some other cool examples with flow data you get the tcp flags right so if somebody's on the network sending sin and fins that's a no-no right there should be no reason anybody should be sending a syn packet with a fin on it

flow data easy to find and then you know just the typical you know TCP scams not necessarily an expert but you know you could easily see Christmas tree scans and all that other kind of stuff and then another cool thing is send packets with data on them right you're not suppose is you are allowed to send data on a sin pack it's just that most people don't so that's something else to look out for us why is a syn packet why does it have data attached with with it now yes I said TCP flags so I apologize but I wanted to keep it on one slide IP headers again you have the IP header in

your flow data the size of the header if it's more than 20 bytes that's a that's a good question right you're allowed to have more than 20 bytes right but as our earlier presenters talked about like source routing you know that's going to be in your IP header well that's going to take up more than twenty bites your standard 20 bite header so look for those things those are simple this is all low hanging fruit I'm talking about right is what kind of packet is bigger than the normal 20 bytes so easy stuff to find with flow data so again pretty TCP flag you know just Wireshark you know look for the stuff that doesn't

look good now what's cool is an S flow they copy the whole packet and that's what you get in the packet it breaks out nice and easy like that in net flow you get a hex value and then you have to you'll get a hex value of you know 0 I better be smart but you know you get a hex value of 10 whatever that might be so you got to figure out which flags are which flags are set so s flow makes it really easy to decode another easiest thing that detect with flow data is if you have your different departments separated by subnets why is it County talking to engineering feels like this

is should come right out of a CCNA book Ryan you're learning how to set up a VLAN for the first time counting an engineering really easy thing to do you're just sending flow data and then all of a sudden summoning accounting is talking to engineering or vice versa that that's how it works it should be that simple some really really easy stuff to find and then I was really disappointed tooted it today nobody talked about virtualization yet right what presentation would not be complete without virtualization am I right come on who is not virtualizing today why you guys all right all right so anyway flow technology it's perfect for this right it's hard to stick a sensor in the

middle of your dull chassis you might want to stick something else in your dell or HP chassis but not necessarily necessarily your your tap what's that uh well I guess we won't necessarily describe it but you guys look like all unique individuals I think you can come up with to something on your own it will call the pen test all right pen test is good all right so it's really cool stuff right you get some flow data at the networking layer of your hypervisors in your virtualization technology and considering a fact I'm not aware of any hardware based switching technologies in virtualization yet you know it's still a software-based technology you really can't afford to capture every packet you

don't have the CPU cycles to do it so you need some sort of flow technology so VMware's using net flow which you know is fine you know Cisco's getting their way and they're ruining it you know putting their little whatever switch layer they have in there but anyway s flow is kind of cool it's all in the open source products so while xenserver is not open source but you know kbm and virtual box from which is now Oracle so it's built into their nests flow is actually adding some application structures so you can you know get like HTTP response times more of a networking networking flavor that might benefit administrators so think about it put your hacker cap on as we

wrap up here when you're looking at flow data all right the base thing to do is just think back to your third grade English class who what when where and why all right who are you looking for maybe what are you looking for when did it happen where did it happen and why if you start thinking those questions and apply those to flow data you'll be you'll be surprised what you could start to dig out you know like a simple things I wanted to know who was spamming easily well it's pretty simple stuff port 25 right and that's that's part of the limitations of flow technologies you know you're not necessarily going to see packet capture attacks right or not I'm

sorry application allaire attacks but you're going to see unique things happen like Lancope was trying to do data loss prevention right that's another bus terminal on virtualization right dlp well the way they looked at it with flow data is okay most people are downloading all the time from the net right they're typically downloading they're not uploading so a way to work with flow data set yourself a time frame right how much time should a person be uploading well essentially it should only be acknowledged acknowledgments they send out to their destination right so it should be pretty small so one thing you could do a flow technology to say okay within a 10-minute period our host on

our network sent more than ten thousand bytes of data in a ten-minute period well maybe that's a red flag maybe that's a deal p issue the problem is with flow you can't necessarily see what the data was but you get an idea that there may be a problem and then you go take another tool so just think about what you're looking for and you'll be surprised of what you can find so thanks I've got any questions or any discussion that's fine I don't say much on Twitter which is probably a good thing nothing important to say anyway and then shoot me an email so thanks any questions comments it was that good yeah yeah you

guys wanted last presentation I do you guys want to look like you want to kill me is my sense of humor just not there today you get what's that yeah yeah nobody's perfect all right well I know you guys are ready for yeah yeah yeah yeah trust me if they hired I better not say anything they're a good company good company thanks guys thanks a lot since we have a couple minutes here just want to let everybody know the