DNS Hardening - Proactive Net Sec Using F5 iRules and Open Source Analysis Tools - Jim Nitterauer

all right hi everyone and welcome to Proving Grounds so I'd like to start out by thanking our sponsors especially our Stellar sponsors verse Sprite productivity tenable Amazon and source of knowledge so this track is being recorded and uh we're also streaming uh at the end we're going to have a Q&A session and I'll be running the mic around so just please bear with me and uh wait for the mic uh our next talk is on DNS hardening proactive network security using F5 I rules and open source anal analysis tools from Jim nit hour there you go got it perfect I got it perfect awesome very good very nice all right let's uh Jim works is a Senior

Systems administrator at app River uh please join me in welcoming Jim I'm only here to speak I'm not here it's not here for me all right so I had trouble with the projector yesterday so I'm going to roam around hopefully this laser pointer will work hopefully everything will go well my name is Jim N hour I'm a Senior Systems administrator at app River I've been working in app River since 2006 first I want to thank my mentor Dave Lewis back there thank you for your help and assistance I appreciate it and I want to thank besides for inviting me to be here today how about a big hand for all the volunteers who put this

on everyone that deserve PR all right we see if this works I hit the wrong button already there we go so a little bit about me how did I get here well I went to no 2015 and was sitting around talking teaching people about a few things that I was doing at app River a couple guys overheard me like this says here and they said why don't you come speak at our next event well so apparently you say a few things and so people over hear you you're a security expert so realistically I've been with app River uh since 2006 this certificate is not mine because my name is Jim I'm not Jack uh when I came to that River uh

in 2006 I started out on their uh security team right now I'm in charge of running their global data centers worldwide we have 12 data centers that we run all of our servers out of and eight offices globally we run a secure tide which is a spam filtering platform and secure surf which is a DNS uh filtering platform the opinions expressed here are not necessarily those to my employer they are mine mine alone so if you have any trouble with it come after me not after all right so today what I'm going to do I want to lay out challenge what we were faced with I'm going to tell you st story about how we

went and put together a solution to secure our DNS infrastructure and when I'm focusing on DNS here I'm talking about our secure surf DNS infrastructure and just the fact that it's just think of it as a DNS cach resolver and don't worry about what the service does other than it resolves DNS any of the things that I mentioned here the different um platforms I'm not endorsing any one of them we just pick them and we use them for our platform so I'm not here to say one vendor is better than another you just use what works so I to lay out the challenge that we fac I'm going to examine some of the security flaws that

we ran across when we brought on our service I'm going to look at some of the tools that we use to solve those security tools or those security issues I'm going to assemble those pieces into uh to show you how we put them all together I'm going to show you some of our results and we're going to discuss some of the future possibilities now that's a lot in 20 minutes I'm not going to go into a lot of detail about some of this stuff a little bit later after the talk I'll put some more technical blogs at tripwire and on the P list so the first thing we start with Basics before I came to app River they had

version one of their secure Serv service they rolled it out the service did what it was supposed to do it secured your DNS the problem was it took about 500 to 1,000 milliseconds to resolve DNS query so if you really like your DNS slow that worked well it didn't work too well so they started from scratch and rewrote the whole service when I came on board they were about to roll out the service for the second time and what we ended up finding out minute we rolled out the service was that when you put a service out there that anybody can connect to you run into all kinds of problems right so what they ended up doing at in our

DNS service our DNS service works at application layer 7 obviously and a lot of people will secure their DNS at layer 7even basically by setting rules where you block IPS and only resolve for certain domains that sort of thing but we couldn't do that we had to work at layer three layer four layer five and layer seven so what happens if you wait to secure DNS layer 7 well you end up with something like this right you're going to blow up your DNS servers there's so much traffic out there there's so much malicious traffic you're not going to be able to support security for that platform so what we ended up doing was developing a plan to figure out how we

could mitigate some of the challenges without closing that DNS we the reason we didn't go with a whitelisted IP based DNS Services because some of our customers are actually remote customers so they move around they have dynamic IPS they small businesses they don't always have a fixed IP they don't want to have to log into a captive portal put in their IP address and wait for that information to be propagated back to the service so what I'm going to go over today are the basically the security flaws we found in DNS that you probably see in your own environment and I'm going to show you how we went to solve those problems in general detail and if

you want more specifics feel free to come find me I'll be here the rest of the week I'll be at deathcon be glad to sit down and go over some specifics I'm not showing any live demos today CU we all know how those go when we do those in talks all right so the first thing we came across if I get tell me DNS amplification attacks who's seen these on their Network have you been a participant or a victim SL right so there a man in the middle attack where somebody Spooks an IP address sends you a bunch of small packets asking for a bunch of big packets and sends them to the spoofed IP

address typically a botet network rent time on them they'll do these to the tune of several hundred thousand botnet members over the course of 15 20 minutes your DNS just doesn't work okay DNS amplification attacks are a big deal they really will bring your DNS down or make you look stupid from participating in them so you have to be careful about these the next problem we face is kind of an interesting one we saw being a Spam and virus filtering company were very aware of what's going on in the botnet networks who's sending out spam and those sorts of things but we saw this in our DNS there's a a tactic that the malware developers use called domain

generation algorithms and what they do is they're built into their malware and they generate these randomly generated domain names along with the domain names that are real and connect to the botet community control networks and they do this on a regular basis so that what they're trying to do is obate their real DNS command and control in this scattered traffic well the problem is on a small Network you may never see them on a big Network like what we're doing we're doing 60 70 880,000 DNS queries a second on our Network these things become problematic because they generate a ton of NX domain lookups so your DNS servers become very slow and don't respond very quickly the third thing we

saw um Bad Name queries in our DNS so everybody here I'm sure your DNS never relays a DNS request outside your network that it shouldn't right of the big ones there's one on here that you all need to be aware of I think it's listed here wpad queries if you don't have those shut down on your network I advise you to dig through the internet and figure out how to shut those off immediately if you don't have a proxy server on your network these are bad news your users can go and this isn't aside they can browse and Wi-Fi anywhere else it will send out these queries somebody can spoof DNS queries and reply to them it will set the proxy server on

their browsers and send all their traffic to that proxy server okay disable that if you don't use it in your domain take me seriously on this but what we also saw were some really strange lookups I don't know if you can see that there's one there there's a couple others down here some pretty poor malware writers what they were doing was using these domain generation algorithms to generate domains for B command and control traffic but they weren't smart enough to make them fully qualified domain names so they just end up creating a lot of havoc on it so we saw these in our Network another thing we saw when we started to examine gns traffic and initially we're examining

all of this traffic using wire shark and some other capture methods we saw um malform DNS packets there's two kinds of things that we saw in this one of them was Mal form packets designed to basically deos your DNS server bring it down and the other was really interesting I don't know if you know about DNS tunneling where you can actually tunnel other protocols through DNS it's done by packet injection and the packet headers that go through your DNS server and you can see that in your DNS requests if you look at it on packets so this was a problem these kinds of traffic should never never ever reach your application servers they should be blocked at the

edge the next thing we saw was um data extration via DNS this is pretty slick so the nefarious people register domain like ps78 was one of them then they go and they do drive by malware download send you some malware via a banner ad for example installs some malware on a computer starts generating traffic looks for the data that it's looking for on that infected machine as soon as it finds it it starts taking that data off the disc and creating this encoded subdomain well the problem is that all of this is legal DNS right goes out to the route finds the name servers goes to the various name servers it returns an IP address the other thing it does is it

takes all this data here at a subdomain and reag back on the nefarious side so they're exfiltrating data this is what this is data being exfiltrated from an effective customer so you can catch that in your DNS and we were able to mitigate that through some of our f5i rules another thing that we saw because we didn't want to close our DNS resolvers down is we wanted to be able to block IPS that we knew were bad there's several lists out there this particular one uh is a drop list from span housee it's free you can subscribe to it you can download whatever you want we wrote a little C program to put it in the

right format this is the format that an IAL data group takes but it's basically just the data gotten from spam house put in the right format we're able to have this data stored in a central location and then distributed to all of our F5 load balancers globally with um a bash script and chrome jobs that are run every so often on on the F5 load balancers so we're able to aggregate these we get this data from various feeds including our spam filtering information the last thing we saw is um actually an interesting way that you can use DNS to D do so several probably year and a half two years ago most of the major DNS servers put in a feature that

allow you to force certain queries to be reass over TCP so somebody wants to do a DNS amplification attack they hit your server with an any request on 53 your DNS server says I'll be happy to answer that you got to ask on TCP Port 53 well the bad guys aren't going to initiate a DNS amplification attack against you over Port 53 TCB because then you know who they are but what they will do is they will spoof the IP address in the TCP packet and send you a boatload of queries directed at the malicious IP or the uh the IP address that they put in there the wrong one so what do you do

you end up initiating a synac flood against the target IP address so one fix for vulnerability creates another vulnerability in your DNS so be careful about these in Windows if you run Windows DNS anybody know what the timeout is for TCP Timeout on Windows five minutes five minutes that's ridiculously stupid nobody needs five minutes of uh time time out wait time out you can change that in the registry the lowest windows will let you set it is 30 seconds but set it to 30 seconds that will protect you quite a bit all right so this is we saw the last thing we saw on our DNS where DNS floods this was kind of interesting when we first

brought the service up several years ago this was a big deal people would just try to flood the crap out of it with these DNS request the latest one happened um last year I think it was October or November anybody remember what happened to a particular magazine they made an announcement about the content that they were putting in their magazine the next day the unhappy people dos Ultra DNS for about 4 hours in the afternoon took them offline this was Playboy Playboy magazine so anybody Ultra DNS that afternoon had a really crappy afternoon all right so what did we use so I'm going to go over the pieces that we used to put all this together first

thing we have are F5 load balancers these were in place when we got there we're just working with the tools that were in place so basically an F5 load balancer has a public facing IP address it's an endpoint for a service could be email could be DMS whatever uh it's called a a virtual IP address and it's load balances to a pool of servers in the background in our case would be 4 53 here and DNS serers in the background it also supports something called I rules which are very cool I'm going to go over those in a minute it lets us actually monitor what's going on with these and load balance across many um servers in the background has some

other features it uses an operating system called tmos stands for time managed operating system nothing fancy it's just F5 operating system and it has the ability to do tmsh commands which run functions within that operating system another cool feature that it has are eye rolles now this is just a view of a sample eye roll in f5's free I roll editor that you can download online so IR rolles allow you to manipulate traffic at the application or network layer both inbound and outbound and do things with it and it uses a language called TCL tool control or tool command language I think that stands for so anybody remember TCL a long time ago well it's

back all right so what we we're able to do threat feeds and I spoke about these a little bit earlier that's the address you can go to that one and look up that particular threat feed um if you have a router or something a firewall that takes in these IQ feeds I would go look at this one in the E drop and just drop that traffic from your network never let it touch your network there's one called drop and there's another one called e drop it's free list that you can download so I was hungry for bacon this morning but I didn't get the breakfast in time so I was waiting for my badge so

I put that in there the other thing F5 allows you to do is remote logging and this is where we kind of put the pieces together F5 lets you log locally obviously but you don't want to do that anytime you're logging locally on a public facing device you're creating iops on your disc slowing your device down so what this allows you to do a set up PA to remotely log all of your data we remotely log all of our data to Rog brog is a choice that we did it's basically the open source alter it to spunk people spunk we chose not to use it we generate so much data the cost for us to get involved with spunk it's a

great product but for us we needed something that we could manage a little bit more the volume of log data that we have is tremendously high so gry log is you can run it as a single machine a cluster of machines uh put the cluster behind an F5 load balancer send all of your data to that F5 load balancer load balance it across the gry log servers PR log has inputs that ingest this log data takes that log data and WR it into an elastic search cluster that lives behind this so we're using elastic search as well the elastic search cluster then indexes all the data based on the fields one of the cool things about gry log is

that it gives you the ability to import data in a format called gelf and I'm going to go over that in just a second what what that is so gelf like elf but not a gnome or anything like that the other thing allows you to do is write custom Java plugin so you can parse data when it's coming in so how many of you like kiwi anybody remember what Kiwi is sis log server was good for its time but it's hard to get information out of that right because it just takes straight up CIS log format and dumps it into a file with the volume of stuff that we're doing it craps out very easily the

elastic search gives you the ability to do flexible search you can format your data in certain ways I'll go over a little bit of that here coming up the other thing that greylog has now it's pretty cool is it lets you from your windows or Linux boxes you can ship your logs directly to it and it has a feature that's called um greylog sidecar it's a application you install on Windows server or Linux server and it manages either NX log or log stash but the cool thing is it reports back to your gry log so through your gry log web interface you can manage all your remote endpoints you can tell it which files and logs you want

shipped back to your gry log servers so NX log is basically think of a log route you point it to that contains your logs it ingests those logs it will do a transform on them put them in the format you want and then it will send those logs onto whatever input you tell it to grabs the log does it thing sends it off that's all it does it's a mid man so what we use this for this this part of it is because we wanted to actually look on private networks what was happening in DNS for customers so in DNS most of these customers have active directory DNS servers active directory DNS have the ability to Output your log

information in the debug logs right it's in a very crappy format it's very difficult to read and it's also difficult to make it rotate now if you want to know how to do that I can put up a blog post about how to do it right so your logs rotate and everything else works the way it's supposed to because what happens by default in adns if you put on the debug logging it will keep the file up until whatever size you set it at 500 eggs 50 gigs whatever size you set it and then it'll delete that log and start again so all your data is gone with Powershell you can set some functions that will let

you roll those logs over and keep those logs for a period of time so gelf gelf is called is short for gry log extended log format it sends data in a Json formatted packet the first parts of this packet are required for gelf format the last parts are the cool Parts where you can actually parse out and add your own fields which we did to a high degree and I'll give you some examples of that but you could add 100 Fields here if you wanted if you can log if you can grab the data you can put it in there if you do any net program any other kind of program there's guil libraries available to do this or you

can do it natively which I'll show you in a second in our F5 I rules what we did is we actually told in TCL this is how you would write one line of code to send one log message to uh through the F5 in gelf format but it's pretty straightforward there's your bracket all the information you need here's the fields that we were adding right so we're adding these fields down here at the end all that data then hits a gry lob server in gelf format there a gelf input on a c port and it puts it right into elastic search the last thing we just started experimenting with in how to visualize our data is Cabana I'm not

going to go into too much about this but it links up directly to the same indexes that are created by grey log in your um elastic search cluster very cool solution so last part of this puzzle is something called critical threat notifications this is built into our secure surf so basically what happens is if a customer hits um a domain that we know is either part of a bot net command and control Network or is a drive by download it will trigger one of these alerts and it'll give the domain that they hit and how many were blocked and I've taken some of the data out of to the customer data and the IP address and

all that but tells you what policy and everything else so what we do then with this is there's a time stamp on this and we can go back to our data and actually find in the data where that is and I'll show where that infection is and what machine on the local network is infected and I'll show you that here in just a second so real quick overview basically a customer from a customer perspective they'll have a DNS server or multiple ones their DNS servers are set to forward their DNS requests to us when they forward those requests they hit our DNS VIP we have an I rule in place it's basically 900 lines of code

several sections each one of those sections addresses one of those vulnerabilities that I talked about earlier the packet passes through those Cascades through that ey Ru if any of the rules trigger a block then the packet gets dropped but everything is loged to our gry log cluster so we can see right away what's happening when somebody does a DNS amplification attack for example the uh F5s then have Chrome back script that go out with tmsh and hit a web server that's in one of our data centers and it pulls out the data from those threat feeds it does that every so often so if we find over here that a particular domain is creating a DNS

amplification attack for example and it's not in our threat feed we can add it and within 5 minutes it's globally blocked on every DNS server that's out there so it's a very quick way of pushing data to a whole lot of endpoints very quickly so let's look at a little bit of the information we get out of this this is a gy log interface the fields that are coming in are over here uh the time frame is up here the query is up here this is actually a histogram showing you per minute uh this particular one is a DNS amplification tag I don't know if you can see the domain name here but that's the domain

name and it shows you how many are coming in I believe that's about a th000 per minute coming in and that's from a very small attack coming in globally now now I know that I happen to be blocking these but this is just the number of queries coming in so we record all the queries that are coming in we can actually split that out in uh Ray log you can take and expand one of the fields click on quick values in this case you'll end up with a list of the top values in there there's that domain name that was doing all the nefarious stuff over that time frame it made that many queries it was that much percent of

our traffic if I want to narrow it down and click that button then it'll narrow the query down even more we have dashboards that run in our um network operations center where we can manage this this is showing an hour view so over here these are all the in this case DNS any queries that are coming in these over here are the ones that are getting blocked so what we can do is we can compare the two and if there's something missing over here we can add it to our iroll data group have it pushed out and have it blocked pretty quickly this is an example of uh remember the spam house list I show show

you I think that's the name of the list up there these are actually all the blocks of people trying to hit our DNS servers coming up here um so there's that many hitting it and these are actually getting blocked so we can tell that our blocks are working this here showing Network compromise there was a particular domain that we saw in in a CTN we were able to go back to our public facing VIPs and find out the two customers that were generating those by their wi IP addresses contact those customers and get them in the process of cleaning up their Network this is examples of the data we get out of the DNS debug log these are the

fields that we created in our um custom Java plugin that takes the normal active directory DNS debug logs and breaks it out into useful information not the craft that Microsoft has in their DNS debug logs you can actually search it find out what's going on I have all the fields blocked but you could turn on the source and destination IP narrow it down to a local internal IP and see all the DNS traffic that a particular user on your Network's using so if you want to spy on or find out what your users are visiting you'd be surprised if you put this in in place you could build a VM and point the stuff to that and figure

out what's going on very quickly I only have a minute left this about about two slides out this is just some more debug data same thing and the last thing is we could actually take that debug data narrow it down by the domain name I broke out I didn't show the IP address but this shows one machine that's compromised with 18s queries over the course of that week we're able to tell the customer that machine on your Network's infected take it off the network fix it and that's the kind of information you get another thing we can do is we can look at where infections are coming from it has built-in geolocation once you install the

geolocation database we're able to tell where any kind of data that IP base is coming from so there's a lot of flexibility in this um we're about out of time so are other possibilities that we can look at we can look at some of these other things and create rules create anything that we can do to export dat and anal can be looked at from a security perspective so I know I gave you a lot of information it was pretty high level and if you have any questions you can reach me there and we're good to go great thank you