Logs and Tactical Defence

my name's Alastair Jelavic and I'm a log aholic I love collecting logs from anything I could possibly get my hands on and this particular place gives me that opportunity i work at the University of Toronto but I'm not part of the Citizen lab I consider the Citizen lab actually one of my customers I work in the core network I deal with a lot of the stuff that actually comes in and out of the gateways that would be the border to the University nothing I say here is to reflect their them these are all my own opinions you know blah blah blah blah I have some challenges I have a really large network I have a

really large user base last count I saw was 141,000 and change paid employees eighty thousand users you students that roll over approximately every five years several people outside the university so on and so forth my network is mostly open it's necessary it's a university network we do lots and lots of research we do lots and lots of sharing we have a lot of different things in there I have a motto that's become more of a mantra every make every model every vintage every skill set so if you can name it chances are we have it we have multiples of it and we have multiple versions of it and what I really discovered when I started here I've been here for three

years now is that you really don't know how much you rely upon certain things that you have everywhere else that suddenly you find missing because i have so many different things i have networks that don't have certain things like let's call them firewalls logs because I got 29 slides I'm going to run through quick but here we go vlogs you have them you probably don't have all of them don't worry about it the short answer is do what you can with what you got I will never have all my logs I can't get there however I look at what I've got I try to flag anything that's unusual and I search for things that cannot happen one of my favorite

statements from a developer is that can never happen that to me is a security assertion that means i can say that is an alert that i can set up and not worry about it because it can't never happen if it happens i know i've got something a tactical defense like to find tact a little defense as the thing you do the moment that you've seen the action i'm not talking about any kind of huge strategies here i'm not going to give you silver bullets these are not solutions that are actually gonna solve all your problems but they are actually going to tell you what's going on on your network and that's kind of cool the

other thing that works really well is the larger your network the more likely you are to get good results so sometimes this will work in your network some of these things won't what I've got here is 6 recipes for looking at logs that'll give you good information all right I got logs incidentally I do not have a single meme in here I didn't include any of them but I did try to give him catchy names okay so trial by fire wall very simple idea here if I've got a firewall up and that firewall is in front of a particular machine or a particular network and somebody touches a port on that firewall that I do not allow why

should I allow them to access ports that I do allow so if they touch 3389 rdp why should I let them go to port 80 so here's an experiment for you set up a firewall somewhere and block access to anybody that simply scans that firewall and log the fact that you've done it try that with one IP address on my network I was gathering between 60 and 100 IP addresses an hour you thought I was gonna say a day didn't you so this works tremendously well this works even better in my particular network when I set this up on a firewall as if it was its own honey pot I put it on a network on the

side but I do the block way over here on this other network that has no firewall so I'm tip I'm pumping an ACL out and then suddenly they're getting my protection over here and I don't have to change their network so I don't get any of the political arguments over you can't do that you can't block me you can't put stuff in front of me some of the actions I like to do now one of the things you've got to watch out for is whitelist 80 and 443 don't do it there unless you really really don't like search engines if you don't want search engines looking at your web sites hey go for it you'll block them out beautifully

don't think about just denying to two critical servers tonight a full networks tonight of the entire network you know have do the readings but log everything you do or get back to that this was my favorite doctor bad touch same concept except more on the honeypot eshe kind of side if you've got a web server that's listening on 80 and 443 how about setting up a couple of more ports how about putting in 22 on the outside interface or 80 80 80 80 s fun and then if anybody touches one of those block them oh there we go sorry now you can you can block them from that particular server you logged that action or maybe you go

even farther block them from your critical infrastructure block them from other machines whatever anybody know artillery by any chance so artillery is a tool written by trusted sec that does exactly this it's a Python script that listens on any ports that you define you run it on a web server automatically sets up an IP tables block anytime anybody touches something his is permanent it blocks it until you choose to unblock again 8443 but I like to alert on that if it's an internal IP address that's touched it and it's not already a web server so that's my smtp kind of stuff or whatnot now the other thing that I've noticed running this is if you put a DNS entry on it it's going

to get probed a lot and very very quickly and not all of it is wrong not all of it is bad so you gotta kind of watch out there I prefer to leave it unadvised also the attackers do figure out the IP real quick so rotate the IP on a semi-regular basis blatant 404 uh ever look at your web logs and you see all of these hits on all of these pages that don't even exist I like that top one there the acceptor password do you action that in any way know you say you look at that and you go oh it didn't work I'm good well the guys just gonna come back and do it again right he's

going to do it with something else he's going to try again the one that caught me off guard was the scans strict scans of my PHP admin they go through and they check every vulnerable version to see whether it exists and if it doesn't they kind of disappear for a little while until the new vulnerability pops up when they scan it again or they come back with something new the directory busting and hunting is something that's commonly done I see that a lot and can scanners they scan for everything so again you get a lot of 4 or 4 messages block just block them right there don't even let them get into your network don't let

them in the network nelena middle the web server why not pen testers hate me now index links if you've already got links that are on indexed in Google and people are clicking on that that can cause some false positives if you do not have a good cycle for actually creating websites because well somebody creates up a website they change it page has gone missing that its index now everybody's clicking on that and now you're actually blocking valid traffic be a little careful the impossible multi off another good one so this one is basically around the idea that authentication servers log where you're logging in from if you start actually correlating where people log in from you

start finding that anytime anybody logs in from multiple countries within let's say oh eight minutes of each other something's wrong so maybe you should do something like reset the password block the account and you have a number of things be careful how you implement the time frame for what you're looking for because well at least in my population I have a lot of traveling people so I find if I try to let's say do a 12-hour time flame if time frame it's completely possible to get from here to Venezuela in that period of time so you got to watch out for that kind of stuff you also need an accurate geoip database I would actually suggest going as far as a

subscription service on any of them to try to get that accuracy they actually implant bad data to see who's stealing it so go to the subscription so you can actually get rid of that two actions ticket it flag it reset it whatever you can do automatically contact the user out-of-band I have evidence that the attackers are actually changing filters on email boxes so that their stay detected undetected longer and one of the things here is don't forget to tell them to change their password anywhere else they've used that same password even if it's not one of your systems so if they're using the password on your system and they're using it over on gmail tell me go change that one too and

then you know maybe after that you might want to have the conversation of why they shouldn't be the same in the first place known local auth works really well in here so if you actually do happen to have logs for your doors you can actually figure out hey this guy is on site but as we learned today with some of the keys the the door locks that could be compromised but you know what if the guy's not actually in the city and you see local on site you still know something's wrong so deal with it you get to get some good stuff going there the questionable single source is another way another angle same log data

if somebody's logging in if you get multiple multiple logins in a short period of time from the same IP address whether they are successes or failures somebody's trying to do something I mean these are obvious things right but nobody's actually actioning them that I could find so you find that you get a whole list of them you can either tag them reset them or do whatever you feel is necessary to be able to clear that problem if it is a problem watch out for NAT watch out for tour if you allow it at all and watch out for proxies the place that the big one that I catch here is when people start signing up to other

services and some of the phishing attacks but I got better for fishing so here's some actions for the questionable single source oh yes by the way repair respect the privacy of your user if you start going at them hard and saying hey I saw you log in from here and then I saw you log in from here they're going to stop one and giving you stuff and sometimes you don't necessarily want to know that your boss just logged in from his Secretary's house that's my boss fake fishing so this one is actually an idea that my coworker came up with that that actually works better than I ever expected we get the phishing attacks just like everybody else we get them in

volume we get non-targeted we get targeted we get every again make every model every vintage what we started doing was they actually provided fake credentials to the fishers we said here here's a user ID and a password go log in and they do now what's been happening in the past is they take the entire list that they fish that they think they've successfully fished and they will then try to login as everybody to see what works quick story I had one group out of India that did exactly this they sent a phishing attack they tried to login as everybody I saw those I reset all those passwords then I watched the Russians come back in and

try to login as all those same accounts and nothing worked what do you think happened next that same Indian crew came back and started doing another fishing run another campaign a lot more sloppy a lot more loud and I don't know maybe I'm over imagining this but I've pictured them having the gun of a Russian in the back of their head doing this work you can also tag on other things sometimes your users are your best source so scam and and other things that people will put likely into the username field is a great thing to trigger on if you haven't seen the phishing attack in the first place this recipe does seem to have a limited life span I can actually

watch them changing their behavior as time goes on there trying to figure out what I'm doing you change that credential use you change the IP space you use you change when you reset them so don't necessarily go in and reset them that moment that it's happened give it an hour and then reset them so a bunch of other stuff also deny the IP address that they came from that's another really really good one it's also fun because then you can actually see what other IP addresses they control because they'll come back with the same list again respect the privacy of the user themselves others so that's kind of my big six those have been great others

that i'm looking into net flow my coworker has been doing a lot of work on net flow he's got some really really good stuff i've yet to pi it together and a correlated way with some of this stuff but i can already see that's what it's my first stop to find out if anybody's talking to a cnc if anybody is contacting any outside service or nefarious earl or anything along those lines refer Earl has has some promise I see lots of garbage coming in there on the few and I mean few websites that i have out of the several thousand in my network signal object HTTP if somebody's loading only that one title gif in your

website take a closer look because sometimes that site is actually or that image is actually being sourced from a site that's not yours but just pretending be you so that's might be the start of the fish DNS for all the normal reasons email logs for all the normal reasons robots.txt this I took directly out of the book of offensive countermeasures in your robots.txt file put a directory that doesn't exist and then watch your logs it's great when they come back and they say hey you got a director here called secrets I'm going to grab this and they try to touch that directory it doesn't exist you get a log you block the IP everybody's happy round doom now

you've been logging all of these actions that you've been doing how about we look at the logs of locks and what can you could do with that first thing that I started doing was aggregation because I'm dealing with such large networks and such such huge spaces I started actually looking at which I peas are the loudest which ones never stopped and which ones can I tie together if I start taking all these IP addresses that it attacked me over time as I block them they start coming up in other IP addresses can I start putting together contiguous IP space and the answer is yes yes you can now these actors are a little bit different than most they're not the

average attacker they're not the guys trying to make a quick buck at least as far as I can tell I don't know for sure but all i know is that i found a / 18 that as far as I'm concerned is completely weaponized and I blocked the entire thing I've never seen a successful connect from them so why would I allow them incorrigible users you know you ever had that interview don't give your password to the tech to the help people they don't need it and their answers but you just asked for it this morning it's a Christ so you've got these incorrigible users or and you don't necessarily know why they're getting compromised all the time maybe

they keep falling for the fishing maybe they're just choose passwords whatever it is but you start seeing these things when you've got all the lists of the IDS that keep coming back and you start having the ability to actually go talk to them in a friendly manner be nice so that you can actually try to figure out what the problem is and deal with it at the source hot spots this goes for countries this goes for shared IP space this goes for a lot of little bits and pieces the hot spot that popped up pops up a lot for me is the open Wi-Fi well the paid for open Wi-Fi at the airport so anybody anytime

anybody's traveling if they hit an airport Wi-Fi it's worth keeping an eye out from my point of view because in a lot of cases I've got so many travelers that just statistically speaking they leave their password behind and I'd like to be able to get to the point where I can actually say okay here's 1800 people let's put together some sort of list of where they're all came in in common and we've got the first round code for that but what we're finding is although it works great at a single IP layer it doesn't work quite well enough at a kind of area layer so we're working on our next pass to say here's an IP address

tell me everybody that's logged in from this radius around that IP and then pick them to pick a random wild number again requires really accurate geoip we'll see how far we can get with that again deny the car deny access crittle systems maybe for just that user if you see a user get compromised or you see that hotspot slow them down confuse them funnel them do whatever you have to do and your research in your sharing I can share I am in a university some of it is considered proprietary and I got to be a little bit careful but on the on the slightly more open side there's things I can share but it would even those things

I can't share i can still research so anyone else talking about the IP what are they doing with it what do they think of it as somebody already reported it all of that kind of stuff also this is kind of where you start thinking about things like showed an shadow server and now sonar where we heard about earlier is that something you want to block maybe yes maybe no i personally will be so let the results help you to guide you on what your final decision is around 3 action reaction you're not allowed to deny your manager your boss comes to you insist you don't you dare block traffic to our site well there's other things

that you can do quarantine works true a tremendously well pick a period of time quarantine them for that time and see what happens if you do this with an alert on the tail end and watch them pounding their head up against a wall it's actually a lot of fun but if not you know it stops them for now reduces the noise so you can see what's really going on because a lot of this is about that reducing the noise redirect redirect asst can be super fun if you can implement it properly it's a little bit difficult implement the redirect so that any time that that IP comes back into your system you redirect them into

something safer maybe you've got a clone of your website and you know what you got now you've got a volunteer pen tester and since you if you're collecting all the logs off of that box properly you don't even need his report and that means you don't have to pay him when you're working for a university that doesn't have very much budget in the first place this works really well when you start tying in things like iptables you can have even more fun if you set up IP tables and masquerading mode on the inbound and redirect them there it's completely transparent to them so like for instance if I see a touch on a port 22 on any of a number of

critical boxes I can redirect them to the port 22 kit bow on my honey pot and then it looks like I've got 11,000 and change machines but they're all actually the same machine running kippah and then I can do some analysis from there more to come on that still playing with other things mod proxy is also a possibility even just a simple location tag in your HTTP server if you want to go that way a number of different ways use your imagination don't ignore your wait lists be able to whitelist anything you're going to need it you want to whitelist the things that you know are going to be touching these things your own vulnerability server any health

monitoring you do you just want to risk that stuff going down because you get the wrong people yelling at you right bus maybe you actually wait list the the actual your boss's IP address as well just to make absolutely certain that everything is claimed so when he wind up when the boss's boss calls him he goes hey it works for me also just because you're white listing it keep alerting on it because it'll be useful later and keep logging that information now this one one for fun I was playing with the HT access and putting in some extra rules into an Apache web site and I realized that one of the things that I can do as a reaction to any of these

people miss visiting these I can make the entire website go away just for them so I can actually make it so that if they touch the wrong thing do the wrong thing or whatever suddenly every page on that website is returning 404 if you're brave do the opposite of this make it so that every page that they visit return something and there's an actual program called web labyrinth that does exactly that it's actually if you want to take it that far it designed to trap bad agents bad search engines so that it generates up a page with a whole bunch of random links each of those random links is clickable that generates up another page that goes it just keeps

recursing when you throw certain scanners against that they die from lack of memory and boof the websites gone magic the after action action I couldn't come up I was getting too tired I couldn't come up with cool names anymore always be able to provide evidence so somebody comes to you in my particular situation somebody comes to me some researchers and says you blocked my traffic you did this to me and I can step into that room with reams of paper but really only the front one matters and say on this date at this time you touch this port that you shouldn't have been on and then you did these three things and then this person from Vietnam

decided to try to login as you and succeeded yeah I blocked your traffic usually they have nothing to say after that be able to release the IP at a moment's notice because sooner or later somebody's from high enough up is going to say I don't care take me off that list so take them off that list but at that point you can actually open the conversation if you're polite and professional and all this things so you can say things like this is why you got blocked this is going to happen again if you do it again and then you do this these are automated systems I don't have any control do what you can with what

you have where you are pretty straightforward none of this is magic but i don't i see a lot of people talking about it i listen see a lot of people writing about it i don't actually see a lot of people doing it and I'm sure that you can come up with a hell of a lot more interesting recipes than i did hear ya

loaded question Thank You Stephen I my job is to protect the internet from our population the population from our internet and them from each other I do not really differentiate I try not to differentiate and i do not have exact numbers to back up that in any way i concentrate a lot on the outside at this time because i'm actually much more interested in the threat intelligence side of things at the moment but i will be getting into that sorry no i know it's not the answer you want but that's the best where i'm going to give you anyone else how we doing for time 1 minute 1 question 1 cool thank you very much