BSidesMCR 2018: Adventures In WAF by Michael Thompson

okay thank you very much for coming I'll start with a little bit about Who I am so I'm Mike hi I'm also known as absolute bloke on Twitter quite active on Twitter thanks I do tweet a lot and about all sorts of things can be anything about current topics current events in information security are you engaging quite a lot of debate around some sensitive topics like why we should have a fully HTTP web I firmly believe we should there are others that don't agree with that and I can get a bit grumpy about it from time to time generally in a good nature can away I I write blogs a little bit more detail shortly my blog out upset bloke

and I'm also a helper so I recognize a few people in the room I like that so greatly on the pleasure from seeing other people flourish and develop and I like to get involved with that in a selfless way I just think that there are good things and good people and I like to help you do all that now I'm also a speaker he can probably tell it's a bit of an interesting journey for me my first ever information security conference was besides Manchester last year 2017 so I turned up completely out the blue had never been to a conference before in this industry just as a punter and a bug it kind of caught me and I

felt really at home felt like I was part of the community for the first time have been involved in the information security industry for a number of years but I just felt like a guy who works in a company that does the security thing and I never really had the opportunity to reach out to other people and this conference was the first time I had that opportunity and it made me feel great I kind of influenced me in my decision to get involved in blogging I came right out of this conference set up a blog and wrote a review of this conference and I tell the world what what I thought the conference and how it made me feel and I

felt it was a life-changing event it then encouraged me to become a lot more active on Twitter I befriended a lot of the people that delivered talks here or who I'd met here and we've now become really good friends as a result of that and then I thought actually I've now been blogging for quite a while I'd like to set the message out in a less static way reading my words on my blog have one thing but me being able to go out and give actual verbal experiences or testimonies about my experiences and what I've done I think can be more impactful sometimes so I made the decision back in the last year that I

really wanted to get involved in talks and I thought well I enjoyed oh wasps oh man I enjoyed b-sides Manchester so much that I was encouraged to apply for b-sides Leeds didn't go didn't get it a big one but went along nonetheless really enjoyed the conference one of my colleagues in the in the audience came on with me I think he'd echo the fact it was we have a really good time but I wasn't put off I did get invited to do a lightning talk at the event on the night before the conference I then applied for b-side Scotland in Glasgow I didn't get that but I was offered a reserved speaker position there which was great because I was

invited up there and I was treated as if I was part of the crew delivering content on the day again met some great people and again had a really good time at that conference again it didn't put me off I continued and then I put speculative application into b-sides of London I actually got that gig and that was in June this year that's the talk I'm going to be giving for you today so when doubt London gave the talk really enjoyed it and that was it I've done it broke my duck and now I'm really I really feel like I'm part of the part the thing I did a talk for OWASP in Manchester last month and that was

really good this talk once again meanwhile I had it application in to speak here but hadn't heard anything and on Twitter I started see seeing people going yeah got my talk accepted it be sighs Manchester I thought well haven't heard anything yes or no delivered the talk it happened that Mark one of the organisers of b-sides was at that talk at the end after questions were answered he said this isn't a question it's more a statement to come and talk at bayside Manchester that was pretty much it was the best moment of the year as far as I was concerned because what that meant was I'd gone from attending as a as a punter at a conference just happened to be this

one to being a speaker at the same conference within a year and I feel really really privileged to be here giving this talk for you guys today so I'm I'm an infra sec guy an ISP a local ISP won't mention the name but tell you about who they are afterwards and I'm also a member of Oh wasp so quick show of hands who's an O wasp member one two two okay keep your hands up because you'll know but a show of hands if you know a wasp is okay quite a few more for those that don't know it's the open web application security project so it's a global foundation of application security professionals and academics and

that come together and share great experiences do good things get good projects off the ground and bring that back out for the greater good there's been a lot of debate on to it which I've been actively Roldan around its reach out to getting new members that's quite indicative of the to show two hands three including myself in this room they're not doing a good enough job of getting the message out and getting people back in and I think that's been acknowledging that the board of director level and they're gonna do a lot to try and make that make a difference that I'm a paid member of OWASP so the two guys let's put the handle P but they paid

members over wasp

so being a paid member will cost you $50 a year as a professional an individual about 35 pounds if you're a student it's $20 a year which is about 14 15 pounds for that you get to contribute money into projects and chapters then get money back so they can put Knights on the input talks on and then share a lot of the the knowledge and experience that's happening more local level but what you get in return is pretty cool too so for your 35 pounds a year or 14 15 pounds a year you get an OAuth lecounte now that's pretty cool you get a Michael Thompson Oh waspey dog email address and I kind of like that because

it adds a little bit a little level of authenticity to you as an in person so I've consolidated down a lot of email addresses I would have used previously to communicate with people and I'm just now use my old wasp account kind of handy but you get a G suite account as well now that's pretty good G suite account is a cool thing it's a really expansive group set of legal tools including mail but you get unlimited storage with it which not a lot of people realize wasp is somehow managed to negotiate with Google to get this huge blast endless space of storage on their on their network on my platform and so you can

use that to put anything there's no restriction on what you use it for and it's endless storage and endless cloud storage and you also get the benefit of filling the part that you're contributing to something that's pretty cool globally but again there's a lot of work to do within the foundation to attract new members and I think that's something they recognize about I'm gonna skip that one because it's a bit boring okay why are we here so who knows what a Web Application Firewall is okay pretty much more or less everybody in the room who's got one a few people skewed figures here because some of the people in this room work with me but not too

many people okay what is a Web Application Firewall a what so it's a layer of protection between your software on the web your assets and on the bad guys and I'm therefore really a layer of protection against intrusion into your data via web applications so we're thinking a wasp top 10 now every 3-4 years Oh what get together and they take data in from the the world so what kinds of attacks are people experiencing and they put all that data together and they'll produce a top 10 now it can be misleading the top 10 isn't the only 10 web application vulnerabilities that are out there these are the ones that are considered to be a more common easy to execute

impactful and have really severe consequences if if there is an exploit so we have a top 10 typically you'll see in the top ten at number one injection so sequel injection command injection and so on and so forth things that allow an attacker to modify an underlying database or an underlying operating system that's at number one it's been at number one as far as long as I've been involved in information security and then you get other things like broken authentication sort of things like having cookie theft and hijacking session hijacking and all that kind of thing then you have things like cross-site scripting cross-site request forgery has gone out of the top zone recently it's been kind of merged away

because there's a lot better mitigation but those are the examples of the kind of threat that the OWASP top 10 talks about they're the kind of things that a wife will provide protection against in the way we've implemented it it's also a method of controlling access to your web applications so we control the DNS of everything that we have on the web and by doing that we're able to spear traffic to wherever we want it to go come-on for that in more detail in a bit we're able to monitor a lot more again I'll talk about this in more detail but we've gone from a position in the past of being not being able to mine as much

so now being able to monitor practically everything so and it's a legitimate component of your web app arsenal so you're here again on Twitter I get grumpy where people say well a laugh is purely a compliance tool okay it's there to say our apps are secure nothing of the sort it's not a compliance tool it's there as a layer of protection okay it's there in the same way that server hardening is important it's there in the same way that secure coding is important it's got a legitimate role so what life isn't it's not at all the solution so I've touched on that in a previous slide it's not the be-all end-all it's not a way of saying well we've got a watch in

front of everything so everything behind it it's cool it certainly isn't a reason for dance to say well whatever our software can go and whatever condition we choose because the RAF will protect it and I'll give you a little anecdote you don't know is that we've got a fairly big dev team where I work and quite a large architecture team and not many of them really if any know what the wife is doing okay they know it's there because we've done this kind of talk for them but we were not going into explicit detail about what it's protecting against but there's a reason for it's quite deliberate and the reason is that while ever they're not entirely

certain of what's happening in front of their software they're not encouraged to be relaxed about securely coding it it keeps them on their toes and I work very closely with developers around secure coding so what kind of bringing that all together into a a kind of a harmonized approach to securing our stuff it's not an excuse not to patch so we see a lot of the breaches that have happened in recent years in the last 12 months have been around not keeping software frameworks up to date so we know that Equifax got got attacked via obsolete version of Apache struts okay patch was available several months ahead of the attack the knew about it but they did

nothing about it and they were attacked a wife might have prevented that because it wouldn't have been able to access the underlying framework it's not an excuse not to patch the stuff though and keep the components up today and it ain't an opportunity to around your boss and go yeah oh we've got a wife guys what's they're doing stuff and a pay rise we've made improvements to the laughs can we have more money it's not that sort so how did we arrive at this this enlightenment okay well we'd we'd had a bad number of days actually in autumn 2015 September 2015 we were taken off the internet for about two and a half to three days and that wasn't just a

website that was our entire platform marketing web site customer portals diagnostic systems contact systems everything had gone we were suffering the biggest denial of service attack that we'd faced as an organisation in the time that I've been there so that was a mixture of TCP syn floods okay UDP and HTTP attacks we were being attacked by putting WordPress helps they were pretty easy to mitigate against because that was HTTP and we were the web guys we could see that and do stuff with it but the TCP and UDP that was networks we were really vulnerable to her and we dealt with it but we dealt with in what didn't feel like a sophisticated or a

repeatable way and that didn't feel good enough so we needed to do something about it so we had a bunch of meetings and this was even board level and the message was we can't this happen again we can't have a repeat of this we need that critical webapps up we're an isp if we're not on the internet that's not very good is it it's bloody rubbish so we need to keep our critical Web Apps up we need to block malicious requests pretty obvious we need to stop access to our origin servers at that point you could if you didn't ask you've got the back end IP anyway so if we block me if we do kind

of office cajon of the dns if we knew the IP you could still kill the server and that was going on it's part of the reason that we have such a problem when it's allies on everything and it's be able to monitor everything that's going on we couldn't see much can now see loss and getting a better understanding of our traffic so where is it coming from how we getting a lot of traffic from the UK where our main customer bases I was seeing a lot of traffic from Europe China Russia the USA we have no idea really we didn't have any understanding of GL IP so we were already using some technologies in pockets of the business

but kind of all the improve of concept so we had a project that we kicked off that was get off the proof of concept plan and get on a delivering as a production solution so we decided that we were all reading AWS summers and web services to deliver certain solutions to customers so why not do some of that ourselves why not reverse proxy connections to our websites via a proxying system so you don't ever see the origin Webster ok why don't we look at IPS as they come in we're currently Trust all the IPS that come in until they do something bad and then we retrospectively block them on on the other web server or on a firewall didn't

seem elegant and didn't seem something we can do repeatedly request routing so what if we get an attack from a particular country what can we do about it okay come on that screen a request so what do our requests look like what's in the user agent well we kind of could see it if we looked in our is log what's in the URI what's of the body of what all the things are coming in the body we have no idea really we could only look again after the event we needed to handle a request so what if a request contains something we considered bad what can we do about it proactively rather than go all the IP try it's single injectors

right let's block the IP until they come in with a different IP and we needed better monitoring so our technological approach was to use route 53 Amazon's native DNS infrastructure very powerful I'll show you why in a bit elastic load balancing so we need something that's not ours to take care of DDoS and large load huge volumes of traffic not strictly speaking anything to do with what but because we're using Amazon we can benefit from their certificate technology so in the same way if you use let's encrypt you can say open it alter and use forever with us the same you set up our certificate management in Amazon and it auto reuse forever we've got a lot of certificates we

talked about reverse proxying so we use nginx and interestingly our predominant technology stack company is Microsoft and asp.net when you hit is here and nginx proxy and so you're gonna get kind of linux thing but a bit of the aspx page coming back these guys actually doing so we can't we like that because it's it's not clear-cut what technology is some between an attacker and us we're using it IP block lists again let's give the work somebody else so we call out to published IP block lists what lists and if we see your IPS we do stuff with it other technology that's always left in because we've got something new every time and I don't want to be keeping up

with updating this we use our actual Web Application Firewall is is this thing here noxee and it stands for nginx aunty cross-site scripting and sequel injection ok it's not signature based pretty adaptive and soap and sauce another theme we'll get to that towards the end Oh what top ten everything in it and Beyond is in that module and it works with nginx and failed to ban that does things with you when you've been doing things bad and we'll come on to that again and help so anybody know Elka's wrote show hands quite a few people it's pretty cool skewed again because these guys but yet elasticsearch logstash and Cabana so it's a really neat way of

taking logs from whatever generates a log so it could be a web server it could be an event log an era log route 53 DNS etc you stick in a big log pile and then it can be indexed really easily into a uniform pattern and then Cabana allows you to build some really neat analysis tools so that's a stylized diagram of what it looks like but I'm gonna skip this because I'm pressed for time a real rough diagram couldn't be bothered drawing this and you'll probably appreciate why so that's you on the Internet so you make a request to one of our websites and you are you good not misbehaving so you hit route 53 route 53 does a

health check on the site you requested if the site is for instance dead then it will take a look at your request process it and send you down to the static stack here which says sorry guys web site bust we're on it come back another time if you are a UK person or Island alamak cetera come through here we do the health check on the site is up we say where are you coming from and you send you down here and ultimately you'll get served web stuffs from here if you're from outside UK we do the same thing bring you down here and we send you this stack so you mentioned remember me mentioning about

being able to control how we direct traffic to places well this is it we do it at route 53 up at the top here if you are doing bad things and say you come through here and we recognize down here that you're on one of these lists that includes tor exit nose and a bunch of other things we drop your connection on the server so you know get near the web at the web servers here we've got your connection here and it's it's basically a you'll get a CRO out connection error yeah if you are not one of these lists and you make a request that concerns sequel syntax then nginx nak see on here

you'll go a hat sequel syntax I'm going to put you I'm going to block you with an HTTP form hey anyone know what that is it's a teapot okay there is an RFC it's a genuine HTTP status code do recognizes it and we put you on a list in failed to ban that says I found you but you gotta fall right if you repeat that three times you get your final four one eight and you get put on a band west for four hours okay we've made some changes since this last talk because there might be a few people in the room that remember we did it a different way which was to ban you first

time we're asking for life I will took away that that was a bit strong so that's what we do okay I know before I was like a hacker comes back and he gets three more tense for my attempts he's down again okay the honest question yeah well the quick answer is the originally there were too tight to use a using top an attacker which case we're not bothered or a security researcher in which case the security searcher we think and would understand that we were taking this approach but it's it's just a list where we think the eighty twenty year old we think the majority of people coming out or are probably doing something that we're not

interested in so it might be security search in which case we'd hope you'll be sympathetic or the chances are you probably going to be an attacker in which case we don't care quite likely but we would at this time at this moment but it's it's it's a good point okay so that's pretty much how it works okay now what you see here is that there's a perimeter a dotted line perimeter everything inside there is in Amazon Web Services everything and for every single box of forgot to mention the honeypot here yeah what we do is when we process a bad request we send you to a honeypot here so you do get a two or newton if

you're running a script based attack for certain things then you can knock yourself out because we're just going to see traffic but it's gonna sink hole into that server there but we monitor it for every single entity in this diagram it generates a log every single thing including down here on premise and they're all drops into ELQ so again we've got a ton of stuff coming in and we're able to keep an eye on it so wrapping up what we're getting so we can modify what we can ban modify the quest we can handle them far better teapot and honeypot ylim we've got an external bot list that we manage your Ganon might look perfect but 80/20 rule is we kill

the majority and about we drop dodgy IPS we look at user agents and we get rid you can't hit our origins anymore it's a big win for us countries considered a threat they're easy i disallow and afterwards if any wants to talk more about that come and grab me because there is some deep there is a better level in that detail we see all the HTTP traffic we get free cert management and we keep our websites up so spoofing and reap yawns aren't easily handle but we do look at other things we look at your request itself so we can see user agent we can see you actually TP request and we can handle it we still

do a bit of manual work so white listing and things like that bad rules can break raps so we've got a remindful of that we've no machine learning our AI and it kind of comes on and off our agenda currently it's not our agenda we need to keep up because new threats need new analysis it's always a game of catch-up so in conclusion life isn't the sole version I think I made that point clear it's part of a broader deeper approach to security it is as valid as hardening a server it is as valid as good Network design it's as well it is secure coding practices or good secure coding practices it's a first-class member of

the layer security family so thank you very much [Applause] finally and I'm I'll take questions afterwards I'm upset bloke on Twitter you can come and see me I do a lot of stuff on there I'm a member of a thing called the beer farmers that might be worth following me just to find out about that there is a member of beer farmers staff in the room head of security and we're all about security right and shitposting and my blog is except bloke calm okay so again thank you very much your time [Applause]