Yet Another Type of Application Layer Denial of Service Attack

hi everybody my name is Sergei shaken I'm web applications Kennard developer Wallace and as as a side project I was played with application layer denial of service attacks and developed developed but - which helps you to test your websites and I'm gonna speak about distributed denial of service attacks in general and application layer denial of service attacks in particular I'll be staring at my iPhone because I don't know how to set up presenters notes on on macbook so I'll be reading my notes from my iPhone if I forget something so so everyday we see more and more headlines about distributed denial of service attacks targeting corporations persons governments the opposition and it's it becomes part of our life life and so

just wanted to remind you what denial of service attack is those are attacks that aimed at blocking access by outside users to an internet service or a website and they usually do this by either overwhelming 101 or more of the resources of the server or just flooding network infrastructure or or devices more headlines and types of attack another service attacks our blue tag or plumpest of tools in the web application exploit arsenal and simplest simpliest attacks employing nothing more than a flood of packets that overwhelm the targets capability to handle such an amount of traffic and if web application can talk they might even thank the attackers for unloading them I'm kidding and just thing is that under such attacks such

dump attacks web application itself might never even see the full effect of the assault because the network stacks or operating systems would fall over before legitimate traffic reach to the application and on the other hand the class of denial of service attacks that target resource consumption like CPU cycles or memory or file descriptors or whatever it is can and should be addressed by the web application itself and/or database or like framework behind that web application or web server platform and so these types of attacks don't have universal applicability that most network layer attacks do and we're gonna speak about them so how I imagine distributed denial-of-service attacks the flute flood type the network layer I imagine

them as a mob and we're like thousands of packets or requests or what else or something else brings down one single server and application layer attacks are like more like this guy it's like sharp straight shooter it can get the target win with one single no can't or Panch and basically that's the case where one brings down one website so it want one kind of relationship that seamlessly some numbers about distributed denial-of-service attacks actually this morning I got a reply from Kaspersky Lab they released a report on denial of service attacks for last what last bathwater and I was wondering why they didn't mention any slow attacks application-layer attacks and they said that there are bad pot nets that

implement implement slow attacks but flood the tags dominate so far and they are decided not to cover slow attacks for the last quarter also I actually chatted with botnet operator over ICQ last week was trying to act as a customer of their services and honestly I was trying to get from them what types of attack are they gonna use and I actually I had a web server hosted on GoDaddy I asked him how much would it cost me to knock out a web server house in the United States hosted by GoDaddy and for one hour he said 20 bucks just pay and enjoy the results I was trying to get the information like what kind of

attack they are going to use the Europe flight we are going to use syn flood and like coded it most likely with filter out syn floods T's then we apply something more complex that's why it 20 bucks not the 5 bucks our regular price because we don't know what we're gonna test so conversation ended up by he gave me the links to the forums internet forums where I can get customer click credentials and feedback it's like REO business and I actually wanted to pay to that guy but they accept only web money and I didn't know how to how to buy web money with PayPal and they don't accept paypal so I'll probably continue chatting with them

later here's the actual screenshot from their forum it's in Russian I translated some main points that they really they think they are running a regular business with that marketing slogans and everything like they e they will take projects with anti those they offer wholesale discounts they have customer credentials and feedback and here's another screenshot from another forum advertising the same botnet they offer pretty simple arsenal everything is flood oh yeah and I'll tell you how this works in at least in Russian market closer to the summer for example travel agencies hire someone they called him SEO specialist search engine optimization but actually the task of that guy to move the company's website he works for to the top level to the top

of the search search results of they for for some reason they concentrate on Russian search engine the Yandex but they all know when the where Yandex updates their database every day at some particular time they call it up time update time and if you keep competitors website down for 24 hours and like between those apps then it guaranteed that competitors website would be at least several ballot booths lower than it's usually there so that's how they get more customers who are just typing I want to go to Egypt and click the first thing and actually the Kaspersky's report states that denial of service attacks against travel agencies in Russia grow five times in high season so

it's a three-wheeler business have some demo time and then I'll switch back to my presentation explaining what's just happened so I wrote this simple tool which actually consolidates all known slow-slow types of attacks like slow loris slow posts slower eat and some others like a particular but we are not going to speak about it today what I have on my virtual machine so it's a

first I'm gonna attack an Apache default configuration of Apache server which I download it on a boon to using apt-get and I didn't touch the configuration files file at all and except adding some dummy content and here is the yeah and I enabled mod status just to see the status to works on any Linux system any Mac hos Mac OS it even works with sanguine on site win so you can run it on Windows I actually tested it for the first time yesterday but it works the command line so here we have the - X means we'll be emulating slow read attack I will talk about it later the URL the connection count which is

thousand in this case and it's more than enough k3 is my pipeline factor if web server supports HTTP pipelining this three this number means we're gonna request the same content three times using the same connection just to multiply the size of the response by three and and that's it and output goes to this plain Apache besides demo file I'm running it status updates every five seconds you see it oops we also better specify the connection rate because before these 50 connections per second which is like 200

let's see what happens with the party as this number of threads connections within the Wright State [Music]

anybody remembers the IP [Music] okay I'll show you it's in here is the same Sun oops now it's not the same I was the same server and it's

oh no it's here we see Oh

we can't load the page statistic here shows that we have 700 connections pending and connected to 93 - connections closed and service is not available I think it's enough we can open the output file which I to generate now we'll see here that on 6th second service was already not available if connection rate would if we set connection rate higher this could happen even sooner than 6 seconds we have constant number of pending connections and we have constant number of connected stuff I also wanted to show a test against varnish cache server which some people might think is securing your website because well doesn't matter why just we'll show you what's happening against varnish webserver so it's the

same Apache web server but it is behind cache a varnish cache server we're running the attack

there's actually only one connection established by varnish because it's the same content should be delivered to thousand different connections so Apache L furnish cares are very smart and its make it makes only one single connection to the actual web server but it is already down long time ago

it is on port their default port and it's again default varnish server but I actually didn't find any settings to play with to make it not vulnerable I didn't spend too much time but yeah yeah it it doesn't even accept new connections that doesn't even put those connections in the queue

[Music]

so again almost the same statistics on 52nd server went down with and actually this chart means that you need only 239 240 connections to knockout the server behind varnish and not even one single server but there could be a form of the servers and you would think that you're speeding things up by putting a cache server in front of it but it's actually acting as a bottleneck in this case oh let's go back to the presentation

okay so I'll be speaking about these three slow type attacks though there are a lot of other interesting application layer denial of service attacks like like sequel injection which can run benchmark was a inch mark for example on sicko see on my sequel and caused a denial of service by one single request or like keep alive attack or THC SSL dos or hash collision or something created to rage exist if server side rejects engine is poor poorly developed mainly I'll be talking about these three types of attacks because my tool supports them aim of the attacks so as opposed to as opposed to flood attacks or those network layer attacks these slow attacks are low bandwidth and theoretically or

not even theoretically I ran my tool on on my jailbroken iPhone though is a it's a long story that's why I'm not releasing anything for iPhone but it it is possible and I was able to run it successfully over edge connection which is much more slower than 3G and it's edge because a I'm on t-mobile and iPhone doesn't support iPhone on t-mobile doesn't work in 3G speeds so so yeah that's that's it and yeah the main thing is that all three slow attacks are aiming the one single thing to fill up the concurrent connections pool which is usually relatively slow a small before we go ahead I wanted to remind you how HTTP request looks like but this RFC

notation is not the tree readable human readable so here is an actual example HTP request starts with request line and then some mandatory and not mandatory headers HTTP headers and as you notice they are delimited with carriage return line feed characters symbols and server would know that it and it's supposed to finish reading the request when it sees to crlf following each other so this blank line or crlf character means end of the receipt from the client and server can start generating the response and sending it back ok slowloris the first slow attack what it does it tries to keep as many connections open as possible by and hold them open by sending incomplete header so it never

sends that final crlf and if you open thousand connections to the web server which supposed to handle only thousand concurrent requests server will start rejecting service to legitimate users here is some stupid animation so we're sending this first part of the request with some legitimate resource in this case it's like main page or /and with host header and some random header with with some hinder random ASCII characters everything is delimited with crlf that /r solution is actually our Allah and then if for example server configure to drop the connection if there is no data coming within 60 seconds we know we experiment experimental we figured out that the timeout is 60 seconds or it's your own

server and you know what the number is so you sent 59 seconds later some another stupid meaningless set of ASCII characters the limiter speed crlf and then again and again and you can virtually prolong the connection forever unless you meet some limit for example server has a limitation on number of actual HTTP headers or on the size of the HTTP request itself or some some other limitation

yeah and unfortunately there is no reliable configuration like or advise to universal advice to protect your web servers but there are some recommendations that could minimize your vulnerable surface and I'll talk about it a little bit later another attack slow post principle is similar to slowloris it again keeps many connections to the target web server and opens to the server open and holds them open for as long as possible but instead of prolonging the headers section of the HTTP request it prolongs the message body section so you send your post request looks very legitimate so no filters or IPS could detect unless they know what to look for and then pay attention to the content length header

where we specify 4096 in this case that means server would be reading after receiving the final that blank line final CR worker characters it would read it would try to read 4096 bytes before processing that this request so we send that final crlf everything looks legitimate 10:59 seconds later we send some name value pair which mimics some forms upbeat results and then some random named a loop error again with ASCII characters and then again and again and you prolong the connection the same way as for slowloris just for example IAS has protection against low header it is is Microsoft's Internet Information server they introduced the protection in the is I think seven I guess low headers they have a timeout

if server receives the HTTP request headers for some period of time and it never finished it just drops the connection but that protection doesn't help in this case in slow read now principle is the same tactics but the attacks takes different approach so in case of slow read and slow Doris we were slowing down the request and in case of slow reads it sends the full request so it doesn't it's not different from any legitimate HTTP request that would come from your customer but it slows down in read English in response processing section so your client reads the response from the server very slowly so server keeps pulling that connection to send the rest of the information but it

can it can't and the result is the same you are filling up the concurrent connections pool and yeah how is it happening it's happening using known TCP protocol behavior when if client says that my buffer is this small like 10 bytes for example the recipient the server in our case by TCP protocol supposed to wait and send probe probe message is asking are you ready yet are you ready yet and if client replies with Windows size zero which is on the next page TCP doesn't have any mechanisms to handle this because I honestly application supposed to handle situations like this it's application things that the recipient is not ready to receive my information for one minute

then drop the connection but most of the applications applications are not handling this case so some details yeah the key point is of this attack is to find the resource on the server which is larger than servers and buffer kernel space and buffer which on most Linux systems is around 65 K so if you find something larger than 65 K then you most likely would successfully attack the server or even if you can't find anything larger than that but server supports the HTTP pipelining you can multiplex the response by tree requesting the same resource several times using the same connection so people fill up the server send buffer anyway

then

there is again some very basic animation so on your right side is the small receive buffer of the client on your left side on your right side is the server with relatively large send buffer but you are going to fill it up let's send buffer connection so your malicious client requests a some big page relatively big page can on h2 on application there HTTP request is very legitimate it can be randomized if you care about not being catched by IPS or ideas but the only difference is that on network layer your connection when it in connection stage you let the server know that your receive buffer is relatively small for example 1459 bytes server accepted the request pick the resource

from the disk or memory or whatever it is made it ready sent it to the buffer so buff so kernel delivers it over the wire but it doesn't fit into send buffer so your server still has to monitor that connection to see when its kernel will let him know let it know that you can send the rest of the data yeah generates the HTTP response tries to send first chunk of data which is like 1459 bytes but this guy client would accept that 1459 bytes and say that i don't have any more space to accept any data so your server stuck there it has to monitor that connection because it has to deliver the entire

content but the client would keep replying that no I don't have an more space in my buffer and just to TCP has a mechanism for probing I don't know 59 seconds later the client can't say that I have 10 more bytes to accept so sorry would send another time way more biased and wait another minute by the link and then another 10 more bytes so basically you are achieving the same thing you achieve with slowloris or slow post but in this case attack is even more invisible because on application where it is legitimate connection and very low very few number of protection mechanisms would pay attention to the tcp that initial receive window size if

it's very if it's small or not yeah to emphasize the difference between slow readings and other slow types of attack I just wanted to extrapolate that to some real-life example so imagine a slow line at a fast-food restaurant and like every person in that client starts thinking cloud one what once he reached to the cashier he starts thinking that yeah I want to get that burger or no I'll get just coffee or basically he slows down the line and slows down the service and then for example one minute later he made his order he paid the he took his order in the next customer approach so basically that's how slowloris or slow post work and key thing there is it is

possible to identify and isolate that slow guy in his request phase why while he's asking for something and in case of slow reads the picture is a little bit different is this the same line it's the same customers but once they reach to the cashier they act very fast and they know what they want but in this case the guy makes order of 50 pizzas and then he even pays he everything is fine but he cannot take all the 50 pizzas to the car so he has to go back and forth with two pizzas in his hand and he slows down the the flow but he is slowing down in the response received reception phase so if

you were monitoring only your requests if you think that your pay loads would appear in the request only then you're gonna miss this attack and just comparison

so summarizing yeah most defense mechanisms are expecting the malicious payload in the request phase and but as to me the entire transaction should be monitored to catch and most of the mitigations are based on monitoring Kong will request like for example in Apache range header you monitor in a packet arrange header attack or like sequel injection yeah monitoring the application where the data for malicious sequel statements in your request is enough but you are not going to catch anything like slow reader are you vulnerable there is a good chance that you are because I tested default configurations of engine excit PDI is apache under mac and under i tested also apache under freebsd and on I tested all

these stuff on Ubuntu and all of these servers are vulnerable at least on Boonta for example I accept I yes of course varnish cache proxies vulnerable shoutcast it's a streaming server which is used very widely now but it is vulnerable though it's not like pure HTTP application but data goes back and forth over HTTP and it is vulnerable why not yeah about varnish it is quite interesting I when I released the slow read attack I got two emails from two different guys from two different continents one from Brazil another from Netherlands asking me if I can take a look at the test result they ran slow a snippet has the - my - and they got some

results and they were thinking that there is a false positive in the results because they were sure that varnish protects them because they have a farm behind that varnish like with seven or eight different virtual Apaches and databases and stuff like that but but tool shows that they are vulnerable and I looked I basically they were vulnerable really and I don't know how easily fix their problem besides like switching to squid or something because varnish doesn't have any configurations abbreviated to concurrent connections it has like number of it changes number of open file descriptors limited by operating system one before it starts but basically I don't know how to fix it and bad thing there is they were they

were sure they are vulnerable they are protected and this is a false positive yeah what should you do if you have or if you know ration go to some Russian website and ask them to attack you for 20 bucks or you can download my two or there are other plenty 20 of other tools just - consolidating all the attacks in one binary and I think it's more easy to to use my tool and I'm also working on Whitehead botnet so check out the slow hammer that me in two months we have working demo it's it's a botnet hosting and BOTS are hosted in the cloud in different geographical zones so it could mimic real botnets pretty well detection

and mitigation drop connections with abnormally small TCP advertised window I honestly I don't know how to do it easily because I'm not a big expert in IP tables but I think IP tables should allow you to do that have an absolute connection time out no matter what because it's really it doesn't really make sense to to rely on client on the client when it will close the connection legitimately because and there is a very interesting thing Apache has a timeout directive in configuration file but and by default it's I don't remember I think it's for example 60 seconds but that six that Apache would never drop the connection that 60 seconds mean means Apache would drop the

connection if there is no data arriving for 60 seconds if you keep sending something bite by bite every 59 seconds that timeout would never trigger limit whatever is possible if your web application is not supposed to generate like 50 or 50 headers with within with one single HTTP request or if you know that browser is not supposed to generate anything over 50 then why not limiting it to 50 for example or like if you have a login form which submits username and password and it's not supposed to be larger than 200 bytes there is no reason to have the Apaches default which is two gigabytes by the way so somebody can drop data byte by byte to your URL which

accepts that login credentials for foreign forever what is it yeah verbs the same Apache or nginx alright it implied TPD would wait for complete HTTP requests before analyzing what the verb the get or post or head or whatever connect or whatever is there before they analyze it so you can have something like fake fake slow verb specified in your HTTP request and Apache would be as vulnerable as it would be with post or we'd get and because it would never analyze what's written there until a connection finished the HTTP request finished that's dangerous because if you are filtering your log by get requests or by post requests polish's requests could be unnoticed and you just yeah what you have to do

you it's better to reject anything you don't know about but in case of a patch it won't help because a patch is supposed to read the entire HTTP request before accepting or rejecting it if you never finish your request Apache would never reject it yeah define minimum data rate IAS has built-in attribute for that mode security has on the next slide what else can you use quality web application scanner has passive detection like passive means it not it's not actively attacking your website it just based on the connection behavior of the server it can tell if you are vulnerable or not mode security has a lot of useful thing both around low lorries and slow post

protection and around slow read in 2/6 they introduced that sacred state limit attribute and SEC read state limit attribute which actually can we meet your connection lifetime per IP address which means basically if you said okay those two attributes to one minute mode security what would drop a connection if it is in read state or if it is in write state for longer than one minute which makes sense because normally unless your customer is uploading a huge image or download or now in case of downloading it won't work if he's uploading something just have a dedicated server for uploads for example snort once I release that slow reads related stuffs North responded that they are working on having ability to specify

a range of TCP advertised window except like range they want you want to decline or accept also Akamai released one week ago they didn't release they actually announced it announced it they gonna release it in April but that Kona side offenders supposed to handle both flood types of attacks what type of attacks and application layer including slow restore Lauri's sequel injections that are causing denial of service and things like that and there is another useful script developed by this guy christian Fellini he's expert in denial of service attacks and script is very lame but it works it's actually we can specify specify for example a geographical zone like classify rule saying if there is more than 50

connections from Estonia and they are all 50 connections are either in reeds they are yeah it doesn't matter they are open for longer than five minutes blacklist Estonia and then or send me an email or do whatever you want you can specify it in the script bridge pretty beautiful think it just script based so it's not that fast but it it works yeah who reacted first when I release that slow rate

say it again yeah this is all my recommendations were based on one single machine attacking you if it's a botnet then combine these recommendations with anti distributed denial-of-service attacks yeah I don't think you can do anything if somebody would attack will attack you with botnet but using these load types of attacks hopefully outcome I would do something about it because they are charging you money but I don't know it will be released in April and I have no idea how they do that but based on their previous products they supposed to do well yeah who reacted first hey gamers reacted first six days after I released that's slow read related post open transport tycoon deluxe found that using slow

reads it is possible to prevent anyone from joining the server and they release the several patches for different versions of open transport tycoon and they even reported CDE entry in the vulnerability database so they took it serious and it'd be nice to see other developers reacting that fast to denial of service attacks vulnerabilities yeah and my summary is that hopefully tool I developed and other guys developed would help researchers to concentrate on fixing things rather than developing proof-of-concept tools and hopefully people will start paying attention or attackers would act faster than we react so hopefully researchers will concentrate on fixing things and that's it that's my final slide any questions No

yeah I I mentioned yeah I'll put 24 first came up with their sock sock sock stress proof of concept which was never released but yeah they were first talking about it then angular 2 developed by BT or some some some guy who developed angular 2 - implemented that slow read attack but he was they they both were using raw sockets to craft TCP packets by hand in there like I'm talking about technical details my approach is easier because I'm not using any row so it gets any low-level programming you can manipulate the your clients receive buffer size by just said so coped by ISO receive buffer attribute so but correct there is a direct relation between set sock sorry ass or

receive buffer and your actual window size that would be declared by your tcp/ip stack and on also my to does implements the attack on or it takes implements it over HTTP but like any server application could be vulnerable to that type of attack and could be dangerous other questions

[Music] ingenious has a lot of even built-in attributes that could be helpful because you can limit almost anything with nginx just thing is that nginx default configuration in nginx it is either 2,000 connections per concurrent connections but on the other hand on at least on Ubuntu they were not changing the open file descriptors which is by default is some low number and by default you are vulnerable and 90% of people wouldn't even care looking at concurrent connections number or at open these three open file descriptors limit nginx is pretty handling pretty well these avi slowloris on slow post and for Apache there is a mod mod ragtime out request timeout which is I think part of

it is I think part of mod security using mod security should be totally enough to to protect your Apache also event NPM of the Apache there is a it used to be experimental NPM what's NPM stands for don't remember but that event NPM handles the architecture of that is very well designed and event NPM handles such attacks very well but thing is it doesn't support SSL connections so it's not a complete solution so that's it thank you very much for voting [Applause]