Damn GraphQL - Defending and Attacking APIs - Dolev Farhi

hi everybody thanks for joining uh this talk is called damn graphql attacking and defending uh apis to get things started uh just a little bit about myself uh i work at well simple these days um this is my first time speaking and also as a participant this is my first time at b-sides vancouver i'm extremely excited to be here and i hope you'll have a great time learning about graphql in this talk so for the agenda today uh we'll start with describing what graphql is about and then we'll kind of jump right into the attack surface we'll talk about how you can attack and how you can defend a graphql backed application and towards the end

we're going to be talking more about uh demos around how to attack it and also uh how you can actually learn and practice graphql if you're interested in learning more about this technology so what is graphql well it's a query language for apis it's json based and you can see on the right side of the of the slide there is an example query uh so this should give you some idea how a graphql query might look like the reason it even exists is that it solves some problems that rest api has and we'll talk more about that in a few seconds uh it uses operations called queries mutations and subscriptions for the purpose of this talk we're only

going to be focusing on queries and mutations queries is basically reading data and mutations is modifying data subscriptions is another uh kind of another feature of graphql you will mostly see this in in applications such as chats that require kind of real time communications with the server it reduces the round trip time to the server by basically combining certain requests into one so if you if we kind of compare graphql to rest api so if in rest api you would say you wanted to get a email of a user on your web server you might be making a request to say api users right and you might get data such as email and a username and so on

so you are basically in this example you're over fetching because you're you actually you're actually only interested in email but you get you got more data than you actually needed so this is a common risk api kind of issue where you get more than you actually asked for and under fetching is when you need some information and you make a query but you didn't get enough information which basically forces you to follow up with another request so you will immediately see that graphql kind of solves this problem by specifying the exact fields that we're interested in and the response will only reflect the fields that we asked for more on that in a few minutes

so uh the the kind of the fundamentals of graphql starts with the schema so schema is a way to describe data and you will notice throughout this talk i will be giving a lot of examples uh from a pastebin data model so pastebin is like this website where you can upload snippets of code or something else for other people to read so i'll be using pastebin as an example throughout this this uh this talk so a schema in our example here you can see that we have a type called paste so if i have a website that where i can upload let's say pastes a paste might have an owner right i uploaded the page so i'll

be the owner it might have some content it might have a title and maybe i want to set some permissions so that it's maybe private to my account or maybe it's public to the white internet so this is kind of how you can describe data in in graphql and graphql always has a schema so the type paste is basically our object and owner title content all the things that you see underneath it these are the fields so i mentioned queries before so queries are used for reading data in graphql um it doesn't mean that you cannot kind of play around with how graphql works so that you can actually change things with queries but it's really

meant to read and not to alter information on the on the application and you can see an example here how we're actually asking graphql to give us all the pastes in the website but only their title and content and you can see that the response looks almost exactly like the request i got an array of all the pastes and in this example only have a single paste and you can see that i got the title and content and i didn't get anything beyond that i didn't get the owner i didn't get whether it was a public pace or a private paste and so on so the exact same information that i asked for and if you actually wanted to create or

delete or change something using graphql you would be using something called mutations so in the example request that you see here we're actually creating a paste as opposed to fetching a paste you will notice that we're passing parameters so i can create a paste with the specific title and content and you will also notice that underneath create paste i'm also asking for the title in the exact same request that i'm creating the paste so if you compare this request and kind of try to imagine how it would look like in the rest api world you will be making two requests with rest right you'll be making maybe a post request to create the paste and a follow-up get request to maybe get

the title of that paste so in graphql you're making a single request and you can actually create and read at the same time so just a few graphql things that i think you will find quite interesting graphql requests are usually post which means that whether you're reading data or you're creating data or deleting data it's always carried over post for the most part um it's a single end point so as opposed to rest api you know in rest api you might have you know slash users slash id or something like that in graphql you only have a single endpoint from a blue team perspective try to think about situations where you are used to reading

log files and try to kind of derive some uh some information based on the path that was requested in graphql this is different so you'll have to kind of get used to doing analysis in slightly different way when it comes to graphql obviously this is a customizable thing so it might not necessarily live under graphql there are other endpoints that you could use and different implementations place the the graphql endpoint in different locations so it depends on on the implementation if you wanted to kind of play with a script that's shot that attempts to kind of identify graphql and where it lives you can use the nmap nsc at the bottom of this slide if you're an nmap fan

it's a script that simply tries to figure out where graphql lives um and last thing about graphql that's quite interesting is that response codes are usually 200 whether you whether it was successful or not what i mean by that is if you're making a query and say you're asking for a specific field like you want a username of some sort of some account and that account doesn't exist you will get a 200 but the error that the account doesn't exist will actually be reflected in the response body so again going back to the analytics part the blue team side of things if you are used to looking or kind of skimming through log files and

looking for indicators around status codes that will maybe reveal some problem or something like that graphql will usually return 200. so if so if you're used to the rest api world you have to kind of keep in mind that graphql works a little bit differently the errors are in the response so let's talk about the attack surface now that we have some basic understanding about graphql when you install graphql in some of the implementations you might see something that looks like this this is like an ide interface for graphql where you can start typing queries and it will auto complete them it will give you the documentation basically that describes the schema behind graphql this is not necessarily something that's

turned on by default in some implementations it is actually installed by default and turned on by default and one thing to keep in mind from an attacker's standpoint is that the graphql or i should say graph iql it's usually under that endpoint as well graph iql it may have different security controls than the normal graphql endpoint um so just keep in mind that uh some implementations or you know some apps may have different routes and hence different controls uh so you want to test that carefully and again like the normal graphql endpoint it can leave it under different paths so you'll have to do some kind of a information gathering just like any other pen test from a blue team standpoint the

most trivial thing is to disable the explorer uh or simply just not install that package um or if you absolutely need to have that you can protect that using some kind of an iep access list so you only allow trusted clients to actually interact with that that dashboard next thing that we have is probably one of the popular things about graphql from an attacker's standpoint when you run into a graphql endpoint one of the first things you want to kind of try is to run an introspection query against graphql introspection query is basically really telling graphql to tell you all it knows about its data so you send a specific query that there's a long query that you can find

on on github it's on uh the link is in the link at the bottom of the slide and if you're lucky the the introspection query will will actually be enabled and you can actually just basically dump the schema of of graphql but sometimes and i think that uh that in most cases in normal kind of stable solid environments introspection should not be enabled unless there is a really good reason behind it so keep in mind that it's not necessarily always going to be available but if you were lucky enough and you managed to get that data you can actually take the response that you got and feed it into something called a visualizer there is one called voyager for graphql

and it will give you it will actually take the json response and kind of convert it into something that's a little bit more user friendly like the kind of the diagram that you see or the data flow that you see on the right side of the slide so for a blue team standpoint you want to first check your own graphql implementation try to run it in your inspection query and see if something comes back to you um if you absolutely need to have that enabled try to try to kind of put this behind some sort of control um there's multiple ways you can go about that but at the very least you can try to put that behind some kind of

authentication or authorization let's say specific sessions are only allowed to do introspection queries and so on or you can simply disable introspection in production in general uh most implementations from what i've seen it is the popular ones no matter what the language is whether it's node.js ruby java and so on they allow you to disable introspection uh if you if you want that uh keep in mind that introspection is a kind of a tool that aids uh developers to interact with the api so there's a reason why it exists it's not a vulnerability it's a feature but obviously a feature that has some kind of a security trade-off um so if you're in if you're kind of a

pen tester or an attacker and the the introspection is actually disabled tough luck but there are other ways to kind of go about this one thing that you should know about graphql is that it's very kind it will actually hint on the possible fields that it supports if you send a request that has some kind of a typo so if you look on the left side there is a an example request where i made a typo in the word paste and i i kind of uh forgot the e so graphql will actually go ahead and tell you or ask you whether you meant paste or pastes so you can see where i'm taking this right you can actually build

a query based on some kind of a common word list and try to extract those suggestions until you can build enough content to actually build a proper query from what i've seen all the absolutely all the possible implementations that are popular today have field suggestions enabled by default and there is no option to disable it unless you patch it yourself so this is definitely a fuzzing opportunity and you can mostly count on it to be available to you so from a blue team standpoint like i said there's no easy way to disable this you have to kind of comment out that that section of the code or kind of patch it in a way that makes

sense in your context so that it it doesn't exist anymore there's a ux impact or user experience impact here obviously because if somebody or if you have a third party interacting with your api then um you know those suggestions may be very helpful if they're making a typo and they're not sure where they went wrong so on the denial of service side of things so from an attacker's standpoint there are multiple ways in graphql how you can kind of overwhelm the service graphql actually allows you to take a series of queries and send them in one shot so you can send an array of multiple queries all at once in a single request to graphql and it will

process them one after another um this is very effective and there is also another kind of interesting aspect here since i meant since i mentioned that we only send it to one endpoint and it's only a single request you can actually evade any rate limiting that exists because it for the most part you know web application firewalls and other controls will not necessarily block you after one request but if you're making if you're able to make one very complex request in a single shot then you could actually overwhelm the service pretty easily and i'll give more examples on that so from a blue team standpoint if you wanted to kind of protect yourself against that

you could check the array that that you received and if there are too many elements in that array you can simply drop that request there's another aspect called cost-based analysis which essentially you assign a value to a query and you have some kind of a threshold on the back-end service and if that thresh if the query cost exceeds or exceeds exceeds this threshold then you simply drop that request so in this example we have a max cost of 10 but the backup system query was assigned a value of 10 so if i send 2 that makes it 20 and that's more than what i allow hence i reject it and batching can actually be disabled so

only use it where it makes sense and disable if possible from an attacker standpoint if you want to overwhelm the service but batching is disabled you have another way to go about that so you can actually send the same query multiple times in graphql using aliases this is pretty interesting because it's still a single request and you can still evade rate limiting controls and still overwhelm the service with one request maybe temporarily right if you send it for multiple ips you can really overwhelm the service and still not get blocked so in this example you can see that we're making four ases that they all call backup system and backup system in this context let's say that it's a very

resource expensive query to run on the server side so this is another interesting area from a blue team standpoint cost based analysis is going to help here as well right because if you have the same query multiple times the the cost is going to increase and then you're going to drop that request because because you have some limit but there's another thing you could do which is kind of like a middleware analyzer that you can kind of count the number of aliases that you see and drop it if that number exceeds what what you uh what you support on the denial of service side of things again uh circular queries is a very popular graphql attack think about the paceman

example right we have pastes and those pastes have owners but the other way around is true as well owners have pace so you can build a query that kind of references one to another until you create a very complex and and kind of a deeply nested query that could overwhelm a service um so you want to check the the schema whether you have any kind of relationships that can actually cause that if you take this kind of query and you chain it with what we learned earlier about batch queries and aliases you can really make a significant impact on the blue side of things you can set a maximum depth limit most implementations support that

which means that you set let's say max depth of 10 but you get a crate that is so nested that it goes like 99 levels deep you simply drop that request some implementations also allow you to set those limits at the schema level or even at a single single request level operation name fuzzing is interesting operation name fuzzing is is basically operation name is this extra information that you can send along with the request that will describe what the request is about this is a kind of not a mandatory field you can it's optional um but it's user controlled right so you can actually try to play around with it so for example in the query here i'm making a query

called get users but i i supplied git paste as an operation name and if the uh you know the site operator uh is was naive enough to think that operation name indicates the actual query that was used then you can try to spoof that right because you're kind of masking the actual intent here and operation names can only also be used to do some kind of decision making and some implementations actually also allow you to specify some special characters so you can see where i'm taking this you can try to play around and see maybe you can get some kind of some kind of an injection on the back end um just keep in mind

that this is also a possibility on the blue side this is a fairly easy mitigation you want to compare that with the list of operation names that you support in your app if it's not on the list just simply drop it uh any kind of any any kind of tampering with operation name and just tampering in general graphql you really want to log that in this safe way there's no reason why trusted client would suddenly change the operation name unless it was tampered with field duplication this is probably one of my favorites and you see that we have a common theme of denial of service attacks uh some implementations in graphql simply do not deduce the requested

fields right so if you make a request to say to get the name of an owner of the owner of all the pace in my app you can simply take that name and duplicate that multiple times and if you you will have to do some kind of a timing analysis here um so the timing analysis if you if you're making a request and let's say of just a single field called name and it takes 100 milliseconds for the service to for the server to respond and suddenly you duplicate that and suddenly increased you know that you have an issue this is a very uh i would say not as known of an attack today um but i have

seen that multiple times uh in the last little while so it's very effective and again you can take this kind of query and also kind of combine that with batch requests or aliases and then you really kind of amplify the the attack using this method um on the blue side here if you want to defend against that there are multiple ways you can do you can actually achieve that you can try to do the fields it's a little bit error prone but it's actually possible again you'll have to do some kind of a build some kind of an analyzer that will help you do that you can use cost-based analysis exactly the same way as we saw before

and you could use something called persistent queries and persistent queries is interesting because you instead of sending the query as is like a json that describes something you send a hash that represents that priority so in in kind of a real world uh web application scenario the client will have a list of or let's say the client library that interacts with graphql will have a list of accepted hashes and the client will actually use those hashes to query for the data and on the server side the server will have a list of supported hashes and if the hash that it received from the client doesn't exist in the database it will simply get dropped so this is

very effective also against some of the previous attacks that i showed but also keep in mind that you can still pass variables with that hash so let's say that the hash is some representation of a query and that decree also accepts some parameters that are uh that the input is based on some variables that you pass with it then the variables are not going to be a hash they're going to be something that the clients the client maybe dynamically provides and these are still injection points so it's a it's an interesting mitigation but also has uh other uh kind of attack uh also has smaller like attack surfaces within it that you should explore so in summary um you really want to log

signs of query tampering this is not a graphql thing by any means you want to do that all no matter what api technology you use but graphql has a lot of interesting areas that you should definitely try to log if you see some anomalies in them um you want to try to find if on the cicd pipeline uh side of things you can try to use os app to try and test your graphql endpoint they have a fairly recent add-on that they they uh came up with i'm not sure about its maturity it does work i've tried it but i'm not sure uh you know how how much effort was put into it and whether

it's still maintained but it can do the work you can also consider setting timeouts there is no um there is no magic number there it really depends on the context of your app and you can utilize some of the protections that we also talked about so in short graphql is really young especially from a security standpoint um from both from blue and red team perspective there are tools but not a whole lot of them and what i also noticed from looking at the different implementations is that there is no alignment on what is a must-have security feature uh and what what's what not so for example uh um php graphql might have persistent queries where python doesn't

so there's some kind of a misalignment there but this is an opportunity to innovate so if you if you really want to contribute to the field whether it's building attack tools or building defenses this is really a good opportunity to do it right now so with that i want to kind of jump into a attack demo so this is a real world kind of cve focused on wordpress graphql so if you don't know what wordpress graphql is it's a plugin for wordpress that essentially gives you a graphql api endpoint for your wordpress instance so think about a headless version of wordpress where you can interact with it get the post post post uh new comments and so on and

so forth um it's pretty popular uh 10 000 active installations and 100 000 downloads so pretty interesting plug-in so let's talk about the vulnerability itself wordpress does not have uh the ability to turn off the batching uh query that it comes uh with by default um which is a concern as you can as you can tell from the previous slides um duplicated fields are not getting deduped and there is no notion of cost analysis by any means so uh the the plugin is not intelligent enough to drop requests that are extremely expensive and it's a partially authenticated api as well and that makes sense because in wordpress you know you have guests visiting your blog and it makes sense

for them to be able to read the posts and the comments within those posts right so it makes sense why it's a partially authenticated api um about the exploit um so we're going to read this diagram kind of bottom up we're going to get all the comments and posts of whatever post whatever blog set we we want and we're actually going to ask for the posts and comments 10 000 times in a single request and then we're going to get take that request and we're going to batch it 100 times right so we're going to have 100 requests that are extremely expensive and then we're going to run a 300 thread requests all at once at a server and you

can see where i'm taking this this is this becomes a very uh expensive operation for the server to actually process so let's see how to go about that so when you install the graphql plugin you're going to notice that there is a shortcut in the admin panel where you can kind of write a queries and see what the response is like from wordpress so we can start writing a query that will simply get a let's say all the content of all the posts uh on my sorry not nodes we need edges and nodes and then we're gonna get the content uh we're gonna get the content of uh all the posts uh in my blog by just

running a simple query um and then what we're gonna do is we're gonna take that and we're gonna amplify that by duplicating the fields and then using the batch queries and we're going to send it all together just like explained earlier so if we're lucky enough we'll be able to take this server down within probably probably 10 seconds so let's see how we go about that so i'm going to run a script pass it a few parameters and it will do its thing if we're lucky we'll see some interesting things going on in a few seconds

simultaneously i'm also tailing the log of the server to see if any errors come up while it's the service is being attacked you can already see that this the my sequel was killed because of out of memory issues um so at this point um mysql is trying to restart and it's not even able to do that it's not able to recover um so if we're lucky we're also going to see like the mysql error as you can see wordpress is not even able to communicate with the database at this point um this is extremely effective um and it's just a single host running a bunch of processes so think about what multiple or hundreds of hosts can actually do to a

you know maybe a more powerful server as well so this was successful um and this should show you like how easy it is to take a service down if it's not well protected with graphql with graphql protections so um with that i want to go back to our slide and talk about how you can actually learn more about graphql so when i started with graphql like a few months ago um i was looking for a damn vulnerable application like you often see uh in web apps um and that doesn't didn't exist at least not officially there were a few projects here and there so i decided to make one and kind of open source that so

this has been live for the past few months now um it's focused on graphql uh and and it covers most if not everything that that i talked about in this talk um it has two modes uh one for beginner one for expert the emphasis of this platform is really on education there's a lot of content a lot of resources tools and and blog posts and um and other things that i kind of threw in there so that you can learn more about the attacks and how to defend it very similar to how the kind of the talk was structured so go ahead pick it up play with it tell me what you think when you first instantiate it it looks

like this and with that that's all i have i hope you enjoyed it and learned a little bit more about graphql um and i hope that you're going to have a wonderful time in besides vancouver thank you so much it was great thank you you