The Power of Guardrails: How to Slash Your Risk of XSS in Half

this next talk is being presented by breaking grounds it's the power of guardrails by colleen dye and grayson hardaway hello today we'll be talking about how using secured guardrails that encourage developers to write secure code can actually decrease your risk of cross-site scripting pretty significantly so i'm colleen and today with grayson i'll be talking about how to slash the risk of crosstalk scripting in half so i sort of want to start off with the question what type of vulnerabilities are you seeing most often in your code so oftentimes these are the same vulnerabilities you see pop up popping up over and over again and this can be a bit demoralizing if you keep finding the same types of

vulnerabilities um furthermore with a lot of companies having a ratio of 100 or more developers to every security engineer it's sometimes easy to feel pretty buried and like you can't actually keep up but we want to talk about how some companies have actually been changing how they run their security programs and it actually seems to be working for them so they're higher leveraged and more effective at actually keeping their company secure even while keeping the same ratio of developers to secure the engineers so today we're going to be talking about secured guardrails which are essentially safeguards put into place to make it easier for developers to do the secure thing and a lot harder to do the insecure

thing so we'll talk about how guardrails can actually drastically reduce the number of vulnerabilities you have to track down and fix um and essentially talk about some original research we've done that examines real crosstalk scripting vulnerabilities in open source software so here's a brief tldr of what we'll be talking about today so we used to do we used to think about eliminating individual bugs through pen testing bug bounties etc but now we'll be talking about secured guardrails which actually give you some more scalable systematic and long-term wins so you might ask how do we know this is effective well we'll be talking about what other companies do along with some research showing that secure guardrails could have prevented greater

than 50 of crosstalk scripting vulnerabilities in open source repositories we'll also be talking briefly at the end how to implement this at your corporation cool so who are we i'll pass this offer to grayson um who's going to be talking with me today thanks colleen yeah my name is grayson hardaway i'm a security engineer at r2c this is a picture of me and my lovely wife over here in a past life i used to work for the us department of defense doing all manner of things but now i work for rgc which is a startup in san francisco doing static analysis along with colleen awesome i'm also a security engineer at rtc and i recently graduated stanford with

my bachelor's of cs and my master's of statistics and what's rtc you might ask so we're a sfa static analysis startup that grayson grayson already mentioned and our mission is to essentially improve software security and reliability so all of this work couldn't have been done without this whole the whole rtc team so we're really grateful for them

so here's a brief outline of what we'll be talking about today so we'll first talk about why we want to move towards secure guardrails and then we'll we'll talk about how do we test that secure guardrails are actually effective and then we'll talk a little bit about what we found through our research what we wanted to to do next and also how to eradicate vulnerability classes in general cool so let's start off with talking about the oauth top 10. so on the left we see this diagram that's the oauth top 10 2013 and on the right we see the os top 10 2017 and actually the os top 10 hasn't changed since 2017 so this is essentially technically

what we have currently so it might seem a little bit similar and um if we look closely we can see that a1 injection um sensitive data exposure crosstalk scripting are all still on the oauth 2017. in fact six out of the ten um os top ten hasn't changed in these four years so in case you're not familiar oops essentially is a non-profit foundation project and their aim is to educate the security community and developer community about building secure software so every few years they'll release the new os top 10 which actually prevents represents the most prevalent and security and critical security risks to web apps so essentially we see that a lot of these have

remained the same over the past years and given that a lot of these vulnerability classes are occurring over and over again we need to take a really hard look at what we're doing and how we can measurably raise the bar so let's take a closer look of what the security industry is mostly currently doing so the security industry has currently focused a lot on playing whack-a-mole with individual vulnerabilities so we've been doing a lot of pen testing bug bouncies and essentially finding individual bugs but we what we want to move towards and what the industry is actually currently moving towards is what we call secure guardrails so essentially this is this involves continuous code scanning

enforcing safe patterns and eliminating bug classes in general instead of trying to eliminate individual bugs so why should you actually trust us you're probably thinking where's the actual evidence that secured guardrails work and also the security industry has actually been finding bugs for ages so why should we change at all well we'll provide you with a few sources including what some forward-thinking mature security programs are doing and also add in our research which involves preventing crosstalk scripting in the wild so let's take a quick look at what a few companies are doing so essentially in um absent cali of 2018 a few people from netflix gave this talk and we see that they're trying to

de-emphasize in their security program manual testing and traditional vulnerability scanning and they're also trying to heavily emphasize paved road and killing bug classes in general which is what secure guardrails is all about

and we see from this talk from microsoft that essentially banning functions can be very valuable we see that they actually got 41 percent of vulnerabilities reduced from their transition from xp to vista just from banning string copy and other functions like this so we know that a lot of vulnerabilities can be removed just from banning unsafe functions and also google has written a book called building secure and reliable systems and in this book they say that it's pretty unreasonable to expect any developer to be an expert in reliability security and coding and to constantly constantly maintain vigilance when writing or reviewing code so a better approach is to handle security and reliable reliability in common frameworks

languages and libraries so ideally libraries only expose an interface that makes writing code with common classes of security vulnerabilities impossible so what google is really trying to say is that security is hard reliability is hard and coding is hard so it's actually pretty unreasonable for us to expect our developers to write secure reliable code quickly so what we want to do is we want to make our developers lives easier and our job is to sort of make it easier for them to write secure code and make it harder for them to write insecure code

cool and we can take another look at facebook too so what facebook is saying is that they invest heavily in building frameworks that help engineers prevent and remove entire classes of bugs when writing code and this is essentially what secure guardrails is trying to do we also see that in their defense in depth sort of layering we see that secure frameworks is actually their bottom layer so secure frameworks is actually the groundwork that underpins all their other defense efforts and in in addition to netflix microsoft facebook and google we can add on top of that our research which finds that using secure guardrails could have prevented 59 of the 140 instances of cross-site scripting in their open source data sets

cool now that we've talked a little bit about secured guardrails let's talk about how we tested that secure guardrails are pretty effective so we begin with a question are secure guardrails actually effective in practice and we realized that this question is a little bit open-ended so we translated this into a methodology that we wanted to follow so this became how many instances of crosstalk scripting and open source repositories could have been prevented with the use of secure guardrails so the first thing that we did was to determine scope so we wanted to find a vulnerability that was pretty high high impact and we decided on crosstalk scripting because first of all it is indeed very high

impact it's quite common um and web frameworks actually have standard cross-site scripting mitigations this means that there's concrete code patterns we can look for and it'll be easy for us to write patterns for we also wanted to prevent crosstalk scripting in our own code of course and if we look at the bottom of the slide we see that the hacker won top 10 most impactful and rewarded vulnerability types includes cross-site scripting so crosstalk scripting continues to be the most awarded vulnerability type and it was awarded 4.2 million dollars in 2020 so the next step after we decided that we wanted to aim for crosstalk scripting vulnerabilities was to determine what languages and frameworks we wanted to support

so we chose javas java javascript python golang and ruby because these languages are widely used and very popular and for the frameworks essentially we chose java.jsp flask django express reacts go http client and ruby on rails because these are the most popular and because we believe that most people were using it and we could have a larger data set so after this the next question we asked is which tool should we use in order to write code patterns and capture these um these patterns that we want to look for we essentially came up with some grep because it's open source multilingual it's pretty easy to write rules and more precise and grab we're also quite familiar with semgra

because rtc maintains some grip so we found it really easy to work with cool so here's a slide about the architecture of our research so what we did is we essentially wrote some crosstalk scripting checks with some grubs so that we could find some code patterns in ruby java javascript type scripts et cetera code in order to find cross-site scripting we then in order to find some open source repositories in open source code we queried bigquery and github advanced search api we use these two sources to collect a pretty large data set and then filter this data set based on relevancy so this includes relevant frameworks languages and also whether it represented a true crosstalk

scripting vulnerability so after that we wrote a script in order to get the get diffs and run the same group checks that we wrote on this data set and the fourth step that we performed was triaging we manually looked through all of this data and and looked whether some group should have caught a vulnerability or not cool so our first step in this process was to write text to detect violations of cross-site scripting guardrails so the first question that we asked ourselves was what rules should we actually create and how can we actually define patterns for all the ways that cross-site scripting can occur this is actually a pretty difficult question so we relied on a few sources

in order to answer this so we looked at popular security guides including the oauth cheat sheets documentation for frameworks and also purposely vulnerable apps we also looked into our own security expertise and used all these sources in order to define patterns and here's an example of a rule that we wrote so essentially the raw function actually bypasses ruby html escaping and this can actually lead to cross-site scripting attacks so it's better to not use this function at all so if we look at this pattern it's actually raw with ellipses in parentheses and this actually captures all uses and calls to raw and essentially if we look at the code on the right we see that it will

actually catch the result equals raw at string call

cool so essentially after we wrote a few rules and we looked into our sources to research cross-like scripting vulnerabilities and patterns we came up with a few policies depending on the framework so this is actually our crosstalk scripting policy for flask and we actually run it on our own code so the goal of a policy like this is to say if you follow these recommendations you're pretty unlikely to have crosstalk scripting in your code base so feel free to check out um the crosstalk scripting policies for multiple frameworks that we've released if you um if you would like to see this it's all open source and you don't need to do anything in order to see this

awesome so after we wrote crosstalk scripting checks with some grub our next step was to collect some open source data so we went on to ask the question how do we actually find samples of cross-site scripting in the wild and we decided upon google bigquery first of all it's a huge data set but it's also quick to mine so um this was one of its pros and it's also easy to obtain a parent commits and the reason why we want to obtain the parent commit is because we want to run some group on the parent commit to see whether we would have actually caught the crosstalk scripting vulnerability um but as we were scrolling through this

large table of data we actually saw that bigquery was last updated in 2019 um yeah 2019. so the question then became for us how do we retrieve up-to-date cross-site scripting information we then turned to the github advanced search api and with this we ultimately obtained around 000 commits um so the pros of course is that the github advanced search api is a huge data set with a lot of up-to-date data um but the cons were that it was a little bit difficult to obtain parent commits and also that the github api provides only up to a thousand results for each search um this became sort of a pain for us later so if we see this we've searched for um

in the messages fix xss as our query and we see that it's showing actually it's showing 200 000 available commit results but actually there is around 100 pages with 10 results on each page so it's only showing us a thousand results even though there's technically two hundred thousand so all of these results are just in the wild um crosstalk scripting results and we just never were able to see all of them so we were sad because all of this crosstalk scripting data will never be able to use

awesome so after we grab some open source data from github advanced search and bigquery we looked at our duplicates and discarded for unrelated commits and this ended up leading to us having under 10 commits for some frameworks so we decided to expand the data sets we needed more data so in order to collect more data we used some new search terms with the github advanced search api so we use things like cross-site scripting goaling fixed cross-site scripting crosstalk scripting ruby on rails and this led to around 15 or more commits for each framework and we were pretty happy with this because it was a reasonable starting point given our time frame and also um it would be difficult to

collect more data for each framework cool so now that we gathered all our data our third step was to actually run the rules that we wrote on crosstalk scripting so we spun up at ec2 instance gave it ample memory because we knew that there would be a lot of things downloaded in this process and we went right to scripting so essentially this is the sort of process that our scripts followed so with the first and just some json it would take the json output from bigquery or github that included the parent commit the commit that said um fixed cross-site scripting in it and also the repository name we then downloaded the repository on our ec2 instant

and essentially got those got that data after that we got the get disks and essentially this retrieved the differences between the parent commit and the commit with fixed cross-site scripting on it so after we got the differences we ran some grip on the parent commit so that we could be sure whether we would have caught the cross-site scripting fixed finally we saved the get tips and some group results in the local database so that we could access it for later

cool so after we got our get this and ran some grub on the data the fourth step was to triage so we actually built a triangular that displays the triage status up on top and you can click whether it's a true positive or false negative or you're not super sure about what this actually is and we can see that the parent commits and the fixed commits are actually side by side and it's very easy to spot the differences and also the same group results are displayed at the bottom so with the help of our triager we actually managed to classify some group results for each commit and we classified the results for each commit as either true

positive or false negative and a true positive means that some group detects the fixed crosstalk scripting vulnerability or some group detects a cause that led to cross-site scripting and a false negative just means that it's not a true positive so i'll show you what i mean by these two cases later here's an example of the first case of true positive which is detect the fix and essentially um this is the rails example and we see that erb template renders with if a erb template renders with html safe it tells the engine i'm super certain that this is actually safe so please render this without escaping and this can be very dangerous and can lead to

cross-site scripting if user based input gets passed into html safe cool so we look and see if our stem grip rules have actually detected this fix and then it has it has detected the exact line so this is an example of detecting the fix and the second example of a true positive which is detecting a cause can be shown by this django example so this essentially we see that this fix actually forces escaping on every email subject and body however this actually isn't the true cause of the problem the true issue is that escaping has been actually turned off globally for email templates so if you look into these lines on line 185 there's other escape equals false for

these email templates and this is very dangerous and can lead to crosstalk scripting vulnerabilities and we see that some grump has actually caught this exact example so this is the example of detecting a cause awesome and now i'll pass um i'll pass this talk on to grayson who will guide you to the end of it thanks colleen uh colleague you've done a wonderful job explaining everything up till now and so what i will do is carry us to the end by talking about what we found in a little bit more detail some research next steps and then going over how you might apply this research back in your home organizations or on your projects so with all of the steps that colleen

outlined by the end of all the gathering and filtering and triaging we were left with 125 distinct repositories with cross-site scripting vulnerabilities in it and in those repositories we had 140 distinct commits and so those distinct commits uh made up the final data set that we analyzed and so as you can see there are a couple of times where cross-site scripting was fixed multiple times in the same repository as a result of those 140 commits the semgrep rules that we applied trying to detect cross-site scripting had a positive hit 82 times for a detection rate of about 59 and so here's a chart that summarizes the data so the um the actual magnitude of the

bar is the number of commits that we had for that language plus framework uh the blue is a true positive uh for that commit and if red is the false negative for that commit and so a couple things that i want to highlight here are on the far left you can see that we only had a very small number of go repositories turns out it was really hard for us to find cross-site scripting instances and go and i'll talk a little bit about our hypotheses of why that is later and then of course on the far right side we see that client-side javascript which is javascript that runs in the browser this is in contrast to regular

java or not regular javascript but like javascript that's in node.js like a server and or some other kind of environment anyway client-side javascript had the most cross-site scripting instances by a long shot and then you can see that java and jsp had the highest detection rate it's the largest blue bar inside of it and you can see that express.js our express.js rules have the lowest detection rate which is the largest red bar and so this is a tabular view for those who prefer the tables it's sorted by detection rate and so once again you can see that java and jsp is at the top express.js is at the bottom and everything else is in between

so i want to highlight a couple of more examples of things that we found colleen already demonstrated a couple when we were talking about our definitions of true positives so i'll just highlight a couple more because they're very interesting um so this is one of our go examples one of our three and so what's happening in this one is you can see that the committer has removed this line that writes a string or a stream of bytes directly to the response writer and instead what they have done is they've replaced it with this error template that is then rendered into the response writer's stream instead and so the reason this fixes the cross-site scripting vulnerability is because

writing data directly to the response writer doesn't provide any escaping whatsoever it has no context it doesn't know anything about html it just it just writes what you write to it and so instead by switching over to the html template package and calling execute on a template that you define this package is designed to be safe as safe as it can be at least and will provide escaping for the html input that gets rendered in that template and so this is safe it fixes the cross-site scripting vulnerability but in any case this is one of the examples of detecting the exact line because our semgrep rule detected the direct write to the response writer it picked that up and then flagged it

here so this is another example this is a java and jsp template example so the committer here has removed or has added this special tag that starts with c colon out and has a value contained within it and so in jsp templates c colon out will automatically escape whatever contain whatever is contained within that tag and so this is one of the few uh correct ways of of making sure that your data is escaped in jsp templates jsp templates are particularly interesting because in so far as we can tell there's no actual automatic escaping uh enabled in the default ecosystem and so by default nothing is really rendered safely and so you have to remember to do c dot out or

or see colon out in one of the other or one of the other alternatives that do escaping for your data in jsp templates so we can see the same grip results here where it happily picked this up it noticed that these values were being rendered directly and were not being rendered in a safe way and so of course we had a handful of instances that we did not detect as well and so i just want to show you one of those here so this is one of our express js examples and what this express.js app does is have a special function for rendering errors in a fancy little svg that they've put together and so the way that they construct this

svg is by manually constructing html using a javascript template string and so you can see that the line that is updated has a message template variable within it and this render error function gets called in several places in the application and in a few of those places those messages are susceptible to user input which means you could technically inject something into this and and do cross-site scripting so of course the fix is here that the committer does is encoding html the reason that some grip wasn't able to detect this in particular is because we didn't have a way of inferring that html was being manually constructed in a template string and so if you detected every javascript

template string uh in the data set obviously that would be too much and it wouldn't be very precise and not very useful and so since we weren't able to do that we weren't able to detect this so we learned a lot through this research and it was very interesting to sort of take a slice of the open source ecosystem for cross-site scripting and to take a look at it so i want to talk about some key takeaways um our first one is that we believe that secure guardrails would work and so the idea of course is that if you if these committers were running secure guardrails with these policies that said don't do this do this instead

that a lot of these instances would have been protected against so overall we had a 59 detection rate with the policies that we developed and one anecdote that i want to draw special attention to is that our react data the data that we ended up getting as a result of running all these rules on react ended up being 41 but the only rule that we were running on our react data was just looking for dangerously inner html so that means out of the 12 or 13 instances that are there um nearly half of them were just caused by improper use of dangerously set interhtml which of course by the name uh dangerously bypasses your uh cross-site

scripting protections and so this is actually uh almost an identical percentage to what microsoft got with banning stir copy and so uh you know seeing these results is really interesting and and leads us to believe that you can go really far just by uh enabling these guard like these really simple guard rails such as disabling or preventing use of dangerous lisa and our html so the um the other 59 of that data set uh was frequently a third-party library that eventually rendered a component with dangerously inner html and so we didn't write rules to cover every third party library in existence because that would have taken forever and so but the culprit was still usually dangerously set in html it just

traveled through a couple of code paths another takeaway is that having no guard rails is a recipe for disaster this is particularly highlighted in the results for java and jsp because to our knowledge jsp provides no escaping by default aside from some plugins that you can use so if you are responsible for this technology i highly recommend you go look at these plugins anyway by default jsps do not safely render anything which means it's totally on you or your developers to remember to escape things properly so java and jsp is not secure by default nor does it make the safe path very obvious and this was reflected in our results where we were able to get a

really high detection rate in java and jsp and then colleen talked about this way back at the beginning of the talk but as we developed these sets of rules to to use for this research we wanted to use our these checks on our own code of course and so we have a flask back end for one of our projects we deployed our fastball flask policy on that and to date um as of uh you know it's been running for about three months it has detected uh and prevented two uh cross-site scripting instances that i know about from going out into production so we've got been able to get mileage out of it ourselves which has been

really awesome other takeaways that are less related to the actual results of course there were only three instances of cross-site scripting that we found across all of our searching in github and bigquery only three samples were where were we able to find which leads us to think that either like nobody writes web apps using the frameworks that we uh search for or cross-site scripting is just really hard to do and go we've looked at go a number of times for a number of different other things and it typically has the safest track record and so that it could be that it's just harder to do vulnerabilities and go than than other things of course on the flip side of that uh

cross-site scripting occurs most often in client-side javascript um there were a total of 44 commits fixing cross-site scripting which was nearly a third 31 percent of the entire corpus that we had collected of note uh in these 44 commits 17 of them were strictly due to assignment to enter html and so that was 40 of all of the client side javascript findings so you can get a lot of mileage uh out of just like checking to make sure that people aren't assigning things to dangerously center html or innerhtml and that will go really far so this is an example of one of those inner html assignments you can see that they're manually constructing a html and then

directly assigning it to innerhtml which creates a cross-site scripting vulnerability some considerations here of course are that we didn't have any measure measurements for false positives um we wrote our checks as sanely as possible so like we didn't try to detect 100 of java for instance because that would have given us 100 detection rate right so we were saying in how we approached this we didn't like try to cast a white that we weren't trying to like fudge the numbers or anything like that but one of the other reasons that we didn't really focus on false positives for this research is because we think that false positive reduction would yield similar results because the iteration process on reducing false

positives for a check strictly takes your total result set and tries to weed out things that were not true positives initially so we think that the true positive results would be approximately the same and the results the detection results would be pretty much the same as well so what do we want to do with this next obviously we want to apply this to more languages and frameworks we found it useful for us just on our flask back end and we want to apply that to other things as as we become aware of them or interested in them and then of course we also as i just mentioned want to spend some time reducing the false positive rate

for some of the checks um we have uh methods of doing this at r2c uh we have like a massive code scanner that goes out and finds much stuff so we can we're pretty confident our ability to do false positive reduction we also want to cover additional vulnerability classes obviously the os top 10 is a very easy list for us to target and a couple of the ones that we've done so far in addition to the cross-site scripting ones are command injection sql injection and unencrypted transmissions um and so if you're interested in actually running any of these rules some graph is open source and you can download them you can download some graph and you can

download these rules for free if you want to scan your own code for these things one other thing we want to do is prove this approach on private code so obviously this research focused exclusively on open source research but we really want to make it applicable to people who are trying to run security organizations um and so we want to see how this test holds or this process holds up on private or internal code so some things some questions that we would need to answer for that are like do companies use internal frameworks rather than open source ones so since everything we did was open source if there are internal frameworks that need custom rules written for them like how would

this apply to that and then another question is like are there organizational challenges to adopting secure defaults secured guardrails excuse me at scale and so you know sometimes it's hard to roll out new checks it's hard to roll out checking programs and so what sort of political barriers or technical barriers are there inside of an organization so i want to conclude this talk by talking very briefly about how you can take this back to your own organizations and so how you might do this we propose this four step process for how you might take this eradicating vulnerability classes back to your own organization so the first step is to select vulnerability class of course you

have to pick one there's a lot to choose from and there are a lot of different ways of choosing which ones you want to focus on and so if you have vulnerability data like a vulnerability management program you can rely on that to tell you which vulnerabilities you experience most frequently or perhaps which ones have the highest impact and then you can select one and start to move down this process and so the second step is to select safe patterns and make it the default and so this is that policy that we showed you earlier it looks kind of like this so you have to identify what are your secure guard rails what don't you want your

developers to do that would result in the vulnerability and then identifying what the alternatives are the recommendations for them and then you have to make that the default pattern in some way and so some ways to do that are steps three and four step three is to train developers to use the safe pattern get the word out uh use your cicd pipelines if you have the ability to inject messages into that while you scan for code patterns do that otherwise hold like brown bags with your developers or if you have any security training or onboarding mention it there any coding guidelines that you have as well can be updated with this those are great places and then we

humans are fallible so even if we've had this beaten into us we still forget things and so you have to use uh technology to enforce the safe pattern and so this would be using some sort of uh linter or like text scanner to regularly enforce that these patterns don't get into your code so usually this happens at like a ci cd level or build time something like that once again this is an example of that policy so this is kind of what it looks like when you're finished all these things have corresponding like uh linter rules that get run and check your code on a regular basis if you're interested in um the cross-site scripting checks that we

wrote for every uh each one of the frameworks that we mentioned we've publicized them all for free because we did a lot of the work and if you want to benefit from that work we figured why not so here you go you can check out the policies like i was just showing you at some grip dot dev slash docs slash cheat dash sheets and they will all be there and if you're interested in more details about how you can take this back to your organization i'll refer you to a talk by two of our colleagues clint gibler and isaac evans at global appsec san francisco 2020 about eradicating vulnerability classes and so this was a

very high level overview of that they go into way more detail here at this talk so if you're interested please check it out and so with that we will conclude our talk please join us in the discord channel for questions and we hope to see you there great great presentation thank you so much for for participating in b-sides las vegas this year and some of this talk i was really excited when i saw this this talk on the cfp so thank you so much um thanks for those of you yeah for those of you uh watching uh this in the discord if you have questions uh for the speakers now is the time to speak up i am monitoring the discord

i will ask the questions as you ask them i also have some questions that i want to ask since i watch the talk so um just throw your questions on the discord and i'll ask i'll ask the speakers um one of the questions i'd like to ask speakers is uh is what was the hardest part of this research for for the both of you yeah um i'll start that colleen do you want to want to add your thoughts after um and the hardest part for this it was probably triaging all of the results so we downloaded a ton of data and we got the diffs between the parent and the fix and had to look at each one and compare

that with the results of our scanner in order to make sure that it was like accurate that took a long time and uh occasionally there was debate about what it meant to be like a true positive versus a false negative or something similar so uh it was a herculated effort and there was a team of people uh involved with it not just the two of us um but that was that was definitely the most challenging part i think and even now even now i'm not certain that we got every little bit right when we triaged everything but we did our best

yeah um how many how many dips we talking about i'm just curious that the number you mentioned lots of diffs just curious yeah uh so we ended up with like 140 distinct commits and that was after manual review of about 700 commits but we started out with we started out with a bunch i think there was like somewhere between three and five thousand but we were we ran some uh we ran some uh um like deduplication and heuristics on that first in order to get us to the original 700 but like manually like our team looked at probably 700 different diffs throughout the entire thing um uh all right what surprised you both the most

uh when you started doing this research um what surprised when we started uh i think the most surprising thing was probably the the difficulty of getting the data because it seemed like it was such a easy thing to get and we talked about it in the talk obviously with this limitation with the github api we didn't show this in the slides but uh we actually talked with like a github engineer and we were like hey could you like give us like your insider access to this api and they were like oh no we can't do it either we can only access a thousand results we're like oh so like it's it's limited you cannot do any more than a thousand

results from a query oh interesting

thank you

um you know you mentioned in your talk uh you know looking at the parent commits what would you say the purpose of scanning the parent commit is versus the fix commit yeah um so the if you think about it the fix commit doesn't actually have the vulnerability in it um and so we needed to be able to scan the parent commit which is any commit prior to the vulnerability being fixed in order for the scanner to pick up the actual code pattern that would have indicated the vulnerability so sometimes depending on how the how the vulnerability was fixed the fixed commit would have been sufficient but in order to get 100 of the time we wanted to make we scan the parent

commits okay um if if a corporation so at the end of the talk you mentioned you mentioned a slew of recommendations um you know i work for a large enterprise who's also cheap so you know if we could only implement you know one or a handful of your recommendations which ones you know would you say would be the the most important ones to implement and why yeah if you have the capability to get a scanner up and running in a continuous fashion um it doesn't necessarily have to be a ci cd pipeline you could also do it with like if you use git get pre-commit hooks out to all of your developers but as long as you're getting some scanning

done with a policy that you've either developed or has been developed for you because we made all of these rules for you for instance that's probably the number one recommendation is like use technology leverage it um and uh hopefully that will help the most calling me out you have anything to add

yeah yeah it's a little bit like having a security reviewer constantly checking like that's the whole idea of it is like making sure that something or someone is reviewing something all the time right nice uh it i work for if i work i'm just like i do but if i worked for a company that's never done you know lots of industries are are finding out cyber security is important now so you know if i work for a company that's never done this type of thing before and i have a 10 year old code base that's never been reviewed um you know what what do you each recommend yeah um i don't know if this is true

i imagine for situations where somebody's responsible for a java stack similar to the one that we talked about in the research this could be the case and one of the things that stuck out to me is that it's difficult to use the technology to your advantage there's no way of like you know automatically escaping stuff in that tech stack or at least that that we're aware of however there was a there was a really cool plug-in that we had found that will automatically escape the jsp templates for you as long as you set up your java code base in order to use that plugin and so my recommendation would be to look for something like that if you have the

ability to use a plugin or a library of some kind that automatically makes your stuffs stuff safer makes it harder to do the insecure thing i would definitely recommend that and so if that's not possible for some reason if you don't have a library um scanning is like the second best thing and so if you can scan for common gotchas um do that and uh you know then you won't have to worry about whatever gotchas you're scanning for anymore

nice uh all right i'm gonna switch gears a little bit here uh what excites you the most about this type of research yeah at the end uh we talked briefly about um false positives and i in particular uh am typically excited about reducing false positives and so um you know we at r2c are the maintainers of sungrip which is the scanning tool that we used and so from this we now have the ability to like build in some features to our own tool to help us with false positive reduction and so in particular um one of the uh examples that we talked about that we couldn't find typically was html being constructed in a javascript template string

and so our program analysis team recently added or is adding i don't think it's added yet is adding html as a language to our scanner and we also have this neat ability where we can like um extract pieces of code and feed it into a different parser so we in this case we would be able to like extract the template string feed it into the html parser and if it like parses then we can scan for the template variables and if we find that uh then we can reliably report that as a true positive much more much more so than not being able to find it at all which is our current capability but like this is in the works and i'm

really excited about being able to like deploy this cool tech in order to find this stuff nice

calling

nice okay uh so you know last question here where can people find the tooling and your research that you were talking about yeah for sure um so we have uh the uh we have the data set on github or a link to the data set data set on github as in addition to all the code that we used to um process this stuff um so i'm going to post that in discord now um there's a couple of things that we're still working on in order to like make this a little bit more accessible we wrote an article about this research uh a couple months ago but it's it honestly just didn't turn out very well like it didn't mean our

quality part um and so we actually found that doing the slides and the presentation was a lot easier than doing the article so there will be an article that will be coming out soon it will be linked to the readme in the repository that i just linked um but for now um most of the relevant information is in that github repository great well thank you so much thank you for being a part of the b-sides las vegas campfire and uh you know if people have questions they can still reach out to you on discord and uh thank you so much cool thank you