Got popcorn? What's on the Vuln Channel tonight?

um this is rob mr rob gerdonic is that correct okay great um you're from the bay area and you've had uh several years of experience in software development of tools and security for companies all right so with that please give rob a warm welcome and here we go [Applause] thank you [Applause] yeah i'm rob tridonic i work at the trust engineering team at roku at roku we call the security team the trust team it's pretty much the same thing but i kind of like that we call it the trust team because it keeps in mind what our end objective is and is that achieving customer trust and obviously security is essential to achieving customer trust

and today i'm going to talk about our approach to vulnerability management i'd also like to acknowledge lilly chao who helped prepare these presentation slides here we have the ceo of roku anthony wood and he was asked a question of what he thought about highly interactive television and he responded in the media that many companies just don't really understand the attitude people have when watching tv they they really just want to sit there drink their beer and watch tv and for beer you could substitute anything else you might enjoy like popcorn or herbal tea or whatever you enjoy but our question was is the same true for developers and vulnerability management but being at the lobby yesterday i think

there are quite a few developers here who really enjoy their beer so i think we're on to something vision is that receiving this vulnerability data should be as simple as streaming the latest episode of your favorite tv series we set out to build a platform to make this vision possible at roku and this presentation focuses on our static code scanning but these same principles can also be used for dynamic security testing as we sat down to think about this platform and create our requirements we decided we wanted to have a plugable framework we have different scanning modules technologies uh for a python application we need a special component to scan python apps likewise for java apps terraform

files etc we want to be able to plug in different technologies as they become available our system also needs to be very scalable we need to be able to scan continuously scan hundreds of applications or even thousands next we need a very flexible integration we have a variety of roku has grown by acquisition like many companies we have a variety of systems for ci cd integrations git lab github other system a variety of systems that developers use and they'll probably be many more in the future and underpinning all of this is developer happiness we aim to delight our customers and our customers in this case are the developers using the system about our the design of our platform

the heart of the platform is a jenkins cluster that we run in kubernetes and this gives us the ability to auto scale up and down and this way this cluster can perform automated scanning of hundreds of apps and it could go up and down as demand rises and falls and and then from there we have our scanning bots these are uh scanning a scanning bot to us is a scanning tool that we containerize according to a specific format standard format and each scanning tool is containerized as a bot and jenkins has the ability to run these bots in a certain order as needed to scan different artifacts and applications next we needed a database and dashboards

to store the vulnerabilities and we just decided to use elk elasticsearch log stash and kibana and that enables us to store our vulnerability data search the data analyze it create dashboards and so forth next we have an integration layer where we actually integrate with the ci cd pipelines and there a main goal is to detect vulnerabilities in merge requests pull requests and notify developers before the code gets merged into the main branch and definitely before the release we also have integrations with slack because most of our developers use slack and this is the way they can see receive these alerts as well so we want to meet the developers where they are and give them

timely and actionable vulnerability information we also developed our own viewer tools web-based user interfaces for users to to view this data in great detail here's a diagram of our platform again at the heart of this is our jenkins cluster and we have different scanner bots for different technologies like go language code python code we have a scanner for java javascript secret scanner and we interface with the ci cd pipelines which could be gitlab github and we also interface we use jfrog artifactory for our container images so we pull from there to scan any images that are created by developers and we have a scanner bot for scanning containers these scanner bots write their output to

s3 storage and that's really just short-term storage and then periodically we have our elk stack we'll import this s3 data using log stash import it into an elasticsearch database and that's where we can query the data we can create dashboards and have long term storage of the vulnerability information there we can also we automate the process of creating jiras to the click of a button and we do require someone in the trust team to approve the jira before it's created when that's necessary

so these are some of the different scanning bots we have created [Music] each one is a containerized tool we have a sem grip we containerize some grip which runs static analysis for java python javascript go and ruby and other languages for java composition analysis we have both oss index and dependency check tools oss index also supports go and python we have containerized npm audit so we can do composition analysis on node.js apps we have banned it for python code we have tfsec for scanning terraform terraform files a trivi for container scanning and get leaks for secret scanning we also use retire.js which mainly for client-side javascript because we found it works well with that all of these are open source tools that

anyone could use we just found it convenient to package them in containers and but we we could also support commercial tools if we if there's a commercial tool we want to use we can just package that in a container as well and run it as part of our overall framework

here's a screenshot of a jenkins pipeline if any of you have used jenkins each software repository a pipeline will automatically be created to scan that that software repository and there's a sequence of steps called stages and we start with a discovery stage where we have a a container that we created which its job is to scan the code and look for what kind of files are in the project so if it has python code will trigger our python scanner if it has java code we'll trigger our java scanner and so forth so that's that process of automate is automated and a nice thing about using jenkins is it has the ability to parallelize the the stages so we can run all these

different scanners at the same time and that really improves performance because if you're giving real-time information to developers you want to have really quick response time

here's an example of some of the issues we can find with our framework if there's a vulnerable third-party library in one of the applications we have the oss in like in this case java we have oss index dependency check we could find if they're using for example a insecure version of log for j we could recommend that they update the latest version of log4j uh for secret scanning we run git leaks and we could identify if a developer put a hard-coded key a secret aws key for example and we can detect that immediately and alert the developer ask them to invalidate that key create a new key and store that key more securely the next two examples are with semgrip

static code analysis when we scan python files it could turns out the default xml parser in python is not very secure if we see that we recommend that the developers use diffused xml it's a more secure python parsing library if we see any use of for javascript if we see like dangerously set inner html that's unsafe function will flag that and recommend a secure alternative

now i'll talk more about our ci cd integration so we have a variety of ci cd systems at roku we're not unfortunately not standardized but you may have other big companies may also encounter this but the git lab is probably the most popular at roku and for this we have a web hook integration it's a pretty simple one-time setup it can actually be done by the administrator so the developer really doesn't have to do anything so we set up this web hook and for their repo and this anytime the developer creates a merge request pull request or and or updates that merge request it will scan their code automatically and they'll receive an alert something

like this it will say the trustee security scans found some vulnerabilities in your code please click here for details we don't bombard this view with lots of detailed information because the screen real estate for a developer is very small and we just give alert basic information and then we have a link where they can click for to dive into all the details and and then from there they can go ahead and fix the issues and once they commit the code changes to fix those issues the scan will just automatically run again because it'll be triggered by the commit and hopefully the issues are fixed and they'll get another alert that just says security scans did not detect any vulnerabilities

as i said before we created our own viewer application just for when the users click the developers click on those links we pull up a detailed view of the vulnerability in this case it was a purposely vulnerable application it had a hard-coded rsa private key in the code and we just point that out on line 15 of this file and we have a link to that file in the gitlab or github repository and instructions to to fix that

so as i said before we are using elk for our database and dashboards just took some screen captures we have dozens of different dashboards that we create with kibana it's a open source free tool and we one of them we can capture all of our vulnerabilities by category across all of our repos that we've been scanning and we can this helps the trust team see where any problem areas might be popping up vulnerabilities that are frequent and then we could focus on our remediation efforts on those highest occurrence vulnerabilities likewise we can we have a number of line graphs to track remediation of vulnerabilities this was from a cve that over time we could saw the cve

had many occurrences and as we remediated the issue we saw the graphs going down and we wanted to see all the these lines going sloping downward and of course if there's a new cve that comes out the the grass will go up and we can monitor over time and see them going down so picture is definitely worth a thousand words so we're not done we're not done yet we're still enhancing our system and we're next step is we're modifying our pipelines to generate a software built with materials s-bomb currently we only flag if a file exists in the artifact or repo if it has a known vulnerability but while we're in there we might as

well create a full software build materials and we can store that spam in elasticsearch and then we have the ability to query that with our kibana tools and the big advantage of that will be if tomorrow there's a a new vulnerability that comes out in a supply chain vulnerability they always ask the security teams do we have that are we using that uh component that's everyone's worried about they they want to know from the security team right away and now if we have this s bond we could quickly query all the s bombs for all of our applications and and we will know if we're impacted by these supply chain issues because there's been several

in the past year and they'll i'm sure there'll be more in the future and um and then because the alternative is would be to just wait until a cve is created for that file and wait until we run another scan of all of our repos and that could take uh many days but with the s bond we could quickly determine if we're affected by any kind of issues like that another area of future work is blocking merge requests when vulnerabilities are detected currently we just simply notify developers if they have any potential vulnerabilities in their code and it's their responsibility to fix them but we are looking into the ability to block them especially for any highly

critical vulnerabilities and yeah we're looking at improving our coverage for different languages and technologies because developers are always innovating and using more technologies and we have to adapt our tool to keep up with that and we want to definitely reduce false positives wherever possible and we'd we'd also like to publish our implementation details to the community because as we've been talking about this with other colleagues at other companies we see there is a big need for platforms like this in the open source world and so and we'd like to consider open sourcing our framework code especially if we find others who are interested in working with us and roku is hiring our security team there's

a link to cyber security jobs if you're interested

in conclusion yeah all of us are using a wide variety of scanning tools both open source and commercial we're scanning our code constantly for vulnerabilities and what's needed is a platform to bring this all together in a consistent way and not only that but a mindset of quote streaming this vulnerability data to the developers that's focused on really extreme simplicity and ease of use developers are super busy and we need to really value their time and give them actionable important information when they need it and you know there are other platforms like this out there there's commercial products that are trying to do something like this i'm sure you go out if you go out there in the vendor booth one of

those companies would be happy to sell you something that does all this but we want you to know that you could build something like this with just open source components like we have done using you could use jenkins for automation of the scans containerized the bots of open source scanning tools and run those you could use elk for your database and your dashboards and you could build something like this purely from open source and you may not exactly end up with exactly what we have but hopefully you can use the same type of principles and think about bringing all your vulnerability data together into consistent architecture all right thank you very much and excellent i really love that talk

i do i mean i i look at source code all the time for customers and um uh what about uh sonar cube twist lock is twist lock free i think sonar cube has a community edition but yeah yeah there's a lot of really good commercial tools out there as well uh we just decided to do using open source technology uh just because we could really tailor fit it to what we needed but the same principles could be applied to a number of commercial tools as well well you all got it together i'll tell you that much all right so um folks out there questions comments ah there we go okay um let me talk to this person real quick

who raise your hand over here okay thanks um i'm wondering what's the latency introduced in build times due to all of these containerized bots and what developer feedback has been sure um so what our approach is to to run these scans during the merge requests or pull requests and automatically run when they create or update a merge request and there is the scans depending on the size of the repo it could just take a minute or two or it can take 5 or 10 minutes for the larger applications but we our decision is we won't block them from deploying their code so even if they want to go forward with the merge request before the

results are ready they can still go ahead and do a build specif especially if it's an emergency urgent situation but typically merge requests do have a significant time because that's when developers do the code reviews when they're reviewing the code before it's merged and released so there usually is a period of hours or more where we have the opportunity to do this scanning so we have not had any complaints about this taking too long and we're not blocking them from proceeding with their build fantastic okay someone else you had a question yeah so you mentioned uh you want to reduce false positives i mean the entire industry actually wants to do that so what are you actively doing uh

specifically to do to reduce false positives right yeah so we um we have scanning rules since we as we're building our own tools we're able to customize the scanning rules and we're able to if their issues are flagged in our viewer as false positives we can review them and if one app reviews it we can consider if it's applies to others so it's a learning learning based feedback system so if as we see false positives popping up we can just adapt our rules to that for future but yeah it's an ongoing battle and but there's no way to avoid it but it's something we are constantly improving and learning learning from excellent other questions out there

oh sorry make you come all the way up here yeah um uh do do you get any refend uh repeating offenders for developing developing teams that continuously release vulnerable code how do you manage that um yeah there are teams some teams are much better than others resolving issues and we we have these dashboards so we can actually track across different teams so we we know from our dashboards which teams have uh are better at responding to these vulnerability alerts than others and then we could that we requires a more personalized approach to to work with those teams and understand if there's any underlying issues why they're not fixing the issues promptly but and we have really precious

few resources in our team so we want to use our resources to meet with those teams that have these kinds of problems the most and that's where the cabana dashboards really help us really zero in on any uh product teams or application teams that need that extra support from the trust engineering team oh very good um yeah i often thought about the future of like using machine learning to pick out bad developers in code or you know bad code always copied off of uh um off the web right so any other questions before we wrap up here all right well y'all have been a tremendous audience rob you've been a fantastic speaker all right thank you very much sir