Malicious Intent in the Open Source Supply Chain - Ryan Voloch

hi everybody hi besides isn't it great to be back yeah awesome awesome thank you to everybody who puts on the besides talks i appreciate everybody who's involved in volunteers and sponsors like like andy said just everybody thank you um so uh you're here for malicious intent for the open source supply chain uh so thank you for joining me uh so a little bit about this presentation its goals are to help you uh understand the open source software supply chain threats uh and help your organization make it better risk-based decisions and recommendations for improving your support your supply chains and your devsecops processes uh so what we're going to do is go uh go through a little bit about me

uh talk about why this presentation as this is a journey much like sid had a journey along with his presentation we'll go through some key concepts talk about open source uh we're going to talk through two primarily two different types of supply chain attacks third party and first party i'm going to talk through some some concepts some example attacks uh and some guidance and recommendations and some really cool tooling i found along the way uh through both of those and we'll wrap up with some conclusions so just a little bit about me you know i'm just a an excited cyber security professional trying to help the rest of the industry learn uh what learned what i learned

so several years ago i discovered there's a whole new world of cyber security within the uh the challenges uh security challenges within the software development community um so i really wanted to learn more about this and so as i was preparing for this talk you know i i found several really interesting things that got me excited along the way like metric frameworks maturity frameworks compliance frameworks tooling and i do include that a lot of these things into my presentation so i tried to make this presentation as interesting as both technical and uh leadership um so i may and also i may go super fast through the material but just a reminder this will be posted

on youtube and the slides will i'll give you a link to the slides at the end of the deck so very quickly about me i am these things in this order i'm a christian husband father and a manager of a cyber security team at the for the national technology energy lab our national energy technology lab is a support contractor under maximus detain i got lots of experience very much like sid you know just many different industries and cyber security i got a bunch of certifications as well but do note that i am not a programmer or developer i do know how to program i use open source software like much of you do but i i am not in software management

for in development for a living but this presentation is my perspective as a cyber security professional to you who are cyber social security professionals all right why so so i help organizations to improve and mature cyber security it's my job so i've heard of many people having experience with the solarwinds investigations and incidents and it certainly was an eye-opener for a lot of many and many in the industry so i also work for an organization that's heavily relies heavily on open source software so and i expect every one of you in the room uh has has all downloaded and utilized open source software and binaries so i had a question i had i wanted answers to

a lot of questions so what i did is i did a ton of research i read a lot of articles i read um i read blog posts i listened to podcasts and most recently i attended the 2021's north american supply chain security con just a few days ago um finally i want to be um i want to enlarge others to encourage you to speak publicly um so i want to be example to others that it's okay to to pick okay and very possible to pick a topic you're not familiar with learn in a short amount of time and present what you learned what better way is there to learn than to force yourself to to teach others

so i didn't get uh i did not get the answers to all these questions but i got pretty close so let's go through some key concepts um you may have heard of linus's law it's the assertion that given all eyeballs all bugs are shallow and that's the the um you know a lot of eyes help make good code quality and good projects um it's also i'd also like to think in the cyber security world it's kind of a separation of duties control this uh um so and this also helps rely make our this all this concept also helps make secure code uh and and and and good quality code security um obviously code reuse and refund code

functioning reuse is an obvious goal we'll talk you know that's just a key concept in open source and software develop in general and then there's an article uh about 35 years ago by ken thompson called reflections on trusting trust basically it was an article that referenced one of the first software supply chain proof of concepts where he wrote some c code to up to attack i guess the compiler of c to do malicious things we're going to get back to the the concept of having a build process being attacked so a single software supply chain this covers so what is a software supply chain it covers from the keystrokes of the developer developing the code

to the source code management solution like github to the continuous integration continuous development uh processes um and uh importing the dependencies where where a piece of software would import dependencies uh like sid was mentioning with ubuntu and packaged just all the way to the distribution points of doubt having uh being provided to the end user distribution and open source uh as sid did mention apt yum uh in the open commercial world it is uh like the apple plays and google stores um so uh a attack against the supply chain is any unauthorized google defines it through pretty simple on any authorized modification to within that process and we're going to go through many different attacks through this process

like i mentioned there's two general uh software supply chain attack types first party and third party but first let's you know talk about open source it is the scope of open source it's huge and just this isn't good just to review some of these statistics 98 of code bases contain open source components and roughly about of 90 of uh enterprise software is comprised of open source code so the the community is huge and it's growing it's almost growing exponentially so that was something that was like it was something that really kind of surprised me as i was doing some some of my research um interesting fact too that the windows operating system is built on

a huge a large quantities of open source code so what i wanted to start the talk uh i want to talk about third party software supply chain attacks basically it's somebody else's software supply it is what sid was referring to as upstream um and we're going to talk so with when you talk about third party uh other people's code you're talking primarily about the dependencies a modern application contains roughly 128 open source dependencies so this this you can see how just a lot of different all the complexities within the supply chain specifically around uh dependencies there's lots of challenges there this is what's known as a dependency graph very very very simplified so you can help understand if you have a

compromised open source project well downstream uh downstream projects or even your commercial soft commercial off the shelf project can be affected very similar to uh inclusions between other open source projects that make their way down into your final open source project this concept can be applied to both malicious code functionality and injection of injection of code and the vulnerabilities as as citizens mentioned

i want to go through an example attack this is an attack that happened maybe two years ago with the event stream open source solution uh basically uh event stream was a dependency a bitcoin wallet called copay so what happened was the attacker made some really meaningful contributions to the event stream project was able to the owner of the project granted that attacker permissions to take over the the package and malicious code was injected by that attacker and so the real key point here is uh you know at the uh upstream uh open source project could affect many different downstream and target downstream uh applications and projects at the time event stream was used uh by roughly it

was used by roughly about 1 600 packages this event stream project uh averaged roughly around 1.5 million downloads a week at that time uh the good news with this attack uh that there was really no known um uh impact to uh the bitcoin users of the of the of the application uh so that's from bitcoin in their statement or bitpay in their statement so this graphic demonstrates a big problem for open source software you know developers work on popular cl pop what is popular and and people stop working on old code um you know what happens is it typically happens is developers typically starts with a personal account the project gets big and then it gets old and then the

projects can can expire this is a big problem that one random person in nebraska can often get hit by a bus or is hired with an organization that pays huge amount of money and forces that developer to stop working on the open source project this problem may have contributed to why the event stream project owner may have granted permission to that that attacker would possibly seemingly see uh maybe little vetting so purposeful zero-day vulnerability injection so sid was talking about maybe in vulnerabilities that were not necessarily uh purposeful but attackers the 2021 state of the software supply chain report by sonotype states that the attackers are starting to more and more inject zero-day vulnerabilities into open source

projects that feed the global supply chain so this is a very scary thing um you know you you you all remember heartbeating struts i mean that was a vulnerability that affected thousands and thousands of organizations and it was a big cleanup now i have no evidence that either of heartbleed or struts or vulnerabilities were maliciously injected i'm not saying that but they are great examples how a single vulnerability can affect thousands of downstream applications that depend on it however with the statement from the the report uh i've in my research i've really found no good examples of a purposeful zero-day vulnerability injection attack so if anybody knows of a good one please just see me afterwards i'd be interested

to hear that example

all right another attack within the dependencies is the dependency namespace confusion attack basically within uh this exploits devops tools um and their their default configuration and how they uh automatically update their dependencies um if a tool is configured to automatically update dependencies whether they're looking for the name typically the name of the of the dependency or the the package and a version number so if they see a dot z.1 update they're gonna go grab that from an internal repository what this attack is does is it takes um some some code tools are oh sorry the attacker posts a public version of this very similar named inversion or a dot one version of the repository that

is malicious but the attacker needs to know some of that inside information to do that so um basically the one thing you need to do and look out for this type of attack is to configure your devops tools to appropriately prioritize where they get their packages from so make sure they're configured for your internal trusted repositories rather your your potentially public repositories or have some other controls to check prior to download all right so i talked about a lot of problems what do we do i want to go through uh something called what i call dependency management health with developers and warning for cyber security professionals go through some tooling and some really cool just really cool processes and tools

the good news is a lot of these processes and and the things that i'm going to be going through are measurable and are helping development teams working toward automation so first a warning dependency management help so project developers as sid was mentioned they struggle with managing dependencies it's a royal pain it's a it's it's it's a balance between innovation and getting the the project done and maintenance of your code they struggle with this and i just want to make sure we're all aware of that struggle however risk-based quantification and decision automation capabilities are improving they're getting a lot lot better but they may not be there yet um just some pro tips for some rec uh

some for working with developer tools or development teams just be patient don't force when don't force your tools or processes let them come up with their solutions and seek win-win solutions meaning when you're looking at solutions don't look at just security as the um as the primary goal help them mature their code mature their and make their processes more efficient at the same time that's a whole other talk we can go to uh but just quick tips the uh the sauna type report i mentioned earlier has some pretty good guidance for developers and there's some eight rules when when to update uh your dependencies we talked about sid talked about updating the most latest and

greatest dependencies this report was suggesting that we do not or there's concerns with updating uh the latest and greatest you may not want to do that so there's some uh one of the the key takeaways of this report is to consider how you can make the best decisions for updating your dependencies so there's a lot of tooling out there github has a the github has dependencies and a dependency graph uh in your in the project you list all your your your dependencies uh there is some alerting functionality that's that's possible from like the beta bot and o wasp dependency track is also another tool to help be alerted of uh of of issues within your dependencies

uh owasp dependency track that primarily uses s-bomb which i'll get into a little bit this is an example of a dependency graph just showing just very how complex it could be and this is a really interesting solution from the open source security foundation uh it's a that foundation has a lot of different really awesome interesting projects i reference a few of them in this slide deck but what this there's a project called scorecard that allows you to input your git repository and the output is a weight-based scoring metrics across 16 different tests um it helps to identify whether your dependencies are safe or not why is this why is this really interesting well it could help you make

automated risk-based decisions perhaps your organization or you can define a minimum open ssf scoring criteria for your dependencies here are the 16 checks it covers a lot of different uh areas of of the package so software build materials is very important with this with this problem um what is the soft build software build materials well i'm going to just call it s-bomb well it it it really helps with identifying with with automation and identifying the trust of a software package it increa it includes ingredients of your package such as source code summaries application builds container images running containers checksums hashes it's again it's the whole complete inventory it's very very very good metadata of a project

a major project will have an s bomb the best product the s bombs are developed at build time because that's where a lot of information is gathered and it also includes the dependencies listed in the dependency graph i referenced earlier this data can be very useful for other for security functions uh in asset management functions uh along the lines for patch management having getting good threat intelligence of what's uh listed in the software uh vulnerability management understanding the dependencies and the the chains there and disclosure and it's also used for disclosure processes uh because there's information that lets you know okay well if a certain uh dependency you can you can let other downstream uh see repository instant

response teams can let downstream um package owners to let them know what's some problems within their their downstream packages there's lots of great tools that make the out there that make this easy there's get bomb and sift and spdx is a standard um uh another standard that's out there is o wasp cyclone dx uh that's a that's a another tool it's been also popular as well um the you're you're to see you know recently the the u.s government has um increased the awareness of the software supply chain attacks um and s-bomb is a a very a frequently referenced um solution to a lot of these problems okay first party software supply chain attacks basically this is your projects

your your organization in its software supply chain this relates to many attacks that the devsec ops controls and tooling prevent solarwinds is a prime example of a first party supply chain attack the malicious code was injected into the orion software at solarwinds through a highly targeted part of the ci cd build process that's how originally the malicious code got into the solarwinds project that's an example of first-party supply chain attack so um here are some points i got these uh some great graphics off of the from the salsa team um and i i give credit to them for the for a lot of the the the graphics in here they explain it much better than i can explain it

but all the way to the left we want running through the software supply chain submitting bad code yep you can have a bad actor like the event stream actor just writing bad code software uh you could compromise the the github page example or your github or your your software control management solution and as you can also attack between the software management solution to your continuous integration and continued development build processes you could also compromise that build platform and as i mentioned before you can inherit or include bad dependencies as the software is built it's it's typically handed off to a package distribution and you can also attack the package distribution point finally you could get the user to use a

bad package and as i mentioned in the um the the article before with the uh reflections on trusting trust uh attacking the tooling of developers is becoming more and more frequent to inject malicious code or malicious functionality within the software supply chain here's an example attack from kovkov codecov was is within the software supply chain it's a developer tool and what it does is it tests your application and uh see it checks how much your applications tested as part of that ci pipeline uh what an attacker did using some stolen credentials from i think a mistake from the container the attacker was able to modify a bash uploader script within the codecov solution that then the solution was distributed

to a lot of developers large organizations as this reference there earlier this year and so um there was roughly around you know kodkov had roughly around 23 000 customers and users and anybody using this compromise version of kodkov would have been com would have been compromised what happened was the attacker the the impact of the attack was secrets are uploaded to the attacker's remote server one of the things that was interesting with this attack and could have maybe could have helped uh was a concept here actually no a key concept here that that did help when this attack was a a code signature was identified to identify a hash mishmash a hash mismatch this is how the malicious code was

identified so signing your your artifacts throughout the process is a very very good thing to do to help identify malicious intention within the supply chain okay more problems i want to talk about a tool called salsa talk about some repository controls and another tool called source rank so salsa supply chain levels for software artifacts what this is it's an end-to-end framework to ensure the integrity of the software you are using uh or or including is secure it primarily focuses on the attacks with between b through g and making sure that that process is very mature and solid it's got four assurance levels four being the most assured uh the most most controls it is a compliance framework and another

another way to put it's a compliance framework and there's a bunch of controls there's roughly around 30 controls um checks to make sure that that will get you these four different levels one i think is you just have a few different controls and four you have all like 30 controls and there's there's levels in between this really helps empower our software development teams to automatically check the integrity of software artifacts when they choose to include into their projects this is developed in direct response to these known supply and chain attacks it also includes s-bomb validation code sign scoring controls uh another thing to note uh that use effective use of salsa could have prevented the kodkov and the

solarwinds attacks or at least could have detected it another interesting fact is google has been using salsa since 2013 well their internal version of of salsa since 2013 and they require it for all their production workloads there are some alternatives some similar alternative models out there there's the oh wasp component verification standard scvs it also has very similar assurance levels one two and three there's a microsoft supply chain integrity model or scim for short and there's also cloud native computing foundation cncf's soft software supply chain best practices all very similar models so let's see how salsa can be applied or a solution like salsa can be applied if we have a compromise open source project that's salsa

assured at salsa level one and by the way salsa each open source project will be scored provided an assurance level so individually so salsa level one if i don't if there's uh the the two middle the two middle uh open source software uh projects there their higher level software assurance levels they may not want to accept salsa level one inclusions into their projects so it's going to be stopped right there um so how so one of the the things that the the salsa team wants to see and help once wants to do is help other or other projects to all get salsa uh assured and uh so that's gonna be very difficult to to gain adoption um i don't know

necessarily sure how they're gonna go about doing it but they do say that they might they are you know suggesting third party assessors to assess your project and then you know give you a give your assurance level okay i want to focus more on to the uh the developers and submitting bad code this is more along the lines of malicious develop a malicious code injection you know people actually inserting malicious code there are some tools like package analysis that help detect behavior-based uh maliciousness using your behaviors heuristics uh to help identify malicious soft malicious code in their software there are some tools for the software composition analysis tools which can help detect malicious code detection uh

however in in my opinion you know with the the research that i did in the attacks that i've reviewed tooling cannot effectively detect malicious function injection um yes security industry is very good at detecting um things like remote access tools um viruses and and those types of things that are that are known but we're here we're talking about developing you know code uh so that is very very difficult to do so what do we what are what are some what are the code repositories doing to help with this problem uh well obviously there's linus law and code reviews have multiple eyes looking at code looking for malicious functions malicious testing that's that's kind of very obvious malware scanning yeah you

know it's it's it works i mean it's it's it's got a purpose any it's not the best but what i was really in my research i was really proud to see and excited to see that most repositories have really good ir teams github's response team pipes python instant response team they all have processes to respond to quickly respond to once identified and notified to malicious malicious code so they have quick take down processes and quick uh advisory process and alerting processes to let downstream projects know of the malicious code github for example they list their malicious code notifications as vulnerabilities so you can actually search them as yourself by looking at the advisory database and searching for the terms

malicious package and you'll see you know a lot of different interesting malicious code injections

okay on the other scale other end of the software supply chain i want to talk about uh packages and repositories and attacks on them again these are the the the youngs the apts and the the solutions that distribute your software an attack that's becoming popular is called typo squatting you all have all meant gone to a website and typed the wrong website and saw a page you wish you never saw it's like that but with with with dependencies you you type the wrong dependency or profit package or project and you got maybe code it actually works um but it actually has laced with malicious code or malicious functionality it's a bigger bigger and problem that's

that's happening lately a lot of uh code repository tools are are looking and watching for these types of things but it's still possible and very possible to happen so package scoring when you want to use a package there's a there's a solution called libraries.io source rank and what it does is it helps collect it's another uh scoring framework that looks at scoring metrics uh for for quality and the input for this tool is our package managers uh oh packages from package managers it looks at things like the project the version and its dependencies um your input there is your package and output is like i mentioned uh scoring metrics uh that that could be very useful for

automation and and risk-based decision making so stepping back at a higher level this is for you managers and risk quantifiers out there to wrap up the presentation um quantifying probability is going to be different for each organization and project it really depends on what your organization does uh to really understand the the the potent the probability for this for attacks with the with to your application your project or your organization um if you're in government critical infrastructure financial crypto they seem to be very high targets if your organization or project serves code that that references a nation state critical infrastructure industrial control systems high value asset then your probability is is increased i would suggest um

the good news is uh i i guess the good news is it does take a high amount of expertise and sophistication to pull off a really a good attack i think i feel like there's a lot of while there are a lot of publicly recorded software supply chains attacks out there there's probably a lot more that happen um that's not publicly posted and again it's just more of the nation-state uh you know actors doing their things so impact obviously could be very high uh cyber warfare especially espionage disruption distraction within is is also very possible so the the amount of tax you know there's there's about 900 a year roughly oh and it's it's still it's it is increasing

uh as reference from the the sonotype report

this is a a reference it's a good reference to look at it gives you some more numbers uh actually talks through this whole this whole project the whole problem of open source software supply chains uh and attacking them this is it's it's a year or two old but it's a very good report again has a lot of different statistics and numbers and presented in a really good way i just wanted a reference i thought was very helpful all right so what can we do well uh we continue to analyze our applications and our organizations for their risk and identifying controls where they might be most important i would also consider uh you know within your supply chains

and your ci cds is to utilize s-bomb and sign everything those two things are just a very quick easy and quick wins for your supply chain uh csps have a lot of tooling and processes obviously to make devsecops easier continue maturing your software inventories cis top top 20 or number two is your software asset management top control number two so that uh is really a very important uh consider the dependency in package management scoring utilities and develop minimum acceptable scoring criteria based on organization risk tolerance okay so i mentioned that i went through a journey a learning journey and i learned a lot did i get the answer to my my initial question yeah kind of

can we trust open source code and binaries not to include malicious functions my answer is well kinda it really depends on what you do you know as i mentioned the previous slide you can be an attack a target certainly financial is going to be a continued theme within the the motives of of cyber attackers um the i did learn also the open source it's it's just huge and it's getting older i learned that there's a lot of challenges with the open source community and and it's with getting older comes comes potential security issues the need for visibility scoring to support automated dependency management decisions is is is is getting better it's increasing but our tooling is getting better

but i can i encourage you to um contribute to these tools uh because it's it's it's really going to help us to continue to as an industry can develop really good secure code i i've learned about sbom and how easy and important that is i mentioned earlier in projects that are uh included in many other downstream projects are very very important and the ones that aren't being maintained or ones that are expired are really really concerning and i know there's a lot of work happening with several different groups to help address that problem within the open source community including government assistance financial gain as i mentioned is a a primary motive and seem to be the top software supply

chain attack motive um vulnerabilities as sid mentioned uh i didn't want to talk about vulnerabilities because there's a lot of different talks about vulnerabilities within the open source supply chain but they're still very important because they can be maliciously injected and but they're still also and it can be purposely not corrected as well the integrity of coding the development tooling itself is very very important those supply chains those tools the developer tools are being attacked and then exploited within our software processes our development processes i did i was very like i mentioned very happy to see that open source code repositories have really good controls to identify malicious intent and respond to malicious intent and malicious

injections i also learned that developing a presentation is a great way to learn and help the community at the same time all right so thank you i that is my talk this is uh i have a lot of resources here that you can you can download if you want to take a picture go ahead take a picture now uh of the url and um but i do have a special i do want to say thank you to a few that listed at the bottom there rick yoakum james stingler mike frankie peter genesis and matt kerr for helping with presentation thank you guys much appreciate it guys are awesome questions thank you thank you questions comments thoughts

serum love you too

software control management solution

yeah so so the question several to make sure i get that right where where is the challenge within the software development life cycle is it is it more on like the github in the software control management solutions is it in our ci cd um or is it in the package distributions well i can tell you that the package distributions are you know there's there's attacks that come and go typo squatting namespace confusion those are attacks that they're the rise and i think us as an industry will correct them as long as with software with the the software control management solutions like github um the attacks that i've seen at least with my research fairly even throughout the attack cycle

they're attacking all different points what i've provided is just more just like a high level architecture view of the attack threat the attack types but i i can't say that you know one specific part of the of the cycles being attacked more than the others because a lot of the data and the research i found i read is kind of across the board good question does that answer your question okay

all right everybody thank you very much thank you again ryan great job