The Past, Present, And Future Of Supply Chain Attacks - Edmund Brumaghin and Nick Biasini

all right welcome everybody thanks for sticking it out for the afternoon we're gonna talk to you today about signed sealed and compromised kind of where we are with supply chain attacks kind of where we've been where we are today and where we see it going in the future as far as who I am I'm a security researcher with Talos I got my start as a stock analyst used to work in a 24 by 7 sock for too many years too many hours and I've been an analyst lead analyst manage teams of analysts I've basically done everything in security except be a developer and I'm very very proud of that because no one ever wants to look

at my code ever as far as my research stuff is concerned I do a lot of research and exploit kits I have for a long time I was one of the people behind discovering domain shadowing I've done a lot of stuff with angler back in the day but today I'll talk to you about supply chain let me hand it off to Edmond yeah so my name's Edmond brummagem on you might recognize this slide if you were in my talk earlier on DNS but I'm a security researcher with Cisco sallows I love malware that's that's what I do on a daily basis we're typically neck deep in in different malware campaigns that we're seeing in the wild so I love

ripping them apart and figure out how they work so that we can protect customers like Nick I have a background in security operations so I've held pretty much every role you can hold in a sock before coming over to into the research side ok let's talk a little let's talk a little bit about what supply chain is what it's actually made up of there's a lot of components to supply chain and I'm gonna kind of abstract this to general terms so at its most basic form what you start with is raw material and this could be something you mind something you create whatever that is some sort of raw object right well that object has to get somewhere

that's where suppliers come in they're the ones that move the raw material and put it into manufacturing this is where you have the people that are actually assembling the software the widget the whatever you want to call it the item that is being created now that you actually have a finalized product you then move in to distribution so now you're you have you went from raw materials to getting those raw materials to the manufacturer the manufacturer creates a finalized product that finalized product then goes into a distribution network which then finally gets to your customers so immediately you should see why people attack supply chain there are so many people involved in this process there

are just a lot of avenues for you to go down so let's start by talking about where this all started let's talk about the first ever instance of a supply chain attack that we've seen and it starts with the gunman project for those of you that aren't familiar this is a recently declassified project from the US government and it actually starts with Believe It or Not a typewriter this is the beginning of supply chain attacks all right yes I'm talking about the late 70s early 80s the time of giant cell phones and terrible suits apparently what actually happened is there was a bunch of US embassies inside of Russia they received a tip that someone was intercepting the

communications inside of their embassies inside of Russia what we actually found was the first known interdiction attack and I'll talk a little bit more about what interdiction attacks are but let me talk to you about what was done to these typewriters this is very interesting so basically what happened is these typewriters were being sent all the embassies they had to go through Russian customs when they arrived at Russian customs they were intercepted what they actually did is they took a solid metal bar that was inside of all these typewriters and hollowed it out inside of there they put a full circuit board that had full wireless communications for every key that was typed on the

keyboard and the only way that they found it actually was through x-ray and the only group that found it was a small organization at the time that undertook the task that no one else could fill called the National Security Agency so let's talk about the logistics behind this they had to in secret remove 11 tons of equipment out of US embassies in Russia to figure out how to do this when they began the inspection no one could figure out what was going on there was no visible alterations nothing obviously was done they literally had to x-ray the machines to see it and actually on this slide you can actually see the x-ray as well as

the actual board that they pulled out of the the typewriter itself as far as sophistication is concerned they actually had the ability to turn this tracking on and off remotely they transmitted everything wirelessly and they actually didn't record the type heads they monitored the monitored the electronic frequency of the rotation of the type heads to determine this what did we really find though the Russians created the first key logger that's basically what they did is they created the first key logger so let's talk a little bit about interdiction this is probably the most common form of supply chain that you're gonna see associated with very sophisticated actors in nation-states the reason that is is basically what

they're doing is intercepting these products while they're mid shipment so once they leave the distributor and are heading to the customer they will intercept the packages in route modify them reseal them and continue along to their final destination okay so that is my example of interdiction attack let's talk about clipper chips just show of hands how many people actually remember or know about clipper chips oh it's an okay amount this is also pretty old-school but this is actually one of those clipper tips so the concept was AT&T released a hardware encrypted phone that allowed you to do encrypted communications between devices and the US government decided that they didn't like that so what they did is they came

up with a concept of a clipper chip a clipper chip was supposed to use proper key escrow to allow only authorized users to decrypt communications and as we all know when you create a backdoor an encryption is only used by the one person it's intended for and it's implemented perfectly every time not really and in this case it wasn't at all initially there was a huge debate around this and then I in late 1994 you had the the revolutionary research come out from Matt blaze where he blew apart what the chipper clip was how easy it was to compromise how easy was to decrypt the data and oddly it never went anywhere you never really distributed it it was

never actualized as a product but what this was is kind of the first example of a supply chain and manipulating the physical supply chain of devices now this is a little bit more difficult because this involves you being on the assembly line as the devices and the products are being manufactured let's talk about some hardware supply chain attacks today they don't exist they don't and there's lots of reasons for that if anybody does have examples of wide scale hardware supply chain attack I'd love to hear about them because I've never seen it there's lots of reasons for this you look at things like circuit boards right and then you start taking it even further and start

looking at chips and then you start taking it even further and looking at traces and you realize that there are hundreds of people thousands of people that all their job is all day is to take chips and strip off layers and start looking at where all the wires go and they do this every single time so if you want to do a hardware attack it is going to be extremely difficult and extremely noisy because you can't just compromise one device on an assembly line you have to compromise all the devices on an assembly line this is why there are just so many more advantageous and easier ways to attack supply chain that will not result in you having to compromise

tens or hundreds of thousands of people all right one last thing network supply chain this is a pretty common one as well I'm sure you all remember the target breach it's a big deal will huge credit card data breach started with one of these on a stage back system inside of their their enterprise and people like this who require access to those devices to manage them and they abused this advice or this access right that's how they were able to get in was by leveraging the access that you provided to your HVAC provider so generally the most important thing you need to understand is validating why you are giving people access why do you need access to this device what

specifically do you need access to coming in and installing and allowing someone unfettered access into your environment because they need to manage an HVAC system is a horrible idea additionally and this is something that's becoming more and more important you really no longer can just can turn concern yourself about your own security posture right you have to start pushing your security posture onto your vendors how do you know what they're doing to secure their software how do you know that they actually have a reliable supply chain as a large organization these are the kinds of questions you're really gonna have to start asking those that you want to do business with and then another key thing of course is how

do you monitor that connection right is it fully encrypted do you have a way to decrypt those communications are you gonna analyze what they're accessing and how they're accessing it and then more importantly what type of connection do they need do they need a site-to-site VPN can you provide them with VPN access do they want RDP access because please got no do not give them RDP access unless they have to go through a bunch of Hoops to get there because everybody knows RDP on the Internet is failsafe nothing bad ever happens there so what are the lessons that they learned about this multi-factor authentication is paramount to this if you have external partners you should require them all to

use multi-factor authentication at least then you can try and prevent things like credential reuse or credential theft from really coming and inviting you please isolate IOT devices segmentation is the thing please make use of it it will save you I know it's a pain to implement I know it can be costly but in horrible times it can save you an awful lot I mean nobody really thinks auditing is sexy in security but man is it important you have got to audit the people that have access to this stuff you have some provider that you provision to access to four years ago it still has credentials that isn't even a provider anymore because nobody looked who had access and why and again key

things segment your network do not allow RDP from the Internet I mean this is just good advice for anyone do not allow RDP from the Internet ok so I'm gonna hand it over to Edmund now to really go deep on what we're really seeing more and more of which is the software supply chain excellent yeah so um as we all know adversaries will typically try to work towards mission objectives right no they're typically going to attempt to achieve those mission objectives as cheaply as quickly and as easily as possible so what a lot of attackers have recognized is that as their targets have gotten more mature from a security capability standpoint it's often quicker cheaper and easier to target a third

party that they know the company leverages and leverage that access to gain an initial foothold within an organizational environment right in a lot of cases the soft target isn't necessarily you know hitting up against the firewall of the company itself it's targeting the supplier software or services or things of that nature and then leveraging the access to that environment to hop through to the the actual organization that they're targeting and that's what we're really seeing from from adversaries on in I'm gonna walk through a few examples of some of the types of attacks that we've seen that have had wide-ranging impacts on you know large numbers of organizations around the world this is what a software supply chain attack

actually typically looks like so in a lot of cases what you'll see is the the attackers will compromise the the vendor or the manufacturer of a specific piece of software that they know is being used by the target that they're actually going after and they'll compromise their development environment and use that access to insert malicious code typically you know backdoor code into the software development environment for that company so when they take the code compile it and then ship it to customers it's already bent back toward a lot of cases it may be signed using legitimate code signing certificates right because they've compromised the actual source code itself so then that will typically be distributed to whoever is using the

software from the legitimate and software manufacturer nobody realizing that you know it's been backed or you can check hashes you can compare the hash of the downloaded file - the one you receive from the software vendor they're going to match because you're getting it from the legitimate source they've just been compromised so once that's been distributed into the actual organizations that are being targeted the attackers can then leverage that to initiate you know command and control protocols bot registration with command control servers and then leverage that to begin to operate within the environment they're actually all along trying to target they're simply you know going after the soft target and in this case you know the supply chain so one of

the most destructive cyber attacks in history is not pet yet right I'm sure everyone in the room is pretty familiar with this one it was very very disrupted to a lot of organizations around the world accosted a lot of companies you know a ton of money due to operational disruption and associated impacts on this was an example of a supply chain attack in in action so the way that this worked was the adversaries behind not petia compromised a software development company called emmy dr. so mu dhaka they make tax accounting software that's heavily used in the country of Ukraine so if you do business in Ukraine or if you're located in Ukraine odds are you

have a system in your environment that has a me dock software installed for for accounting purposes um so what they did was actually compromised the software update servers associated with the ami dock application so for victims when you open the software and go check for updates it goes out to any dock servers and pulls down the latest version well in this case they had inserted backdoored copies of software updates to those distribution servers so the next time potential victims went out reached out to update a me doc they actually got that back the word code into the environment and so the attackers actually leverage this sort of access to those distribution servers for a period

of time to distribute this backdoor and gain access to all of these organizations that had any dock software installed in their environments so when the attacker chose to activate non petia they were able to leverage that access to push this malware to all of those environments as we're probably all aware this was a destructive self-propagating a piece of malware that was designed to cause operational impacts so the backdoor itself that was planted in those software updates functioned pretty similar to what you would expect from a backdoor I'm not going to go through all of the specific that were available to it but they're pretty common when you're looking at backdoors and the functionality that they typically provide for adversaries

so this is the the payload this is the ransom note that was associated with not petty looks pretty similar to what you see from Master Boot Record based on our although there's a lot of examples of those out there this is what the the actual ransom screen that was displayed once the malware was activated in was finished with the destructive activities that it was performing on infected systems once it was finished attempting to propagate to other systems you can see how this sort of malware creates kind of a snowball effect you start out with a patient zero in the environment you know that system gets effectively you know just destroyed and then the the malware attempts to propagate throughout

the network more systems become infected that propagation continues and before long you go from one destroyed system to a significant percentage of your overall network architecture so in this particular example the the malware actually had several capabilities that it could leverage for propagation to other hosts within the environment to facilitate that destructive capability that I was talking about they could use WMI they could use PS exec they actually could leverage eternal blue and eternal romance as well so they would scan the subnet of the infected system look for potential candidates for propagation and if found they would propagate to those systems the destructive capabilities would start and they would destroy systems so there was when we started

looking at this there was some interesting elements right you've got a ransom note there's a Bitcoin wallet address there they're basically saying you know pay us Bitcoin and we'll give you your system back but there was several interesting characteristics that you know lead us to believe that this wasn't designed for financial gain this was in financially motivated and that was because there were several destructive operations that would take place that literally could not be reversed by the adversary even if you did pay likewise the the way that they were managing the ransom payouts they just had a single email address no unique identifiers to manage you know who's actually paid the ransom to know who to give any potential keys or

recoverability capability soup second one so when you talk about like the types of companies that are typically targeted in these sort of software supply chain attacks on you know you're not typically gonna see companies like like the ones on the slide targeted because it's it's extremely difficult these are companies that have very very good security postures you're gonna you're gonna see organizations that provide niche software or niche services in a lot of cases these might be related to the medical field or the tech field or the banking industry or aviation in a lot of these industries they use extremely small companies that provide very very niche software products are very very niche services and as a result you know

that they don't typically have the same level of security posture right when you're talking about you know three developers in a garage developing a software application that's used in mission-critical applications you know they're not necessarily gonna have the same emphasis on securing their environment on securing their software development lifecycle on securing their network architecture as you would with you know larger more mature and more established companies and that's really where the adversaries are focusing so another example of this in action was with ccleaner which was a pretty big supply chain attack that had wide-ranging impacts as well so if you're not familiar with ccleaner it was an application that was initially created by a company called pure form

that was subsequently purchased by avast CCleaner is effectively an application that you can use to kind of like centrally perform a bunch of administrative and management tasks on Windows endpoints you can manage installed software install updates you know clear caches and whatnot it provides kind of a single pane of glass to perform all of those you're not going across a lot of different control panel applications so this is a pretty widely used application this is from the ccleaner website just some statistics on the actual install base associated with ccleaner you can see over two hundred or over two billion CCleaner downloads worldwide that's a pretty large number of users that were impacted by this so

it was interesting because when we were taking a look at this we initially found this because of some of the heuristic detection that's that's built into our endpoint security program and we started looking at it we noticed that you know it was being retrieved from the legitimate software download site for ccleaner it was actually signed by the company that's associated with the development of this we're um and in what we were able to identify was that the attackers had actually targeted pure forms software development environment like I was describing earlier and leveraged that access to backdoor a specific version of ccleaner that was in subsequently compiled by the legitimate a application vendor and made available on their

legitimate software download site so you'd go out you'd grab this you could clone once again compare hashes make sure you're getting it from the right location but at the end of the day you were installing a back to her copy of ccleaner so this was a pretty interesting fact were on it featured a kind of a multi-stage c2 connectivity mechanism where the Malheur would connect out to a specific IP address and as a fail back if it was unable to connect to that IP it would use a domain generation algorithm so it would dynamically generate a list of domains to connect to it would perform a dns request to the active DGA domain at the

time for whenever the malware was operating and then whatever was returned by the the DNS response from the DNS server for that domain they would basically take the IPS perform an operation on them and then leverage the results of that operation to define where they would actually go for command and control so as a pretty sophisticated mechanism provided fail back for for the the adversary in cases that the the primary command and control infrastructure was taken down or suspended or was unable to be operational so we reversed engineered the Evette DGA and we were able to generate a list of all of the domains that would be leveraged by the malware for the next 12 months what the the

malware's DGA domain was was set up to do was it would rotate once a month to a new domain based on you know the the DG output or the DGA result and so we actually sinkhole of all of those domains because they hadn't actually been registered by the adversary yet and when the primary command and control server was taken down when we were doing this research we were able to monitor all of the activity the DNS activity associated with the command and control infrastructure on that we had sync hold and you can see how it starts it at one domain and then when it hits the next month it switches over to the to the

next domain in the list but since we had sync hold all those domains proactively on the adversary wasn't able to regain control of the all of the systems that they had backward so the the back door was a stage one of what was ultimately a multi-stage infection process the back door basically gave them capabilities bare minimum set of capabilities possible to operate in the environment so that if they chose to operate further or if this was a target that they were interested in they could deploy additional stages of malware and begin to operate towards some of those longer-term objectives so one of the things that would do is it would grab a list of all of the installed programs

the running processes a ton of information about the the system that was infected when it would be canal to commanding control it would just send all of this information out it's pretty much all the info an adversary would need to stand up a lab environment that completely mirrors the endpoints in an environment they might be interested in targeting and then testing payloads against so make sure that you know they could remain undetected so we were able to gain access to the command and control server database that was associated with this this first stage of that malware infection and what we found was a PHP application with a database back-end that was being used to track

all of the bots that were basically beaconing out to command control all of the infected systems that were operating with this backdoor and what they had done was they had defined an array list of domains on the c2 server and the logic of the the server-side application was such that if a system beaconing to command and control with stage ones backdoor was originating from or was part of any of the domains in this list they would choose to serve a stage two payload so this was the list of companies that this particular adversary was interested in operating against it was a pretty interesting way to do that because at any given time based on changes in tasking let's say they were

interested in high tech companies you know this month but wanted to operate against a different industry or a specific geographic region or you know strategic targets that could simply change this list and in the very next time the systems were beacon in they could drop additional now our payloads and begin to operate towards those mission objectives you'll notice that Cisco comm is actually in this list of domains but since we were able to you know look at the the backend database all of the information about all of the systems that had been - see - we were able to determine that none of our systems had actually been impacted by this but this is a list of a

lot of a lot of high-tech companies and a pretty interesting way to create a dragnet a very large botnet that you could use to immediately begin operating further against strategic targets on an ongoing basis so we did a little bit of analysis of some of the code functions that were available to ccleaner and we actually did some comparisons against other known malware families that have been operational across the threat landscape and we found very very interesting similarities to some existing apt related malware families that have been seen in the wild so this was associated with group 72 if you're not familiar with group 72 it's an apt group you can go online there's a lot of

research that's been done it goes by various code names so you can see a couple examples on the on the slide there but so a pretty interesting example of a software supply chain attack and just how dangerous this sort of approach can be in organizations where they're doing everything right in the retrieval of you know software they're getting it from the right location they're checking hashes they're making sure the code signed it still creates a problem because by targeting you know softer targets you're able to you know subvert a lot of the security controls that organizations may have in place all right now I'm gonna talk about what was the obvious eventualities which is when criminals realized hey there's a

whole bunch of money to make if I attack the supply chain and that's exactly what we've seen this is actually a really interesting one that I'll start with so this was disclosed as a zero-day at the app sac village of DEFCON this most recent DEFCON so like a couple months ago that was in WEP min was the product that it was a zero-day turns out wasn't necessarily a zero-day they were actually hit with the supply chain attack so what happened was is they compromised one of the development servers they modified a script and then time stomped it to make sure that it didn't look like it was modified they found out that the server was modified

reverted a bunch of stuff but since they didn't know that that file was modified it ended up continuing through the process and made its way into a much later version of the code and that was actually what the adversary was taking advantage of it wasn't that they had a zero-day in the code it was that someone added code which in effect is a zero day but this is just one example of what we're seeing we've seen examples like pH compare where you have this hard-coded IP address that it was beaconing out to web extensions so this is a gigantic target just gigantic if you think about average users and their browsers and what extensions have access

to by starting to attack web extensions you can immediately hit a huge amount of systems very very quickly and do simple things like click-fraud with very little difficulty and then what we've seen most recently as just the dam is broken it's it's like every week we're hearing about a different supply chain attack in some industries both large and small a lot of it has been done for crypto mining they've done all kinds of interesting things and then you can't forget about mage cart the thing that's skimming credit cards all over the Internet has been doing it for not six months a year now almost a but let me finish up by a very interesting specific example the event

stream incident so event stream is an open source software program there is a sub product of that called flatmap stream and what happened was is you had a software developer theoretically who started contributing read up regularly to this particular software package and was really moving it forward so the person who actually owned that software package receives an email from this person who's been actively developing it and they said hey I noticed that you don't really have a lot of time you haven't been developing this very much lately would you be willing to let me take it over and as the owner didn't really have time to manage it anymore they're like fine here you can take the

library and become the owner of it and they very quickly modified it to add a whole lot of malicious functionality and then they profited pretty significantly the reason is they were targeting a Bitcoin wallet software called co-pay what they basically did is they modified the code to determine specifically if co-pay was running they then detected the on the mobile platform windows platform whatever and they looked at what the balance was of the wallet and they made a decision on whether or not to dump the wallet based on how much money was in it did a little math the balance had to be more than a hundred Bitcoin or more than a thousand Bitcoin cash so for every one

time this code executed they would steal either four hundred thousand dollars or two hundred and fifty thousand dollars you can start to understand why adversaries are like wait a minute why am I wasting my time trying to compromise just random systems on the internet I should start focusing on specific products and what we've really started to see is open source is becoming a massive target github repositories NPM packages web extensions anywhere where people are writing code and sharing code bad guys are realizing if I play good for a while and help the project a little bit I can get levels of access that I never would have hoped to achieve by compromising a company just

straight away and now we're really seeing is this is going to continue it's going to get worse and you really have to start looking at this and then you have the disaster that is ad networks I don't know if anybody's looked at ad networks but my god is it a disaster there's so many systems so many domains involved and there's so many points where you can infect this process along the way so what are you supposed to do so the one thing you should do before you really start worrying about supply chain attacks is ask yourself do you have all the basics covered because if you don't stop wasting your time if you have RDP open to the internet stop

worrying about supply chain attacks if you have a flat network and you don't patch and you don't have any visibility stop worrying about supply chain attacks because it is the least of your problems at that point but if you are amateur security organization and you do have the bases covered and here's just a list of the things you should have if you don't know the assets in your environment stop asking me about supply chain attacks you don't patch you don't segment you don't understand the users what your file access control you have no user education if you're not even doing threat hunting stop worrying about supply chain attacks there much bigger problems in your environment

to deal with but if you do simple guide install the update on a single system or a handful of systems and wait and wait wait some more hurry up and wait just keep waiting and then start looking for things start figuring out what networks and devices is this connecting to is it connecting some random server in a random country on a random port as if it is that's probably weird and maybe you should look at a little more what type of document data is being sent out are you seeing huge amounts of data leaving your network that could be a red flag really scrutinizing the connections that are coming into your environment realizing what you're doing and

isolating your test clients if you're gonna test software updates in production isolate segments make sure that you're willing to lose those systems you don't want to have this be widely deployed before you understand that it's been backdoor and then again I mentioned this before but you have really got to start considering pushing your security to your vendors your security controls do not only apply to your environment anymore there are years and countless resources spent securing environments but vendors don't have to worry about that right you're you don't validate their best practices what you really need to do is generalized industry has to take some education from government and military backgrounds they've been doing this for a lot longer

than we have they've been worried about this for a lot longer than we have they already know that you have to have a secure environment and don't assume because you have a trust relationship that they are secure not the same thing you really need to validate that they actually have a secure environment ask them what are you doing to validate your code how many people have access to it do you have a closed environment what types of checking are you doing on a continual basis these are the types of things you need to understand from your vendors and then is the final thing you know how big is this really everybody is paying attention it is already

threatened critical infrastructure the United States it continues to affect this organizations and it is a real real concern for everybody if you're not concerned about this and you are part of a mature security organization you really should be because this is where an adversary is gonna come after you if they can't get in the front door they're definitely gonna try going after supply chain alright I think we're just about on time so does anybody have a question or two awesome all right when we get windy up here just about on time no thanks good thanks