BSides Cairo 2019: Selected Short Stories on Supply Chain Attacks - Grace Nolan

all right all right hello um it's a wonderful uh it's wonderful to be here um so this is my talk called nightmares untangled which is a selected short stories on supply chain attacks so i am my name is grace um i work on the detection and response team at google in sydney um i'm actually doing a joint talk later today with chris who is my manager and we'll be talking more about exactly what i do at google so hold tight for that um i i'm from new zealand originally we have a population of four and a half million which i think is roughly the equivalent of one suburb in cairo um so people typically know us from like

things like lord of the rings and having like pretty mountains and stuff like this um i also like to do watercolor painting in my spare time and i also recently organized a small conference called purplecon and this is our page we focused on talks for developers and defenders and this was right uh before a bigger hackathon called kiwicon um in new zealand which is kind of like our small version of defcon sort um alright so people in egypt have had a profound effect on all of society you know from inventions in art science and engineering to single single-handedly causing thousands of years of cat versus dog debates um cat memes were which populate around 55

of the internet wouldn't have been possible without you so thank you for that um i'm very thankful um to be here uh before coming here i did a little bit of research into the security scene here and it was fairly quiet uh but it seems to be like that um there's lots of hackers who come from egypt who are really like killing it in the bug bounty scene which is awesome um and there's also been some like interesting projects that have stemmed out um but it was still like a little bit difficult to find something so i'm really excited that something like besides cairo is happening here it's incredible to share knowledge and meet people who are

really passionate about security here you can learn about different kinds of attacks share ideas for building better defensive systems or perhaps you can gain an appreciation of this field which might be adjacent to one that you're currently in so um going to conferences has been like a really big part of the community for me and that's kind of how i got into security uh the first um conference i went to was kiwicon which i mentioned before um and that really inspired me to change into computer science and then eventually get into security um the world is becoming ever more connected and simultaneously divided technology is moving at a rate which we can like barely keep up with

information security is crucial to keeping people safe going into the future sometimes it can be hard for us to feel like we can are contributing especially if what we're working on is small projects or working locally or studying but what i'd really like to show you today is what you work on does matter and i'm going to share you some stories with you some of these stories may be familiar and some of them are funny some are a little scary each story highlights an important problems and in securing our software supply chain so we're going to go into story number one this is about npm which is a node framework package manager and it tends to favor small often single

purpose libraries and i'm going to talk about one package in particular today i know that there's been a bunch of different kinds of security incidents that have happened with npn but there's one in particular that i'd like to highlight and this is the male parser package [Music] this is used for handling emails but one day a simple little function impaired inside of it and this was called the get cookies function this function was not like the rest because it had some odd functionality a particular interest there was this one line of code so let's take a closer look at this here we can see that it accepts some data delineated by g ghi upon further inspection of surrounding

code which i've admitted from this presentation it was figured out that it was taking a command and some arbitrary code the command specifying what to do with that code um so in this case it might have been like loading in the code executing it or clearing um clearing the buffer and that code could be absolutely anything now who could have put this code in here get cookies as a popular function everybody uses it every day and you might be wondering which company makes get cookies how many millions of dollars do they make per year what's this skyscraper office like who's this ceo you might think it's a metacorp but actually it's just this guy dustin87

so unfortunately dustin is just a stock image it's a github account created using using the stock photo and it built three modules which they released as part of mailparser one of which contained that backdoor now let me just show you what the web pager for mail puzzle looks like so this is mailpazo the library that contains getcookies this page doesn't really look that interesting right we can see that the library is getting around you know 60 000 downloads a week that's quite a lot and there are 200 other libraries that actively depend on this one but the thing is about this library is that it's actually deprecated that this isn't the library that you should be using but how would you know

that this isn't right the right library it doesn't say deprecated anywhere on the page it also has this weird thing called node mailer here what even is that what does it mean but this is the correct library um this is the one that is not deprecated being actively contributed to if you notice they look very similar like how would you know the difference we can see that it's got an order of magnitude more people who download this every week and it has almost two and a half thousand libraries that depend on this one so it's really hard to tell sometimes what it is that you should be using there's also a bit of a plot twist

the back door code was never actually called it was put in to get cookies that nothing ever did anything with it [Music] but why would the why would the hacker just put it in there and never use it um you know maybe it was just to build trust with the project first maybe it's because they wanted to just see if you know maybe people would like keep using it like how many people would continue to download this library with the backdoor in there before it got noticed um maybe it was their first time doing hacking and they were just a little too bashful to like run the back back door in their first time so

in summary the bad code was in line for a few weeks and the potential impact was at around 60 000 downloads a week maybe you'd get more than potentially you know almost 200 000 backdoored machines but it was never actually called and this is kind of roughly what that a diagram of this kind of looks like you have the github account you have npm um going straight into whoever is like machine who's like going to be um building uh building this package or the app that's using it in summary we'd say that the impact of this is pretty small and it's interesting in the sense that maybe if it was put in node mailer it could have had a lot more damage

right like if it was in there for a couple of weeks it could have been almost like a million downloads where people have the back door on the machines um so essentially we got lucky in this sense because the hacker didn't press the big red button that said you know do the hacking please like put the arbitrary code on people's machines all right so we're going to look at story number two so this is for something a little bit more fresh um there's lots of people who have access to change code to linux which is pretty exciting uh it'd be a shame if somebody were to do something to that take advantage of this if you want to relive the 90s you can

talk about linux mint on a forum where you um can go and you can talk about all the latest features and you might know that forum software is notorious for having devastating security bugs unfortunately an attacker also knew this an attacker took over the linux mint forums specifically an account of an administrator very conveniently this was also an account that could upload new versions of linux mint make modifications to the documentation and download an entire copy of the forums in database so useful so helpful to have an account like this there's also this very handy message right here where you can fork and patch the source code and do absolutely anything with it so what the attack did is they put in a

known backdoor called tsunami um tsunami uh connects to an isc an irc channel which you can configure and it will just sit in there and it'll wait for commands usually this is used for denial of service attacks which is where it got the name tsunami from so they uploaded this bad iso and this bad linux version that they created and they put it into every place on the forum where a version of linux mint could be downloaded they really took this open source thing too far not only that the checksums were changed um to match the malicious iso but do people really even check these anyway you know we live in a fast-paced society where we are too busy deciding

which celebrities we don't like anymore and scrolling through our phones to even check checksums but you know what they say check some yourself before you wreck some yourself i'm so sorry so how do we know what happened so we know this information because the attacker who did it confessed their sins to a journalist of all people um it's this was done by a guy called peace who lives in europe and no other affiliations to hacking groups they've also offered private exploit services on secret marketplace websites so he was on um the forums for one hour where he managed to get a few hundred shells from a thousand plus installs so this is a very popular distribution

of linux like to be to have control for one hour and to get that kind of result but what about the forum dump what he did is he sold these on the dark web for around 85 dollars per download um this is this was also added to have i been owned i'm not sure if you're familiar with this website um but this is like a really helpful place to look up um you can put in your email address and it will say what websites um have had their databases dumped that contain that um that contain that email address as well as uh what data was dumped as part of this um you can put in your own email

address in there or you know somebody else's nobody's stopping you from doing that so this had like a bunch of accounts that were added so in summary we had 71 000 accounts that were added how many of these were reused how many of these passwords were reused the linux mint os was backdoored managed to get hundreds of shells within an hour but fortunately it only lasted that long and i think the forums were taken down for a weekend after that to get everything rebuilt um it's only managed linux mint is not managed by that many people so it took a lot of work to recover from this attack so that's kind of what this is a

simplified version of what this supply chain um looks like we have the code that is open source um we have the forums which are distributing um these isos we have the malicious iso and we can see where the attacker attacked this project so was it really a tsunami it was more like a scary wave it was large but very short-lived in this attack next we have story number three and this is with mega which is a platform for storing and sharing content in the cloud and they have a chrome extension and it looks something like this this is a screenshot from the chrome web store and it happens to be made by this guy who is a german guy living in

new zealand my home country uh dunking on him is a cultural tradition including by police um his mansion was raided uh for a few reasons um he was doing some like pretty bad stuff on the internet uh and he's like got a big personality he even um had a movie made about himself claiming that he is the most wanted man online is he the most wanted man online we don't know only the audience can say anyway an attacker got hold of mega's chrome extension what they did is they grabbed the source code again which happened to be on github i'm not actually dunking on github it's just a coincidence that these projects are using github

you can also this can also happen with proprietary code and your you may not hear about it because it's proprietary i mean there's like some rules that are going around like laws that are now trying to make corporations like talk about um times when they do get hacked and stuff like this but i think typically with like open source um projects where also maybe more likely to hear about it so what they did is they hijacked the developers account and then they released a malicious version of the chrome extension and this has uh at the time it had 1.6 million people who had this installed so if we go back to here we can see that it's now got um 1 million

like 800 000. so that's like a lot of people who are have this installed and one of the things that it did is that they specifically looked for passwords from our credentials for these websites um where do they look for the grids it would often look for um sign in box and sign in boxes like this or it would even look in like urls for any traces of like logging in and then in the packets that were being sent it would look for any variables that were called like names that remotely look like they were to do with signing in places they'd just try and grab everything that they could um it would also send

all the interesting stuff that we find to this address which is um hosted in ukraine it contained crypto mining because you know why not always worth putting crypto mining into everything that you can these days that seems to be the hip new thing for attackers to do maybe maybe right now not so much since it's taken a little bit of a plunge but who knows we'll see how that goes it would also try and steal private keys for people's wallets but it did not look for mega credentials itself it actually just didn't care about that site um which makes you think that maybe this was financially motivated um if they're you know have a minor and they're

looking for wallet keys that was a weird sound

so we think that this is probably financially motivated but for this attack to work the user had to agree to new permissions whenever you change the permissions and a chrome extension you have to explicitly say what you are looking at so you know when you click this button it's like add to chrome here and then you get a screen that looks a little bit like this and you're like wow drink water reminder sure really wants a lot of things from you [Music] so really so i really pay to pay attention to these all right so they have a blog post where they talked about this compromise um and they had this line in here saying

that they were investigating uh what happened with this attack and i thought that was really interesting so i was hoping that you know when i clicked next to have a look at the next blog post uh that they published that maybe there would be more information about this [Music] but instead one month later they were adding two-factor authentication so maybe there's a connection here between what happened with their account being compromised uh and also the need to have better authentication so in summary we had 1.6 million affected users where their passwords are maybe stolen um and a clean version was released after four hours which is pretty good like that's really um fast response time

um but you still had a lot of people who are potentially affected by this and also when you're trying to get 1.6 million people to change their password it's a little bit like trying to have an argument with them and trying to convince them that android is better than iphone and and that everybody should just calm down like it's so hard to get everybody to change their password and this is like change every password on every site that you may have logged into while the malicious chrome extension was online um so quite quite a challenge definitely annoying for many people so in summary we would say um that this was like pretty impactful but not the end of the world

so this is the final story that i'm going to talk about today um this is also maybe one of the scariest stories that we've had more recently and this is focusing on not pittier sorry that's a little bit hard to read there so i'm going to start with the initial infection vector for this so um so not patio just to just to summarize this was a um ransomware that uh spread out throughout the world um and it was like pretty devastating and we'll get a little bit more into what this actually did and how it worked and it started with intellect services um which is a company based in ukraine so emme dock is used by

80 of companies in ukraine it's got around four hundred thousand client clients um it is a tax uh service so essentially it's allowing people to file like taxes tax returns and tax claims and stuff like this i believe um and it's specifically made for ukrainian businesses so this app will allow people to send and discuss financial documents between internal departments as well as file them with the ukrainian state tax service so pretty important um pretty important software but also not particularly interesting in a way you know like it's just like maybe not something that really sparks joy but just something that everybody has to use [Music] all right so the bad person whoever that was um they

managed to get credentials into intellect services organization um so maybe they did this by fishing we're not really sure they had a look around at the rest of the org they were scraping creds along the way and trying to get whatever information they could they kind of sat there and observed for a while um they also found the medoc update service so this is a server that whenever they want to um whenever this company wants to update their software and this is how they deploy those updates to all of their clients and they added a backdoor to this accounting software and they pushed this through the legitimate update survey so this server didn't actually use

um a like command and control server in the way that malware usually does and because they had compromised the organization they could just sit and watch the legitimate update survey and what they did is they um would gather information about the computers that the software was installed on um and they would bundle up this information inside of a cookie and then they would send it back to the update server so this is not very suspicious looking at all if you are in a security engineer or a sysadmin on um on a company and you're looking at outgoing traffic if you're looking at it at all um this is not going to really raise alarms if you know it's sending information

back to uh you know where the company that is um you know maintaining the software then that just like doesn't doesn't really raise alarm bells [Music] um so they did this for a while they just collected information on people's computers like just what software was installed like what version of their machine did they have and was it patched and all this kind of stuff and then eventually it sent the malware they sent the bad packet um of information so inside of this it had a few commands inside of their uh back door one of which was um to install this like script or the malicious payload of this malware that they wanted to to send out

and it would install on that machine and it would be given a special name it would run for a little while and then it would attempt to delete so let's have a look at what they actually sent across because this is really interesting so here we have that script and what it does is it will grab any credentials left in memory using like a lightweight uh credential scraper that's sort of similar to me cats um and this is part of the way in which they would try and spread through the rest of the network by being able to get information in fact i used a few different ways to try and spread across a network

including the forbidden nsa hacks which if people are familiar with uh if you've ever done like oscp hackers love eternal blue um thank you nsa for giving us such good exploits um to use um so it would use this external blue and external romance as a way of getting around um it would also use ps exact which is a lightweight telnet replacement that lets you execute processes on other systems complete with full interactivity for a console and it also used windows management instrument or wmi which is used for consolidating the management of devices and applications on a network from a windows computing system so those last two ways of getting around a network are like legitimate

microsoft tools that they have built to help you manage devices on a network so um so this malware used these four different paths to spread across the network and then once it had done that it would overwrite the master boot record or which is like the first uh so that would be the first few segments um of your bootloader essentially and if that was unsuccessful then it would overwrite the first 10 segments of your hard drive interestingly enough it also looked to see if you had kaspersky antivirus installed and if that was then it would also overwrite the first 10 segments of your hard drive it then had a ransom that asked for 300 in bitcoin

but this was a little bit of a trick right they're overriding um these like very crucial components of your computer and they did that before encrypting everything um which is like maybe a little bit suspicious it's made worse by the fact that there was an entire a timer installed so uh after a period of time regardless of what you tried to do the computer would reboot itself which obviously since now it's overwritten like the bootloader and slash or your hard drive uh it would not successfully boot and the only way you'd get your machine back is if you factory reset it there are um [Music] there are other interesting things about this like that there was no command and

control server this malware was just distributed but it had no way of talking back to the original attacker and this is pretty strange because typically with ransomware the attacker wants to be ready and available i believe even some um groups or organizations that are running ransomware campaigns they may even have like call centers and like have really good support because they want you to pay the ransom right um so they'll very helpfully make sure that you can get your computer back again because that's a good way for them to get money so this is a screenshot of what the not pitcher looks like it has a thing here saying like decrypting sectors but this is lies and slander

you could not get your machine back um if you had been infected with not pto um it was made purely for distraction um destruction sorry which is really interesting right as a campaign like it's an interesting choice to make so this malware spread like wildfire [Music] and i'd like to talk a little bit about an example victim um and this is maesk which is a shipping company and there was a really awesome write-up by wired magazine um and the information that i'm telling you kind of it comes from that from the article i really recommend reading it because it's super interesting and goes into a lot more detail so this is a company that's actually

based in copenhagen and it's i think the world's largest shipping conglomerate um they had an office in ukraine which used emmy doc globally they have nearly 80 000 employees they have 574 offices in 130 countries around the globe and they account for around the fifth of the world's shipping so they had around 150 or so domain controllers which [Music] is the way that they you know manage their accounts globally and these all got knocked out by not piercer this spread across their whole organization and took over every single one of their computers except for one they had a domain controller in ghana which just i think one or two weeks prior had been knocked offline by a

power cut and it just hadn't been put back online yet they just hadn't got around to it yet so this was sitting in ghana um and they uh the restoration effort for not peter was happening in england so they had to get the server from ghana um they had a staffer from the uk fly to nigeria where they met another staffer who wasn't able to get like a visa in time um to fly to the uk and they met in the middle handed over the server which was then flown back um to london for the restoration effort which is just incredible you know like they um they were really lucky that they had this uh

but maybe now they're thinking a little bit differently about how they um how they organize their network so while this was happening while their networks were shut down they had ships that couldn't leave because they didn't know where they needed to go they had trucks that were all backing up they had goods that were spoiling and they had to pay back people for goods that were lost and damaged along the way so this um so this cost me ask like a huge amount of money i think it costs them like a hundred and six million or something like this um just from not pta knocking out their networks um globally it's estimated to cost to have cost around 10

billion dollars for organizations where similar things have happened so it took a really long time for companies to resume functioning so the impact of this was huge right like this was really scary for a lot of people it really made us think more about what it what is happening in our software supply chain what do we trust where are these attacks coming from what will the future be like so lots of people were asking a lot of these questions right so i want to talk a little bit about attack surface area to get a general idea of what a software supply chain attack can look like or you know what it might look like to an attacker who is

considering this type of attack so you might start with your computer you're a developer and an organization you install some tools that you use [Music] and they each have libraries that they use and these often reference each other like different libraries are used by lots of different packages and then you have multiple people who are working in your organization that have the same sort of thing but they might have a little bit more variation they might have some other software that's installed that you don't um but the graph kind of look roughly looks the same and this might be your organization and these are different organizations that are talking to each other [Music] and this is kind of similar to maybe

what a like physical supply chain would look like say if you had like pharmaceutical company career company hospital you can see that these are pretty important organizations and each of these are filled with computers that have lots of different stuff installed on them and maybe your project is just sitting right there and one of these people's computers so essentially you are here this is all malware probably [Music] so for an attacker this is like a pretty like exciting thing right you can attack any node in the graph and then get access to other nodes through transited trust and the more machines that you hack the more progress you make the better it feels um you know it's like really

rewarding when you're making such great progress uh owning an entire network um but what can we what can we do about this do we just unplug everything probably not maybe it'll be a little bit more like this [Music] we just don't trust each other but there are some things that we can do we can reduce or eliminate risk from each node in the chain we can increase costs for attackers and we can increase risk and this is like a huge effort right this is something that is more than what one person can achieve but we really want to have a look at each part of these graphs to see what we can do and how it might make a difference

so if we go back over the stories we can have look at examples of ways in which um and these are just simple ways as well fairly simple right like uh of being able to improve um some security that might have like made this uh those attacks difficult um to do so for example with the npm they could have had a code review process where somebody else was checking the code they could have made sure to be using um two-factor authentication where if you are a developer on a project then you have keys that you can use to sign your work and make sure that it's coming from you like a verified developer you can also have a look at monitoring

for your own machines or your network where you look at what outgoing traffic is going so these these things like uh can take a while to to build but these are things that can be pretty helpful in trying to prevent these um that kind of attack then we also have um linux mint where again if uh perhaps if the forum administrator had a security key like one of the physical like yubikeys type thing then that could have been used to verify his login so somebody couldn't do it remotely and they could also have reproducible builds which is a way of compiling a project in a way that every time you compile it you ensure that it's the

resulting binary is the same and so it can be reproduced and making sure that you're trying to verify there isn't malicious code in there so this is like also a complicated task debian has been doing a lot of really interesting work in this space then we have the mega chrome extension again having like um two-factor authentication would be better um also uh signing of um signing of their extensions uh is like a good way to do this so they did um the way that chrome works is that it does auto signing but which is not really ideal because really you want to make sure that the developers who are working on this have like the control um over their

chrome extensions so perhaps that's something that we at google can work on right um the other thing is is to you panic when you see permissions when you see those permissions if it's asking for your first one then just do not let it have control over your computer we're not patio um those uh attacks the nsa eternal blue um if your machines are patched and up to date then it should not be um possible to use those uh however as noted it did use other legitimate ways of spreading through a network uh using microsoft's very own tools to do that so system monitoring is also important and having isolation too so making sure that if one part of your organization

gets taken down then other parts don't go down with it um and things like backups like we talk about backups all the time right but it's still really it's really important that we um take the time to do that i guess you could also try not to have nsa zero days in your stuff i guess but that's pretty hard as well if you've got zero days right so one thing to think about is who uses your software you know maybe you don't feel like people um like your project is that important but maybe people who are using your software might be targets and that might be something to think about um who are your clients and would anybody

want to hack them that kind of can maybe help you think a little bit more about uh maybe or at least inspire you to think a little bit more about like your own security practices we've got to remember that we're all in this together right like this is a really complex environment that's continuing to get ever more complex and we really want to work with each other and making things better um attacking other people is just like it's not the way to go um we want to we really want to support each other in this except at least one of these people in the graph is probably a hacker so thank you very much