Hashing the $#!+ out of firmware

hi everyone so that indeed is the title of my talk we're just trying to get the controls is it working cool so we're gonna go through a few things mostly I'm just gonna get into a talk about me because that's important then we're gonna talk a little bit about firmware it was read a little bit about vulnerabilities and a little bit about hashing yeah I'm very sorry if you're familiar with all of these topics really really well but I figured not everyone would be yeah so cool I'm Brent I work in a kind of research and development role I mostly I was gonna say I like to play with stuff and I like to break

stuff and I like to fix stuff but I feel like now those things really ever happen I kind of poke at things and then sometimes find interesting things I'm not particularly good at these things so please don't think I'm the expert in all of these topics yeah this is what I've found along the way and a lot of stuff that I think is generally interesting yeah so the first thing what is firmware I thought we should ask Google because Google knows everything can you read that because that thing shakes a lot yeah permanent software program did to read-only memory for the for the most part I figured for all of you firmware might be associated with

something like a Rooter yeah this is where at least for most people I feel like they first encounter the term firmware they are they have to perform a firmware update or something along those lines or like most people they never perform a firmware update and they read about it in the news a while later after all the accounts have been hacked or something along those lines now firmware is not exactly something I would call maybe sexy it's not something that appeals to many people I don't know too many people that he's go home and like research firm way a lot yeah the Google trend shows this over time people seem to be less and less interested in

firmware or at least less and less interested in tapping it into Google unfortunately Google Trends is not a great way to tell people's interest it really only tells how many people tapped a word into their search bar but I think this is a lot about the current state of things people are less and less engaged with what they're actually doing on their Reuters and their devices they're not looking up how to install these things they're not looking up how to use firmware and what it is yeah tapping it listen listen to the search bar because a lot of the stuff either just doesn't get done always done for you if you're really lucky so for the most part just

simplify firmware I'm just going to say we needed to make our devices work yeah because for our purposes and the purposes in this talk that's what we really need to know the other thing is that generally it is supplied by the manufacturer unlike an operating system which you must choose to install most firm ways you don't get a choice the manufacturer is going to supply something to you and that's that sometimes the manufacturer is nice and sometimes they actually supply updates but for the most part the manufacturer is actually the problem here they are mostly useless I'm generalizing here if you are a firmware manufacturer please don't hate me most manufacturers are useless somewhere along the line they generally forget

that they made a product and just stop supporting it mostly because they like to sell new stuff okay and I'm sure you've probably all encountered this before how many people here just out of interest have actually updated their route of firmware it's like amazing yeah I mean it's probably the only room I can ask that question and get that many hands up but let's be honest how many of you how many of you updated you root of film eight this year cool we're down to like half yeah now of those hoth that didn't put your hands up but did the previous time was that because there's no firmware available all those who didn't have firmware available again not that

many that's sad but please go update your ruta firmware if you care but basically this is a general problem we have to update these things luckily they might get done for you yeah now in some cases when you go to update your ruta firmware you get a change log yeah a change log basically is just going to tell you what changed in that version of the firmware in this version provided by tp-link they were very very very great they just say it first firmware yeah I feel like that maybe they've their firmware repo is based on git or something and they just riposte the word replace the word commit with firmware because that felt better to

just have something in there I thought it was a bit of a strange thing because I didn't think they'd need to state it was their first firmware as we move on this is the same product line by the way I just picked one at random and took three firmware updates in a row to kind of show you what I found to be the general trend of how these things go yeah this is our first update ok the first one is the first firmware this is our first update improved stability very little off to that there is no description that's all we get ok more info would be Nast they didn't say what they actually improved other than the

stability they did not allude to fixing any bugs ok now often this happens if they weren't called out on a bug they simply don't talk about it they fix it and they move on they say it's more stable yeah then we got into something a little bit better finally this is looking like something we could actually use yeah got quite a lot of info there now what's quite interesting you might have noticed and asked yourself these lands here fixed some bugs and fixed some vulnerabilities and fixed a bug that some function will not take effort or effect off to restore the configuration power doesn't even make sense unfortunately for the most part changelogs are useless yeah be really

great if they told us everything we need to know very seldom well at least in my experience they don't haven't even tell you which vulnerability they fixed yeah you have to go do the reverse search go to CV eat up in your device and hope you find it there yeah that's kind of counterintuitive if you're installing something at least if I'm installing something I'd like to know what they've done and what's actually inside yeah so how do we know when we install something new how do we know what they changed well we could install it and literally see what changed yeah unless you're like you know updating Android or something you're probably not gonna notice big

changes when you do an install and most Reuters they're gonna fix a bug and you'll never see it yeah it was a bug that you probably never encountered because you never went to that settings page yeah the other option hash the file and see if it's actually different yeah I mean if they've released a firmware update it should be different right yeah I've actually found a firmware update once before where I do not believe they did anything the hash was the same as the previous version that incremented the version number yeah now I do agree that this is probably an anomaly and definitely not the way to tell whether your firmware works yeah for the most

part whenever you get a new version of firmware it's just gonna hash to a different hash of the three firmware versions that we looked at the change logs for when you hash them it comes out something like that is everyone here more or less familiar with an md5 hash is there anyone here that really wants me to go through it I'm so glad yeah now for those that really wanted it here's a little slide like they can review it on the video yeah but the most important part is that md5 hashes are reduction okay they reduce your input down to a set output yeah it's a set fixed output limit in terms of characters yeah as a quick example if

I put the word cake and hash that and I input the word cakes you can see we get two completely different hashes yeah I only changed one letter in fact I added one letter but you agree that at least to me those hashes do not look the same yeah if you do see the similarity in those I suggest your outer paper what we do have though is something called fuzzy hashes yeah fuzzy hat hashes are context triggered piecewise hashes yeah these are hash is specifically designed to instead of be completely different when hashing small changes to actually be more similar most of you might have come across this in the form of something like SS deep yeah a way of

performing a similarity hash to see files that are similar but different yeah now a lot of these some hash or fuzzy hashing systems have a way to compare hashes so even though the hashes are different you can compare them and get a percentage similarity yeah my experience that's not always great sometimes it works sometimes it doesn't yeah I took the good old block of text lorem ipsum and I hashed it yeah you can see it there at the top that's great it just gave me a hash we've really got nothing to go on the top one here is the essays deep for lorem ipsum yeah just the first paragraph then I simply took or Epson and put cake at the end

literally the word cake at the end and the only thing that changed in how essays deep if you have a look is the last character now that seems fair yeah we changed one word and we got a one character change everyone agree oh I thought that was fair small change small change on the hash now that's great because it means we can take blocks of texts that are similar compare them and get something different out yeah we get slightly different hashes out but they are comparable now instead if I put cake at the beginning we start to see something slightly different yeah we don't just get a change at the beginning my lands have moved I'm not

sure how anyway who imagined the purple line slightly closer to the text so you can see at the start and then about halfway through our hash we've got changes now this is because the way the context hashing works changes at the beginning make quite a quite a large effect on how the hash changes okay for this reason something like essays deep is not great for hashing a lot of things specifically things like pictures if you wanted to tell if two pictures were similar it is going to do nothing for you for the most part yeah especially if you're using compressed pictures because obviously uncompressed data it's going to be even worse but on good old plain text files works quite

well yeah now similarity hashes unlike cryptographic hashes should not be used for storing passwords if you would like to use them for storing passwords I hope the only reason you are doing so is to prove some kind of bizarre point they do work well on text files and code now when I say code I mean source code yeah something important they do not work too well on compiled data yeah they can but as many of you may expect when you compile something and it gets optimized the change may be quite a lot larger than you might expect yeah now this in terms of the hash itself means that you can get completely different hashes with a very small

change in source code but due to the way it was compiled a very different outcome in terms of the hash yeah that's just thing to think about when you're wanting to use these tools if you want to compare things yeah so as we've said just simply hashing the firmware not a great way to compare it instead what we can do is we can take our firmware hash the actual files that make it up now a great way to do that is to simply pick it of us download its firmware have a look at the changelog and go ahead and pull it apart now what I did just to kind of show this I picked something

that I actually happen to own so that I can actually test it downloaded it took the firmware pull it apart well that went way over to the right yeah and what I did simply did a diff between all the hashes that didn't match yeah simply I took all the cryptographic hashes of files and folders if they matched perfectly from one firmware to the next I removed them yeah I took those as unchanged then took the ASIS deeps and fetched all of the files that are associated with them these are all the files that were added yeah that are newly added to the firmware now in the changelog it actually mentions that they added functionality for D blocking the pin

automatically so it's possible that's what we're seeing Nakula I did not investigate exactly what all those changes were I mostly just wanted to see that it kind of works that you can see what's actually come in you can now take this and go verify whether this is something you actually want on your router before you actually go and install anything you can make sure that all of these things make sense on top of that you can go and make sure that there's no bugs within them yourself if you're you know that England yeah now the next thing is what I'll call firmware evolution most manufacturers are pretty lazy they reuse devices they reuse boards and they use

almost like the equivalent of code Lego blocks when they build these routers and a lot of networking equipment they use very simple bases and they simply add on the features that they require to sell that product yeah and for this reason a lot of devices track back to the exact same base firmware image yeah and when you start hashing up these firmwares and comparing them removing what changes between them you very quickly see what remains the same you can very quickly see the kind of the the beginning of where these firmwares came from and you can see how they gained in complexity as other the price of the equipment or scale sighs port counts and so on increase

yeah now there's not a lot you can do with this information it is mostly just interesting but I think it's really cool to look at what's changing and look at what doesn't yeah now sorry I'm just not sure the slide supposed to be here all right yep so it's a little off but it's fun so in terms of how much something has changed once you've found out that you've got a whole bunch of files that actually have changed from one firmware version to another it is good to look and see how much they've changed this deep and fuzzy hashing allows us to do this if we've got an SSD cache we can compare them between our two firmwares

two files that are different but at least let's say have the same name if we're talking about things that are getting renamed it pushes up the complexity a little bit but we can still use our SSD and look for files that are over a certain percentage of similarity like I said this pushes the complexity up a lot because then we're literally comparing every file to every other file that's great if you're only doing two firmwares when you're wanting to check the entire d-link TRR family that becomes quite a large problem very very quickly so can we verify change logs yes we can we can at least go through things and see what changed is it perfect not

at all but we can at least build tools to get us a lot closer and I mean this was a very quick pass over how we might get there but I think you can all at least take some of this information maybe use it in going forward and verifying what you install before you're going to do so and when all else fails it's always good to just get more data go for more change logs go pull more Fermi's and start comparing all of them now if any of you were here because you wanted to hear about vulnerabilities that's still coming so vulnerabilities obviously you've all propped hopefully you come across this information on vulnerabilities can be

found largely through CVE now a lot of people believe CVE is kind of the be-all and end-all of vulnerabilities for the most part it is simply a naming convention it allows us to give a unique name to a vulnerability and a description that everyone agrees on something that describes that vulnerability and ties it to a name there's an example of one that's all it is if we actually wanna go a little bit further we can look at something like mists nvd the national vulnerability database take CVEs and then actually tires and links those CVEs to products to vendors and to weaknesses so they've actually got a list of known weaknesses and known known vulnerabilities not known CVEs but known

vulnerabilities themselves and they actually put all all of this in a database so that you can search through it I say you can search through it it's not really fun they've not built a very good system if you want to just go dive through it but it is all there and you can download it all so that's quite nice so if we have a look at what we what we've got this is the number of a CVS listed per year as you can see there was a sudden increase now I did not look into why this increase happened there was a I would have said a fairly decent trend upwards and then a lot of

vulnerabilities in a very short period of time yeah now purely looking at this data my assumption would have been that it would have had to do with the speculative execution problems experienced on a lot of processes the timing seems about right ish but some interesting stuff we can see big increases if we then go and just show how much from each of those years how many vulnerabilities were actually linked to firmware that's what you can see in ER it yeah now sure we're not talking half we're not talking like half the vulnerabilities but I mean how many of you own more than one Rooter yeah oh thanks for being a few people to put up their hands yeah

just you know keeping it interesting yeah how many of you own more than one laptop or cell phone yeah a lot more yeah so you wouldn't expect Reuters and firm way to be a huge chunk but they make up a surprisingly large number of vulnerabilities and it's something that we all ba put in a room in a cupboard or something and then literally forget about most people maybe not us but a lot of people we have to talk about the general public here as well a lot of people put these things down and never touch them again so to see any large red bars there is bad yeah so if we explore that a little bit more thanks to nvd we

can actually go and tar some CVEs down to products n vbe is quite cool when you pull the data you can actually go and Link the CVEs through to products that actually were affected by those CVEs and here we can see there are some CVEs that are affecting the 600 different products yeah that's quite bad yeah now obviously that on its own is kind of useless this is the main one yeah was a vulnerability that affected Yamaha devices notice Ruta firmware yeah now personally I've never encountered a Yamaha Rooter but apparently this was bad apparently it affected a lot of Yamaha Reuters and a lot of different in this case there'd be different SKUs of the

router different firm ways that were involved yeah not 600 Reuters 600 versions of that Rueter yeah so that's quite bad if we look at the next one it's a little you know maybe a bit more common than Yamaha Reuters yeah but yeah we've got a kernel bug here we've got drivers this is linked to Dell but as you can see there are a lot of big manufacturers that are affected yeah and manufacturers that are you know as the previous talk said you know a well-known brand you would expect more security from these are not unknown brands yeah this is a list of some of the most affected brands by a number of vulnerabilities that affected them yeah

if we look once again I don't know what you were hearted wrong but Wow I mean Cisco has fewer vulnerabilities in Yamaha so but as you can see these are some big names dealing Cisco yeah HP Intel not small names these are not small funny little companies these are big companies sure they probably release a lot more products which means they're much more prone to just and in a numbers game having a vulnerability found within their product but something to think of at least when you're buying a Rueter you could need you know go on to go on to nvd go find the least useless root and bar that don't go on to take a lot and

just pick the one that's best selling it chances are it's one of those right at the top with tons of vulnerabilities in it yeah so what if we want to go a bit further what if you want to look at unpublished vulnerabilities yeah now this is obviously a bit of an odd area because for the most part you are the fan vulnerabilities you know you're a bug hunter you're doing bag bounties or you know you're making sure your products don't have a vulnerability in them but there's a weird in-between that we can actually play with and that is the fact we've got all this information about previous vulnerabilities and for the most part most manufacturers don't

stop you downloading their firmware so we can go and we can start looking for vulnerabilities now to do this well added is I just downloaded a lot of firmware yeah like about 1.8 terabytes of compressed firmware which unsurprisingly when you uncompressed it was a lot more it it got bigger that's fun but it also hadn't quite full through the complexity when I started this project are very quickly limited to only d-link so that I wouldn't die but what I did I took all of this and started building up a database of hashes yeah much in the same way that I spoke about using it to check change logs I went and hashed the firmware the actual

binary itself yeah recorded what type of packing it's used with or squash FS all of that information then took every fowl after I unpacked it and hashed every file in every folder using both md5 S&P sua's hashes yeah this gave me I won't lie a bit of a big database yeah but it meant that I could very quickly search for things that were similar so I could go through pick a product that I knew was vulnerable and start investigating it nvd was great for this I can go pick a vulnerability and it would tell me all about it in this case we've got an authentication bar pause and it tells me all the information I need to know the

born has something to do with you know category underscore view and folder underscore view the nice thing when you've got a massive database of firmwares their files and their hashes you can simply go look up that look up that vow look up its hash and see where else it was used to find every other firmware that made use of that exact file yeah according to nvd they were ten affected firm ways yeah I found more than thirty seven other affected firm ways making use of the same file yeah now whether all of these firm ways have the exact same authentication bypass I contest unfortunately virtualizing firmware is a fun but tricky on its own yeah

the only way to really test this properly in a real-world situation is to literally go bar all those Reuters yeah this was supposed to be fun not expensive it's marvin of fun but I wasn't quite willing to go bar that many different rooters so unfortunate I couldn't taste everything I could only test one Reuter and it did have the same purpose yeah and that just happened to be because I managed to find somebody that had one yeah now obviously 37 firmware images it's a large number but something to remember here is there are a lot of funny variants especially tp-link and d-link they'll sometimes release an Africa version and EU version and a US version of the same firm way

yeah so saying 37 might be blowing this up a little bit but that's still an a bunch of different firm ways that weren't covered in the CVE or nvd listing yeah so these are things that people aren't looking out for if you're watching the sea V's you know you're very good on your security you watch the sea V's and you make sure your stuff is secure they would never have posted this yeah because it's a variant of the firmware that maybe wasn't heavily adopted possibly they fixed it in a future update a lot of these were old firm ways the change logs don't state that they fixed the bug yeah they just might have brought out a new firmware

but the fact that it's an unlisted bug means people might not notice it and watch a slip bar yeah do we just go back to the same page so is this great not really yeah when I say not really this is a lot of processing yeah we have to do a ton of searching I try to automate this it's where those things where you think I've got this database I can now just tell it to go find a firmware go find a vulnerability found the file return me every route so that's affected and repeat that's a lot there's a very very large search area so unfortunately this is much more of an exploratory project as much as I would like to release a

tool that I could just put online and it would run and it would automatically update the CVE listings and fix all the firmware versions that were left out that's just not going to happen yet yeah when someone gives me a nice big server that much yeah I was working on a GUI that would help visualize all of it showing how firmwares of all from one to the other very sorry that I couldn't get that done by this talk I think it would have been very nice for you to see what files don't change what files do change and which kind of base format of the firmware moves from product to product okay especially with brands like d-link

and tp-link they've got these massive product arrays and very very small little variants of products very often the base of their firm weighs quite similar the next thing fuzzy fuzzy hash searching is slow so for instance if you do find a vulnerability that same vulnerability Mart exist in another foul but due to a tweak in the foul or a slight difference in what is required by that firmware you might not get an md5 hash match yeah simply due to any number of reasons the interface different colors different regions all kinds of things would be really nice as if we could use fuzzy hashes to look for things with a 90% match yeah and expand on it and go and find vulnerabilities

there were even less likely to be found on their own unfortunately that becomes an just simply unrealistic problem so in summary we can find changes we can follow firmware evolution and we can follow the reuse that these manufacturers are doing yeah we can even possibly look up products that had a bad run so a bad line of products and make sure that the next product you or your company are buying isn't based on what is inherently a bad product or an unsupported product yeah you can do this on your own as well I mean you can simply look and see how often they change the product if they refresh it every year and stop releasing

firmware it's a bad product yeah but I think most of all the ability to start getting into finding these understood vulnerabilities gives us quite a platform to go and explore and at least improve upon things get rid of some problems that weren't fixed by the manufacturers so thank you for listening hope that was interesting for everyone yeah any questions cool so you want to know when I was going through all the farts when I was taking the firm we're going through it with I was hashing just the internal of the foul or the bathroom so I was actually hashing the bad stream now that was mostly because I was using it as a running digest for the entire directory

so as I was going through files in the directory I was building up a hash of the directory itself as well but I was using the whole file

who just to repeat that as much as I can so basically saying can I not do both yeah hash you know the inside and the whole foul to kind of try and capture as much data as possible now yes I can in future that's a good thing to do literally just means more data when I started this I was actually doing everything with sha-256 just because I was playing around with different things seeing what I could do and then realize that all I was doing was wasting a stupid amount of space yeah now obviously that doesn't change much but definitely there are a lot of things you can do with changing how the files are hashed think that one winter

first yeah so the question was any tools recommend tool recommendations to extract firmware besides Ben walk the besides Ben walk kind of closes the question for the most part it's been walk you can use the firmware manipulation kit fmk it is not too bad on that note though something to keep in mind if anybody wants to extract firmware and recompile it something I did come across while I was playing with this you can totally take firmware extracted and recompile it but something that sometimes happens people don't you follow the spec Squasher face is actually a spec there's a way it's supposed to be implemented I came across multiple instances of firmware that don't actually use the correct standard

for squash FS so they mess with it Ben walk will happily extract it but if you try put them back together they won't work so if you are planning on messing with things just be very careful because if M Kay will claim that it managed to put it back together and you very possibly might break your hard way but if M Kay even walk top recommendations do you have any okay

the guid reaper sorry to repeat he asked whether using hashes and then treating it much more like using tools similar to how you would analyze a git repo if I could use that in a visualization to that cover it cool so the git repo would thing was actually something I looked into right in the beginning because I thought that network graph of how things change over time and how people commit I thought might be something to look into and for the whole firmware itself it's very good for showing when firm wastes change but what I found is it was very hard to represent how the firm itself changed just it could only mostly show that it had

changed there's a lot that you can do with git though Ballmer is building the same database of your file trees in get you can actually get quite a lot out of it so I think it's something totally worth exploring