Bug Hunting in Firmware and Hardware at Scale: Tools, Tips & Real Vulnerabilities

into the old days of going into a uh you know ripping the hard drive out of a computer uh reading the contents doing a modification and putting it back so that you can gain that persistent access you need the nice thing is ocean comes in really helpful here chip manufacturers uh almost always publish their data sheets uh always the first place to start this along with photos from the fcc's tests database and the fcc publishes all this stuff so this is a chip used in a lot of ip video cameras and it shows how to implement it in a bunch of situations and it shows us a bunch of stuff we're probably interested in looking at here

to get initial access to a device that maybe uses this chip so from a debug standpoint we see okay you are there's support for serial consoles here for on-chip debugging um there's support for spi which is one of the protocols used for persistent storage access that we can access and nand which we can get access to flash on these devices awesome cool probably things we want to target uh also uh the pin outs for the chips are available so we can look at okay you you know the data sheet says that there's a uart console on this chip where on the actual circuit board is it exposed and usually that's published and documented and we can just start looking that up

and reversing the traces on the actual circuit board so pull some examples you know we just we bought a cheap t peeling grouter off amazon just to um just to you know tear down and provide some examples of this stuff so we can learn a lot from just looking at the hardware so all these devices the first thing is in terms of how how we do this systematically and at scale just have a system and methodology for identifying chips correlating to their data sheets and getting that key information out of them so yeah i've built a little database that i keep of all the chips i commonly see so that we can um reference that and

quickly know what kind of i o they have so you know if we look at this circuit board we see okay a mediatek cpu the ram the flash storage which we're going to be really interested in extracting it probably has firmware um always interesting on hardware we see these four pins here sticking up out of the circuit board on the right uh not connected to anything so that provides us a thought of okay so what does connect to that thing you know when was it used was it using development or manufacturing uh very often on circuit boards we'll see um spaces that should have chip that look like they should have a chip attached to

it but they don't again what was the intent there what was it used for conversely we're starting to see more and more and a lot of devices extra chips on there that provide functionality that's not necessarily advertised on devices so as an example in home gateways we're starting to in a home internet gateways um i've been seeing uh more and more bluetooth and zigbee wireless chips in there you're thinking okay you know the manufacturers must be doing some kind of home iot play here they're not advertising it on the box yet what's going on those are all interesting things to look at so i mentioned that four pin unpopulated header um this is the quickest thing you can

look for on every circuit board it's the universal this is going to be a serial console sign if you take away nothing else about hardware security this is probably the easiest thing you can always look for nicely on this one the manufacturer also decided to label the pins on the circuit board to tell us hey there's rx the serial receive tx serial transmit there's a debug console here they labeled it great thanks um so what do we do with that well we can hook up a simple logic analyzer and just look at the um look at the logic levels the binary data coming out of it um serial consoles operate um at standard you know old-school modem speeds uh the

most common of which we see is one on five 200 pod you know analyze those eight bits at a time what's coming out of that serial console awesome we're recovering ascii text okay cool yes there's something here and the hardware we can get access to um there are these little ten dollar devices um that uh interface between uart serial and usb called an ftdi hook it up great we're getting boot logs from the device often also this allows us access to a bootloader where we can do things like set the environment variables that are used for boot so here's an example that i did against a qualcomm chip which was get into the bootloader set the environment variables

um to uh didn't work the first time but init equals bin sh right so the device starts up and it starts a console so that we can get into it we save those environment variables restart awesome that might give us access but what we're starting to see all good things have to come to an end um you know the serial consoles via hardware has been a mainstay of you know initial persistent access to embedded devices for a long time talked about a lot of security conferences manufacturers are rising up to this uh and starting to lock down serial so toward the end once we get to talking to firmware i'm going to give you a demo

of a new open source tool we're releasing to help subvert some of those locks that manufacturers are putting in place so we can actually get access to these things so that works maybe maybe not what do we do next okay well where does that firmware come from we had that picture of the circuit board with our flash memory here during boot up of this device the contents of this flash memory chip needs to be communicated to the cpu so that it gets its operating system and it can access all of the executables it needs to run things like this so we have an opportunity to read potentially modify what's on that chip now this simple type of chip is called

spi flash or spy flash it uses a four pin communication system with essentially a cereal in serial out a clock so in comparison to uart cereal that i talked about earlier that just uses a set baud rate this uses a um a clock that's maintained usually a couple hundred kilohertz clock um and a chip select essentially an enable pin and again look up the data sheet we see a ton of these micronics our nxic spy flash chips across huge range of devices look up the data sheet we can get the pin out of it if we look at what those signals actually look like on real chips while a device is booting so again this is that

tp-link router example we hooked up a logic analyzer to those serial pins and we see all that communication starting so when the device boots in box one here we see this top line that's the enable or chip select pin goes low which low and for a lot of these protocols means active that goes low saying yes i'm going to start talking to this trash chip the clock starts oscillating great then we see serial data coming out in box two that's probably a command for a read then in box three here we see the other serial channels start to go low and see serial data here luckily we don't need to do all this stuff manually

if you have a raspberry pi there's a great open source toolkit called flash rom that you can use you just hook up the gpio pins on your raspberry pi and it can dump all of the data off those spy flash chips there's also some cheap purpose made hacking tools like the good fat um that's super easy to just hook up dump read write the flash on these devices the industry standard for then what do you do with this stuff so you get a big blob of binary data how do you actually analyze it is bin walk um and what bin want does is it's going to walk through that blob of data and look for

strings header information so that we can start to extract what's actually on there and that usually and most of these devices are lining space we actually um we actually get back a uh linux file system this starts to get a little more complicated if you pull apart uh some more advanced recent devices this is one of the not the most recent generation alexa but um a little bit before this um uh like gen two maybe um you'll see chips that look like this uh for their persistent storage um these are harder to work with you have to remove them using an ir oven if you have it or hot air gun um and then you get something that looks

like this it's a ball grid array um the only difference being you need a hundred dollars of tools instead of twenty dollars of tools to interface with this but still possible so we grabbed this chip off and alexa walked through it and got a file system back um for the commenter in the channel um on the topic of the there should be a chip the bloomberg 2018 article um definitely uh you know if anyone hasn't read it the bloomberg's big hack article you know take with it what you will there's still a ton of debate in the industry what was true what was not true but it did bring up the um importance of hardware validation and hardware

security in the manufacturing process so it's a good one too good one to mention uh let us not forget however sometimes we can take the easy route and you know companies like linksys which i'll demo on uh have published all their firmware online so uh you can go download it you don't need to go through this rigmarole uh but let me jump into a demo show you cool we have some firmware what do we actually do with it i'm gonna pop over to terminal here so um cool i have this links this image from a wh w 302 prod it's the first router that they were advertising on their main home page and i downloaded the firmware of it

so we can run bin walk across that to start seeing huh what's in this okay linux kernel arm boot executable that's good to know so we've confirmed this device is linux based arm architecture cool we see a little further down there's some compressed image uh some compressed data here don't quite know what that is this ubi file system i know that bin walk doesn't always do the greatest job extracting uvi so for this one last night i manually carved that out and i'll show you what that looks like um but we can run bin walk across it to tell us what's actually in this weird blob of data um i'm gonna just exit out of there because

taking a little bit the other thing we can do just to see what's in there just run strings right okay what's in this thing is there anything interesting to us do we see you know a hard-coded password something that says private key whatever it is okay you know there's some text in here it doesn't look encrypted i bet we'll you know yeah we're finding actual human readable text in some of these so let me switch over here um i'll show you what when we actually carve out the file system from one of these devices what this looks like um you should all recognize this as a pretty standard linux file system right you have your

binary executables you know etsy directory your libraries here on a lot of embedded devices you have your um embedded web server sale with static content and handlers for your embedded web server cool you can start doing some analysis on these so if we were to do all of this manually where do we start right well we probably want to start looking for hard-coded uh passwords and keys so great pop into the xe directory i looked at this one a little earlier we don't have anything in password or shadow files so there's no hard-coded credential on it um but we do have um i think under the search directory okay server.pen that's interesting looking let's pop that open what do we see

um okay hard coded private key i'm sorry to whoever's private key is now on a conference talk you should probably rotate your key um but that's the type of stuff that we're gonna that we're gonna see a ton in um across devices right um i noticed on this device i have never seen so many shell scripts control every aspect of how this device works from how it determines who's connected to its uh wi-fi network how it does in the security stuff firewall everything this makes me think there are a lot of opportunities for command injection right so we can start looking through and saying okay what are all the things here how are they controlled where the

inputs come from looking for opportunities there right let's pop over to libraries um when we're in libraries here we see there are a lot of so like lib lib d bus um libyu c lib c um there are a lot of very standard linux libraries compiled into this binary so we can start going through and looking at saying okay what version of the binaries are in there uh and what are the known cvs against it um as you're probably getting the sense from everything i'm talking to you know there's a pretty standard process that you can follow with going through these um it's slow and that's the issue that i constantly run into is you know how do you you can't

necessarily spend 80 hours digging through a file system and loading up binaries so we're going to talk a little bit about program analysis and automation and how we can get beyond that that is not the right thing there we go so that's manual firmware analysis if you haven't downloaded some firmware used bin walk give it a shot uh to and start poking through it and see what you find so let's talk about programming program analysis to do vulnerability to discovery so the good news if you can call it that is in embedded devices um the vulnerabilities and problems are those of yesterday in other in other uh security domains it's things like stack buffer overflows

uh command injection and it's really rare to have any sort of um aslr or um binary exploit mitigations that are common uh for computers uh it's it's fairly uncommon to see them against against embedded devices we see constant buffer sizes unchecked balance tons of opportunities so what can we start looking for if we were to automate this process well we can look for vulnerable c functions things like um stir copy printf system calls mem copy things that may take an input from one place and execute it or write it to memory without checking right so a really common example of that is a stack buffer overflow where we take something in uh from whatever source it may be

and let's say we do a string copy to copy that input to a buffer but we don't check the size of it and make sure that um that it fits within the allocated buffer well on these devices because there are no binary exploit mitigations we might be able to overwrite let's say the return address and as an attacker we can insert uh an input that will change that will return us when the function returns to a different uh address to potentially execute code that we've loaded onto that device so cool we want to start finding things like that in in um in our firmware and we have to do this because of the billions of devices coming online

and just the millions of unfilled cyber security positions we have to start figuring out how we automate it so i looked at some at 175 devices that me and my team had done over the last couple years uh 225 vulnerability findings to give you a sense on those hardware and firmware vulnerabilities we found 234 issues related to serial console some devices have more than one sometimes it provides you access to more than one thing 70 devices that give us boot loader access 42 that gave us root access to um via serial consoles without any major major work uh 52 devices with command injection usually in the web ui of the um of the automated uh web ui of the

embedded devices 56 with buffer overflows cool awesome interesting stats but that took like three years or something to do right that's a lot of stuff i want to do more of it so how do people approach automation to this stuff today well a lot of people do source code analysis um that works really well in some domains uh it works less well in embedded devices and um firmware due to a couple issues one we usually don't have the firmware right we don't have the source code apologies um and uh we wanna still do security analysis against it but we can get the firmware from the vendor's website off the chips right um the other issue is that

um when you compile firmware for a device so i'm gonna pick on this router oh i just dropped my demo okay when we compile um the firmware for this router um the um there are a whole ton of things that get compiled in from the chip vendor like for example if you use a standard third-party wi-fi chip um you don't have access to their source code for the firmware um and we need to be able to analyze that the vendor doesn't give it to us so we have to analyze from the firmware angle the other thing is uh we want to start doing this before these devices get into the field and a lot of the current techniques you

know if you load up a firmware and ida and you get a cultural graph that looks like this no amount of monster energy lets you do that um so good question in the uh discord about to exploit most of these hardware vulnerabilities you need physical access so yes to get things like a serial console or to utilize j10 debugging on devices you need physical access um however most of the time you're using that access that you obtain on one device to you know get access to the firmware leave a persistent network connection open so that it's sort of in a new what i find is that it's sort of the initial entry point to allow you to gain a ton of

information to do future exploits so buy one device exploit many in the future all right so how do we leverage uh automated analysis against firmware so there are some open source tools firm flaw and others um i kind of ran through real quick we all know how to poke through a linux file systems and find maybe a password and a shadow file uh what i found is my team started scripting a lot of those things against file systems and we ended up i'll show you what we did to kind of toss all our scripts in one place uh to automate this stuff um we created a little pipeline that we run firmware through to extract it

and extract all the weird file system formats to analyze it and produce some vulnerability results um at uh echo party in this past year we did a little 20 devices in 45 seconds talk about uh against a bunch of these types of devices and here's the kind of things that you can find in automated program analysis so we're going to take a tender ac 1900 it's a router made by tenda and my favorite feature of it is anti-hacking which you know as a security professional always makes you want to take a look at it um so i mentioned you know things we want to look for are uh printf vulnerability system calls but we find a bunch of system calls

in this and we found a few that were exploitable uh from the web ui so input from a user on the web ui is ex exploitable for command injection uh there's a cv on it three years ago still unpatched and the most recent firmware and we can automatically show that it's vulnerable um this code will look a little weird because it's um recovered for it's not actually source code it's a representation it's uh medium level il recovered from the actual binary but we can see so v5 here is our uh vulnerable um vulnerable value that ends up getting passed to system unsanitized and uh we got a command injection i mentioned printf as another really

common type of vulnerability this is another tender router that has another printf vulnerability that we can again automatically using program analysis um get uh be able to understand that that's vulnerable um and we see the vulnerable input being passed to printf here for command injection and i'm going to show you in one second how we do this on an automated scale but again against tenda my favorite company to pick on um let's talk stack buffer overflows in their http daemon there's a string copy in the ssid field that you can actually exploit just in proxy which i'm sure many of you have used between a user and the web ui on a device so i'm just going to show you and if any

of you are interested i'm happy to the point of this talk isn't to dig deep on this tool but um just show you a little demo of it so what my team did is we took all those scripts uh things that we had written over the years python scripts etc to pull apart firmware uh put it in a web ui so that we can start so this is for example that linksys firmware that i looked through um and we can find you know okay all of those libraries in the slash lib directory all right we know so lib grill 5.3.0 cool that's vulnerable to a whole bunch of cves um the um you know printf vulnerabilities

in and where those are called so this is something that um that we've been working on knowing that you know when you start doing this stuff you want to scale it up start scripting it small and uh put it all together into something that can start giving you these vulnerabilities i'll show at the end some of the stuff we've open sourced related to this um and i will uh show you a couple other techniques we can use

cool so again you know on the firmware side script script things to any measure you can python scripts are great for looking through file systems comparing strings things like that but what about hardware right how do we scale up hardware well there are some really good tools out there already that we can use in a standardized way like j tagulator finding the pin out on jtag in an automated fashion um jtag and um there's a serial enum to look at when you maybe have 20 pins exposed on a circuit board that you think might have debug functionality which ones you can use and that that helps replace the tedious slow process with a logic analyzer and microscope of figuring out

what's connected to what but um as i mentioned earlier one of my issues that i wanted to solve is how do we how do we destroy fewer devices while we're doing hardware testing um and i talked earlier about these serial consoles on devices you know that is i'd say 50 of the time the initial access that we can get to a device uh to be able to do analysis against it but manufacturers are starting to lock those down so how do we get around those uh the way those are being locked down so there's a really effective way to do it called glitching attack the basic premise so we talked about the fact that um on the hardware there's a flash

memory chip with persistent storage and that um the firmware that's on that flagship needs to get to the cpu during boot so that the whole boot process can actually happen however what we've found is that if you disrupt the communications between those at the right time the majority of devices fail into an open failure state where you can actually interact with it and do some interesting stuff with it um now the problem with this attack that we were finding is we were doing it by touching wires to pins on chips and this was just not going great it was destroying a lot of circuit boards um so we paid the four dollars to get a cool logo

of something on fiverr totally worth it um and we're going to open source a new tool that i'm about to demo to you called that we called slash bash which is automated glitching to get to a root shell on embedded devices when these have um when these have been locked down so um the way this works so this is sorry this photo's blurry but i'll show it to you in the real world in a sec um when a device is booting we can look for certain serial output to in a very targeted way do our glitching attack to get to force the device at the right time to fail open and we do that by this is a

logic anal analyzer trace you know you see the part of this uh serial of this boot log that says verifying checksum if we look at that on a logic analyzer trace we see that when the device uh the top line here is that serial output the other lines here are the communication lines between the flash chip and the cpu we see that after it says verifying checksum about three characters after that um it starts sending the firmware from the uh from from the flash memory chip to the cpu that is exactly the time that we want to initiate our attack so let's get really targeted about this i'm gonna try and do a live demo we'll see how it works

because live demos with hardware are always fun um so what i have in front of me i will try and show all of this on a so let me switch to terminal so what i have in front of me again so this is a the circuit board from a d-link router it's just a cheap standard device the serial console on it is again like i mentioned at the beginning it's these it's this four pin header at the side here that i have connected to a serial to usb adapter called an ftdi um but it didn't give us any interaction when the device booted up and uh we wanted to get into this device and interact with it um

i have a little clip on the back that's clipped to the serial output pen of the flash memory chip this is the pin over which all the data for the firmware is going to come on the other side of it all this is connected to a raspberry pi um with a little hat on it just for the wiring and but that's what we're ssh'd into right now that you see on terminal so i'm going to hook this up and i hope when i dropped it earlier and i said oh crap i dropped my demo that nothing broke so we're going to hook all this up and we find the power cord for the router so let me uh i'm going to open minicom

because it's just opening our um serial view for this device awesome i'm gonna plug in this router and start it up and let's see what we see cool okay we see some serial output awesome we see that verifying checksum thing that i talked about earlier um so now this is when the firmware is getting sent from that flash ship to the device um all that's gonna happen i'll let that happen and then uh kill it turn it off um you know we see some other information that if we were to let this boot fully um would be useful to us but that's fine i just killed the power on the device just to show you gives us some serial

output but we can't um we cannot uh communicate with it so i'm going to open up a tool that we built that again is going to be that we're open sourcing with this flash bash project and i'm going to run from this tool that we built so when i run it first thing it asks me is what serial baud rate am i doing am i you it does this device use um these are a couple of standard ones this device uses 38 400 awesome okay um it asks for a serial device descriptor this is just on what port is my serial device while i know this is on dev tty usb0 awesome and then it says what phrase

would you like to glitch on so we know based on that logic capture i showed you in the presentation that the serial data transmission starts at um when we see checksum right so i'm going to say find the word checksum when that happens start sending electrical pulses to the serial output pin of that flash memory so i say check some there what phrase would we like to see switching on well so i happen to know on this device it's going to drop me to a real tech console so i'm going to say real we can also do this based on time in glitch for 5-10 seconds whatever we want but i know that that's going to happen so um

it's ready to go so what the device is doing now is sitting in a ready state watching what's happening on devty usb 0 looking for the characters the ascii characters check some over serial at which point it's going to start sending pulses over the uh and grounding the um the serial output data pin so let's see what happens here and i hope it actually works okay cool we see it found checksum initiated glitching so we're repeatedly grounding and moving to 3.3 volts the um the serial output pin and the flash right now glitched cool that means we found the statement real awesome i'm going to exit out of this program and go back into minicom

and hey now i'm interrupt i can interact with this device over serial i hit the question mark this looks like i am in the bootloaders um configuration menu which otherwise i never would have been able to access because this is locked down in the default state on the device um from here i can set those environment variables read and write memory do things like that that i otherwise would not be able to do um so and that is one way so using tools like this and there are a lot of good things out there and we need a lot more like them using tools like this that can really accurately uh control the uh state of

hardware and not destroy our devices but give us the security researchers a leg up to automate our process to get into them these are what are ultimately going to make us successful here doing this stuff at scale so um i know i'm at about a minute till um we wrap up for questions um so you know what can we all do about this dumpster fire well for the market pressure help companies push their security processes into their r d um keep doing what we're doing today um build more tools open source more tools um and if you're the ones building these things always think hardware first um on security really starting with chip selection so

you know back to those t-con the key concepts from the beginning we talked about large attack surfaces but they are manageable with the right tools and analysis but you have to get it right from the start those hardware constraints are going to limit our capabilities um but they provide us opportunities to push the chip vendors to say hey every device needs secure boot needs a secure round call for the encryption keys um and yeah this is going to get worse but some vendors are starting to get with the programs and with better tools to understand and test the stuff we can do it right um i'll plug some open source resources again these these will be available but also

some open source hardware tools that i'll update this link once we put this flash tool on github next week um and some firmware tools uh related to our automation and also our hardware hacking 101 series that um and my team wrote on how to get started with this stuff and walk you through some tutorials um if you're interested in in trying out the firmware automation stuff i mentioned feel free to ping me um and that's about it i'll be on discord today uh shoot me an email twitter etc so all right all jeff thank you so very much for uh for your time and your and your in your knowledge uh one quick question what's the uh mountain behind you i'm a

ski instructor so i was noticing the picture behind you and i'm like okay we're not too far away maybe yeah no no that's that's new hampshire where i where i would probably rather be right now um so that is uh lonesome lake in lincoln and the background is i guess the uh franconia ridge so okay all right yeah i'm gonna i'm gonna ski club right in the area there uh uh north woodstock there all right well again thank you hey hey everybody uh jeff jeff's uh presentation is being recorded uh give the folks from uh besides boston a couple of weeks they're getting everything out online uh the slides and and the recordings uh jeff you have a chance

to decompress now we thank you and everybody else uh enjoy enjoy the rest of the day and hey you know this craziness will end at one point