MQTT: Tiny Protocol, Big Vulnerabilities

[Music] all right thanks folks thank you for joining me in this first virtual besides i'm super excited this is my first time speaking actually at b-sides i've volunteered for a few years and i've talked to a bunch of the con organizers because they helped me out with defend con so i'm super excited to be speaking to you today on mqtt security tiny protocol with big vulnerabilities all right well so let's get started oh maybe let's get started slide slides virtual there we go there we go there now we can get started a little bit about me i am a principal security engineer in iot which is why i'm talking to you today about mqtt i've worked in security for about 20

years now um i've worked in iot for about 45 days and the reason i bring that up is there are probably people sitting in the audience right now who may be more junior to the industry or maybe they're starting in a different niche of the industry like me and maybe you've thought i'm too new i don't have enough to speak because everybody already knows what i know well about two months ago i didn't know what i'm about to present on today it wasn't part of my job and so i've been spending the last few months getting really familiar with iot and applying my background in security to that new space so the reason i bring that up is so that

i hope folks in the audience who are thinking that you don't have enough knowledge to speak or you don't you know you don't want to put yourself out there encourage you to join the community and help folks that maybe don't know what you already know right now uh some of you who know me probably know me from defence we are in a break year uh which actually kind of worked out uh because kovid messed everybody's plans up so kudos to the con organizers at b-sides for pulling off this virtual event it's amazing we are hoping to bring defence back in 2021 as a virtual event but we're still ironing out all the details because it is a lot of work

i'm a us air force veteran i spent some time with the army i've done counterintelligence missile defense uh i have done basically a ton of different things in the military and federal space and i my hobbies are open source information collection so you'll see some of my talks on disinformation is more traditionally what i speak on i also do photography and i raise a small child which takes an awful lot of my time so i have a full plate and i'm i'm happy to be able to be here to do this on a weekend while my husband is guarding my child so that's very always very exciting all right so the first thing we need to

figure out is what exactly is the internet of things now there's a bunch of different definitions and i think they overlap a bit and it causes some confusion basically the most common definition is that it is a set of physical things with embedded software or sensors that's then connected to the internet to talk to other things so the most common explanation of this is like your home automation system so your smart lights your smart doors that's where most people kind of get exposed to internet of things but what i think a lot of people don't realize is that it's actually much bigger than that so you have a huge medical industry that has an internet of

things presence manufacturing power grid nuclear power plants anything you can think of there's now a military internet of things so there's a ton of applications of iot uh sorry guys this isn't good for me so it's a little wonky uh there's a ton of iot implications that span more than just whether i can turn on and off your lights at home this technology reached 100 billion dollars in market revenue in 2017 and it's expected to grow even more than that in the future so what is mqtt mq telemetry transport or message queuing telemetry transport is an open standard basically designed to be lightweight publish and subscribe it's a way of getting iot devices to talk to other iot devices

and the reason it was invented is that iot unlike traditional computing environments often operates in areas of high latency low connectivity where there's not a lot of power and there's not a lot of uh ability to get messages to other things because you never know what's really going to be online or not it's designed for devices with really small footprints right really small code footprints really low power these things can exist out in the environment for five to ten years on a battery and they may never see the inside of an actual physical place they are put out in a farm somewhere and forgotten about so what are the benefits of mqtt uh it's scalable you can connect a bunch of

devices to it and that's really big because when you're dealing in iot you're usually dealing in hundreds of thousands if not millions of devices at the same time it's also packet agnostic and what that means is that it's basically just a sender think of it like a mail envelope i can put anything i want inside the envelope as long as i have the correct addressing on the outside i know it's going to get to my destination and that can be text messaging that can be binary data program data it can be really anything which is good but it can also be really bad it's also really reliable so unlike more traditional low footprint protocols like udp which

is fire and forget and you may not know if your end destination gets the message mqtt actually has reliability built in through quality of service levels so you can actually set it so that you know the end destination will eventually get your message it's also decoupled in design and what that means is that sometimes the server's offline sometimes the end device is offline they don't have to be online at the same time which is really important when you're thinking about these geographically distributed devices in areas that may not have great connectivity it's also really really efficient it's very very lightweight it's designed to operate in networks that have very very low bandwidth and it also is

designed to operate in areas where it has high latency so there's some key ideas you need to understand about mqtt and the first is a publisher so publisher is exactly what it sounds like it's somebody who publishes a message usually this is the actual thing right the actual thing in the environment like the light bulb or the sensor that actually is sending information to some kind of controller that's pushed through what's called a broker which acts as a middleman it sits in the middle of if any of you guys have done the network plus exam you remember the star network design very very similar idea in this broker sits in the middle it's accepting published messages from end devices

now on the other side of that transaction is a subscriber so you can think of this like if you have ring or nest you have an app on your phone it is the subscriber to messages about your light bulbs or your garage door um basically it's the thing that controls the messages in the in the infrastructure and then there's something called topics and topics you can think of like sections of a newspaper imagine if you could subscribe to the new york times but only the sports section or only the finance section that's what you can do with mqtt only subscribe to the information that you want to know about and that way you can act on that

information without getting spammed with a bunch of other information you don't really care about so it's history it's created way back in 1999 by engineers from ibm and eurotech it was created for oil pipelines so again that's a very traditional use case right you've got these oil rigs out in the middle of basically nowhere they often operate with satellite connections which are incredibly lossy and bandwidth constrained so there had to be a way of getting that information back to the controller that was usually in a mainland installation so they created this protocol this super lightweight but the problem is just like when http was first invented they really didn't think about security because that just wasn't a concern for

them at the time so authentication wasn't even really thought of until very recently in version 3.1 so this is basically what it looks like it's a very very simple idea so this is an ir sensor uh you could use this i have one that monitors the water level in my cat's water dish because it's in our back room and i'm really bad about remembering to check it so now i have an ir sensor that sends me an alert when it gets to a certain level so that my cat doesn't die of dehydration because i am a terrible human and that would probably happen um in the middle is a broker and the broker like i said acts as a

middleman think of it as like a little male person sitting in the middle of the network directing all the traffic um below it you'll see a database this isn't required for mqtt but in any large mqtt installation you'll probably see one this allows the broker to store and buffer a lot a lot more messages than it would if you were relying on the on device broker and then on the far right you'll see a smart phone this can be a smartphone it can be a control unit it can be an automotive control unit head um it can be really anything that has the automation programming to understand the message and act on it and you'll notice that

the ir sensor is a publish under sensors and distance so it's in a group of topics called sensors underneath that distance and what that is is if you think of it kind of like your folder structure in a in a file system on an operating system you have a big group a smaller group an even smaller group and that way it's easy to categorize and understand information so use cases where is mqtt used honestly it's used just about everywhere iot is used um you can think of it in automotive instances uh manufacturing instances power grid healthcare cardiac monitors if you can think of an iot instance odds are mqtt is probably hanging out there which is great because it allows this

protocol to be very standard but it also is bad when you start to think of some of the vulnerabilities that exist within it all right so let's talk about some of the types of attacks we can think on and this is where you know i as i said i'm a little new to iot but the good news is for those of us who came up you know older than we care to admit security doesn't really change that much it just is a different application of the same old principles so when we start to think of mqtt we look at denial of service attacks we look at poorer authentication we look at poor authorization we look at software

vulnerabilities mqtt the broker is just an application like any other application transport security how do we actually protect the communication across the wire because remember that mqtt was not designed to be secure it has no native encryption in the application so all of the encryption has to be on the wire so deny all of service um basically i can do this by seizing all the available broker connections so basically just spamming the broker and consuming its ability to respond to authorized clients i can also spoof client ids to bump legitimate client ids off so one of the recommendations is that you have unique client identifiers which is great except if i can get your client

identifier and get to the broker first it'll actually bump an authorized client off in favor of my request you can also do multiple unauthorized publish and subscribe so this assumes that you have some low-level credentials on the broker but you just keep trying to subscribe to topics that you're not authorized to and eventually the broker falls over and then there's malform packets uh this is again a very typical application security attack if you just start throwing bad packets at it eventually the broker crokes falls over and then because it's a star type network if you make the broker fall over everybody else falls over too poor authentication so again like i said by default there is

no authentication they didn't add support for authentication until version 3.1 uh version 5 is out now but it's still not required you can still when you install it you don't have to put any kind of authentication and the bad thing is that even if you do go the extra step of deploying authentication it's sent in the clear there's no native support for encryption when people deploy these think that they're deploying them to multiple millions of devices so doing individual user credentials can be really really hard so what happens a lot of times is that usernames and client ids are sourced from really easily guessable information like serial numbers or mac addresses or things that are

very um sequential so user one user two user three things like that um you could also have a lot of default device credentials so even when people are trying to do the right thing with authentication and using certificates a lot of times they'll just use the same certificate for every single device on the network which if you're sitting in the middle of the network or you're able to gain access to the hardware you can then grab that certificate and become an authorized user all right so we've talked a lot about authentication now let's talk about authorization i think if anybody's worked in an enterprise environment you know that permissions are hard permissions are a lot of work and especially on a

network of billions of devices with hundreds of topics it takes work to get really really good permissioning and unfortunately like in our traditional compute environments that doesn't always happen so a lot of times what happens is that they'll just allow any authorized person any i'm sorry any authenticated person to just subscribe to all the topics so that means if i can get a very low level let's say i take over a light bulb but i'm in an industrial setting i can maybe get topics on the manufacturing robots or i can get information on you know the power conduits for things that i'm not really supposed to be able to have as a light bulb um it also has a lot of support for wild

cards um by the way that's my cat sorry she's fine she's not dying uh she just acts like it so wild cards are plus and hash so plush plus you can figure out if i don't know the root of the folder i put a plus hash is if i don't know the subdirectories so the reason it was designed like this is again it's supposed to be really easy to use which makes it really easy to hack unfortunately so the other problem we have is poor device segregation when i'm deploying something in an iot environment a lot of these networks are legacy they came from the idea of industrial iot spaces where your network was protected

by the fact that it really wasn't connected to the world wide web it might have been connected to an internal network but you assume that anyone who could get into the building probably had off access to be there then we talked about software vulnerabilities so the mqtt broker doesn't have a ton of discovered vulnerabilities right now i expect that as it becomes more widely known and more widely adopted there'll be more cves published on it but the vulnerabilities that do exist are actually quite severe things like being able to gain all kinds of credentials or information disclosure so anytime you have an application that's record that's functioning like this you have the possibility that that application itself

could be taken out uh you also have vulnerabilities in data aggregation and facilitation dashboards so if you use like i said a home automation solution or a home router a lot of times they'll have an admin console that aggregates this data for the user and that admin console may not actually have good authentication and authorization so you can log into the dashboard and then publish and subscribe to mqtt messages and topics so transport security we talked a little bit about this traditional mqtt environments are clear text over the wire so you really have to depend on the transport layer security which is hard in a small footprint device because encryption is costly especially modern encryption

suites like tls they are very expensive to run over the wire so what happens is then you get these lightweight versions of cryptographic algorithms and some of them are really really great some of them are less great uh some of them by lightweight they actually just mean really cryptographically and secure uh one of the problems you run into especially when you're using the device itself to generate a certificate is that because it's low power it often lacks sufficient entropy to create truly random numbers so the certificates become easily guessable um basically just doing crypto without a lot of overhead compute is really hard to get right so where's the problem who cares if i can publish and subscribe to mqtt

we assume that people are doing the right thing and you know at least doing the basics of security for those of you chuckling in the background who've been doing security for you know longer than 15 minutes you'll know that assuming that users are doing the right thing is probably a bad bet every time so we go over to a fun tool of hackers in iot everywhere showdam and we can search for iot devices that are listening on port 1883 and that's the default port for mqtt brokers who operate without ssl we can also see if we can attempt to connect to sysbroker which is basically the default route of most mqtt brokers and if i can connect to that i can

pretty much be assured that i can connect to everything else so when i did a scan of mqtt on showdown i came up with roughly 38 000 results these are mqtt brokers across the world that have absolutely no authentication whatsoever we'll let you publish and subscribe to topics from anywhere on the internet no questions asked no passwords required um when you think about it in terms of total devices that number's not really that big but when you start to dig into these you'll see that these are healthcare instances and power grids and you know water dams and things that could potentially have very negative consequences if you were able to publish and subscribe to messages that you weren't supposed to

um so now that i know this i was kind of like well okay i've got these 40 000 devices roughly well what do i do with it uh i'm too cheap to pay for a showdown subscription i think it's a fantastic tool um but it is quite pricey if you're gonna be running a lot of queries so i found this tool called mass scan and i am super in love with it it's like nmap on steroids and it's legitimately designed to scan the whole internet um it's published by robert davidgram you can get it at that github link it also works with common in-map syntax so if you have used nmap in the past this will be

a really easy learning curve for you this is where i get to do my psa of the talk just because you can scan the whole internet don't scan the whole internet uh for folks that are starting out um this can be really tempting and even folks that have been doing this a long time remember that you should really only be scanning things that you have access to scan hopefully with a written pen test agreement and a clearly scoped agreement on what you can and can't touch there are parts of the internet that react extremely poorly to being scanned uh in 2016 there was a whole kerfuffle about people were trying to hack one of the

state's election boards because they got an nmap scan maybe they were trying to hack it maybe it was a board script kitty i don't know and neither do they but nobody wants suits at your door asking why you tried to scan the mqtt broker for uh the power grid in alabama so but hypothetically if you work for a large cloud service provider um you might have access to a lot of ip addresses that you are within your purview to scan and so that's where mass scan can be really useful same if you're a pen tester working on one of these engagements with a cloud service or a very very large enterprise environment mascan can be super useful all right

so this is a really boring screenshot of my math scan um this is where i confess that i made a typo and cider notation is is real fun and real finicky so if you make a typo sometimes instead of scanning 64 000 addresses you can scan 16 million addresses so shame to me i should not have done that that was an oops but you know now i have the data so i can't give it back so that's fine um so basically what you can do with this is you can figure out okay from that data i had 225 000 hosts that were alive 225 000 hosts that responded back to me and either said yes that port is open

no that port is closed of those 225 000 i had roughly 2500 hosts that had either port open and then about a thousand hosts that had specifically port 1883 open now not saying you should disregard the remainder of those machines on 8883 but if you have limited time and limited um ability in your test environment and you really want to go after the low-hanging fruit i'd recommend just sticking with the um the non-tls version because it's just a little bit easier to hack and so you want to kind of take care of those things first and then move on to the harder challenges so now that i know sort of the hosts i would want to scan i have to become

really familiar with how mqtt actually responds and how it responds much like http is with response codes um and this is the list of the response codes and the the goal or the thing you hope to get is zero connection accepted right so i wrote this little python script uh this is just a snippet of it basically you can iterate through a list of ip addresses um connect to the mqtt port on 1883 see if you get a reset uh or an rc code zero back and if it is then that is probably a host that will allow you to kind of dig in and manipulate stuff around now that's not to say that you should

disregard any other of these return code responses because for example if you get a number two code back it means the connection was refused but just because your identifier was wrong so that means that they have some kind of client id scheme that they're enforcing if you're able to figure out what it is it's possible that that's all they're doing is just a client id one of the interesting things about mqtt is that you can set a username with no password so you just because their username or a client id exists doesn't actually mean it's protected in any real way so you may be able to either guess from a serial number or guess from a mac

address or just start trying things until it lets you in um same thing with bad username or password um you can just start trying a lot of passwords i'm sure if you have been around security for a minute you'll know that password one two three and admin admin and all of those things are still pretty much the same uh so don't just disregard them unless you're on a really limited time engagement but for this particular example i accepted uh just kind of see what i could muck around with so i spent some time playing with pajo mqtt and um i'm much more the engineer that likes to grab other people's code and see if i

can modify it and actually write my own code that's just really more my style because i'm a little bit lazy and i found this awesome awesome tool called mqtt pwn by akamai highly highly recommend that downloading it if you're gonna get into this area of research it's so so so much fun uh and it's also really easy because it comes there's two variants you can use you can use it locally uh with python or you can use it in a docker container and the docker container is magical i love it so much i cannot understand how much i love this tool um it has modules for brute forcing username and passwords it has a module

for sort of creating your own botnets it has a module for topic enumeration reconnaissance basically anything you'd want to do in the mqtt space is probably in this tool and it's just really really well written um so one of the things i started to do was look at topic enumeration because i specifically wanted to know what types of information i could grab from these hosts that were allowing me to connect to them so this is just a snapshot from one of the services that i was able to grab kind of in the middle mainly because uh this host uh the top ones were the top ones were kind of very specific about what kind of

host it was and i didn't really want to put that in a slide deck but you can see here that i can subscribe to all of the messages received in the last 15 minutes all the messages sent in the last 15 minutes so that would be really useful information for me um it allows me to kind of figure out what this actual machine does what the broker is doing and then also what kind of information i can get from it so real world use cases i could talk to you today about really important things like power grid or how you know hospitals could be overtaken by you know rogue published messages uh how dams could be opened in flood

cities but i don't want to talk to you about that today i want to talk to you about something even more important i want to talk about the internet of cows because i feel like this just doesn't get the news coverage that it deserves so there is this is a true story i'm not making this up you can google it after this talk internet of cows it's a real thing cows are becoming increasingly more connected to the world wide web um a particular company makes a device that is basically like a remote cow steering device if you can think of it it sits on the cow's back and sort of nudges them when they get

off-piste so the gps will kind of set back to the controller it's like hey you're not supposed to be there and it nudges the cow and the cow walks away uh a lot of these don't have the best security uh don't have the best implementation of these protocols and so imagine if you will that you could disable the tracking and guidance system so now you have cows sort of loose on the prairie running around or maybe you know you just do it to annoy the cow if you're a really crappy person um but what if what if i could get access to the guidance system and drive these herds of cows and create the internet of zombie cows

think of that think of that when you go to sleep tonight about why iot is important do you want to get attacked by zombie cows no you do not so that's why iot imp security is so very important all right but the good news is it's getting better it's don't worry it's in the cloud everything is awesome in the cloud that's everything is awesome it's in the cloud there is part of that that's really true um because a lot of devices are migrating to cloud services cloud services are able to make security easier to onboard to and less friction tls and ssl enforcement is much more broadly adopted by people who are entering the space because they make

it a requirement to onboard to most cloud service providers um the better mqtt you'll notice i make this typo a lot it was a typo in the slide or in the topic i am really dyslexic and i do not well double comments mqtt broker defaults are a lot better in the cloud because they're managed by cloud service providers um there's also the ability to use a lot more detection and automation uh because you get access to cloud services that monitor your logs and send alerts and can kind of protect you from ddos in the on the cloud layer so it's it's it is better there are things that are better but there are potential downsides

cloud environments are highly configurable which is good that's a good thing right we want to make it easy for the customer to join and onboard to the cloud but if you worked with users you know that if you give them the possibility to undo security there will be a situation where it happens so the cloud kind of does make it easier to do bad things at scale it means that you can now connect billions and billions of devices and then turn off all the security which is bad um you can imagine that systems may not be designed with the appropriate redundancy if you're relying on the cloud for your mqtt broker services say in a factory you really have to

think about what you do when the cloud is down and if there's a cloud outage uh it's something that people are really starting to grapple with now as they move their iot installations to the cloud and it's something they have to be really careful with when they're designing their systems so key takeaways mqtt is a lightweight widely adopted protocol used for iot publish and subscribe actions it can be and often is insecure because it doesn't support security security natively except for username and password which is set in clear text when deploying iot devices with mqtt brokers it's highly recommended that you use over-the-wire encryption and also on device encryption because if i can get your username and password by

getting one of your devices odds are i can still get to your broker um the ideal situation is that you have very specific access control policies and device certificates per device so that your light bulbs can't access your garage door and your garage door can't access your coffee pot and so on and so forth um resources i will be putting the slides up on my website which is very very tragic right now it's really just a place where i dump my slides uh but mass scan is the tool that i used to enumerate some of these hosts mqtt pwn i talked about that's the the repository where you can get it um i also if you're looking to get into

iot and you you really haven't um had a chance i really really love this instructables uh document i think it's four or five parts it walks you through creating an iot device with raspberry pi setting up an mqtt broker and then hopefully once you get that set up you can use mqtt phone and kind of play around and figure out the vulnerabilities in mqtt yourself uh you can follow me at that security check on twitter and that is the end of my presentation you