Don't Panic

foreign okay now do you all need to stand up and shake up a little bit you're good to keep going all right so i'm uh going to introduce you to the next speaker that's uh dave manucheri the security analyst for ai dot moda now david started his professional cyber security career half a decade ago after being recruited by the department of national defense he has since worked with the startup linchpin labs which is acquired by l3 harris technologies and started up his own company after becoming an independent security analyst consultant in 2020 he has found and disclosed half a dozen chrome zero days aside from his professional experience he has over a decade of experience with

linux security and being personally targeted by apts in his free time david enjoys researching cyber security trends in the industry so i welcome david manitori to the stage and don't panic

all right is mighty working sounds good thank you for the intro and thank you for being in this dark room instead of outside by the beach so a bit of background on this talk uh if you're very technical there is going to be a couple things that are relevant that you'll enjoy i keep thinking this talk isn't relevant technically anymore and then every couple of months somebody asked me a question like oh here's what you need to do but if you're not technical don't worry i am going to go over i'm going to try to make it understandable for everyone so bio already had a good introduction but the main thing i'm just trying to point

out here is i have done a fair amount of disclosures so i'm familiar with when disclosures go well and when they go poorly this is a story of one that didn't go well i do have a gap because i worked for the government they don't want you to say anything so for anyone who's not canadian which is probably about a quarter of the audience but uh rogers it's a large canadian isp at the time that this story is going to be taking place this is the beginning of 2017 so they have a little over 2 million about 2.2 million users hitron is a company based in taiwan they mostly make like cable modems so you know

the stuff that like you know you see like logic have the supplier hitron's one of those companies now i haven't been able to find any hard numbers on how many modems rogers has that are from hitron the only number i was able to find is that hitron overall globally has shipped 40 million or so devices with rogers i believe it's it looks like a little over half of their modems at the time were hitron so my guess is around a million i don't have hard numbers on that so that's just speculation now for some people who have dealt with iot this name might be familiar uh mirai it's a it was or it's not as big anymore

it's a botnet mostly targeted embedded and iot devices and it mostly yeah it had it kind of its peak in 2016 and 2017. now this botnet in my opinion was not as impressive or not as scary as it could have been it mostly was used just to take down websites uh the thing that's kind of scary with these type of botnets is even that this particular one only really just tried to do ddos attacks if you have a botnet like this you can do a lot more than just take down websites if you have a botnet that is you know everyone's modem you can do a fair amount of stuff now you can't see like encrypted traffic but

you can still determine you know what websites every single person's visiting even with ssl that's still possible so this particular botnet in my opinion wasn't as scary as it could have been but it's scary knowing that it could have been a lot worse uh mira's mirai is also interesting because the source code was leaked so you don't have to do any fans fancy reverse engineering which i don't like doing so i promise this is the only code graph and you don't even really need to pay too much attention to it but the way miri works for finding other devices to infect is basically tries random ipv4 addresses on the internet so just randomly tries an ip address and

checks uh it tries two ports it tries port 23 and 23 23 and it just randomly tries to pick those not very advanced but surprisingly this works remember remember that number 23 and 2023 it will come back later and i'm going to go over a bit about what code injection is so and don't worry i promise this all ties together so this is a tweet i put out where usually like on a modem if you don't work for the isp it's not you know yours usually you're not supposed to be able to run commands on it so command injection is a class of vulnerabilities where you're able to run commands on something that usually shouldn't

so this command injection for this to work you have to be locally on the network so this is not remotely accessible it's only if you're on the same network now i know there's a few of you who are probably judging a little bit that i tweeted this don't worry i did google it first one of my friends actually disclosed it like a year before i did so this was not the first instance this had already been known for quite a while i was a bit disappointed when i thought i was first not even close now when my friend disclosed it he didn't make a sarcastic tweet so it when it got ignored uh turns out sarcastic tweets do get

responses very quickly so about within a week rogers had fixed this issue which was kind of surprising to me i'm like oh i guess i guess people do sometimes read stuff but only on twitter now i'd come forgotten about this moved on and i tweeted this which this is i'll explain what's going on here so you have i think everyone for the most part is familiar with what an ip address is now there's basically two there's ipv4 which is what everyone's used to you know 1.2.3.4 there's also ipv6 ipv6 it's a much larger address space it's people often forget about it it does exist it's not not as commonly used but yet it is there so

what this is is uh the firewall on this particular device on this modem uh they had forgotten to configure the firewall rules for ipv6 they've done it for four but not six now also in this tweet again i didn't put it in there but for the ipv6 part here this is only accessible if you're on the local network so i'm able to now reach telnet on port 23 but again only if i'm on the local network so on a scale of like one to ten about how big of an issue this is i would say like one or two um it's just like an initial you would still have to you know be on the network which is

usually once you're on the network then you don't really care about attacking the modem so yeah again a you know a one or two it's only just a minor misconfiguration they've you know they blocked it on ipv4 but not ipv6 now here's my disclaimer slide everything i've talked about before here uh it's i can prove it it's you know fact everything after this point this is speculation this is before canada had a mandatory disclosure law so this is from all the slides on here's my recreation you be the judge the times match up suspiciously so it's speculation but it probably happened so after i made that last tweet uh it seems like probably somebody at

rogers panicked a little bit it's like oh no like this must be a serious issue we got to do something so here's my guess of what happened so they saw this is the original um thing that i tweeted where you know you can see when you connect to uh through ipv4 you can see that you know telnet is open so they saw this was like okay we got to fix this we don't want to see open all right so they they changed the port so now port 23 shows is closed great yeah mission accomplished here's the thing though what the actual issue is and you don't need to worry about most of the the syntax

here through firewall rules and this again this is my recreation of what i assume it is is they were blocking port you know 22 and 23 for ipv4 but the the firewall rules are different between ipv4 and ipv6 so the issue was not the port number changing the port is not really the wasn't really the issue all they had to do was just have those exact same firewall rules uh for the other uh for ibb v6 as well that's all they had to do now the thing is is i like i can still change i don't have to scan for 23. i can just check 23.23 and yet now it's it's still open changing the port did nothing

and not only did it do nothing because their firewall rules were for ipv4 we're only blocking port 22 and 23. now it's accessible via ipv4 as well so they've opened it up to the local network which is not great they've also now opened it up accidentally because their firewall rules they changed the port and they didn't change the firewall rules before now they've opened up telnet to the internet on 23 23. now this is an issue a very very big issue and yeah after i you know tweeted that i completely forgot about it and then now we're talking maybe a year close to a year later i see this post on hacker news so i click it

and it's some it's an anonymous person talking about how they tried as a hobby to interfere with botnets to make them not work so you know the guy talks about what he was doing and even talks about a timeline and there's one part of the timeline that really caught my eye and it's this part where he says you know in late january 202 2017 the first genuine uh large-scale isp takedown occurred when rogers uh in canada carelessly pushed out a new firmware with an authenticated root shell of singapore 2323 that date looks very familiar to me very familiar extremely familiar so this is where again this is speculation but i'm gonna put two and two together

and say that they're probably related and as i mentioned earlier with the uh now it's listing on the internet even then i didn't notice because i'm busy doing other stuff in life that automated scanner where i mentioned it was getting on port 23 in 2023 it was discovered by botnets basically instantly and this person this anonymous person's claiming that it breaks a large number of modems now this is anonymous you know maybe maybe we shouldn't believe that person maybe they're just making this story up for internet fame so let's check reddit all right one person's reporting about it all right let's check rogers uh they have a they have a message board let's check them

uh people are reporting about brick modems more people but these are all just regular users right okay here's a rogers employee saying that there's some brick modems that's not great and then so ideally when you do a blender like this maybe you know you can just like push another update and fix it so let's see what the the fix is so again this is a rogers employee now at this point saying you know if you have one of these modems which by the way it was more modems than this uh you have to go into a physical store to swap it because they can't fix it remotely so this again they don't have to they didn't have to disclose this but

i imagine having thousands of people across the country need to return a modem just like in shipping and fees uh engineering time it definitely wasn't cheap uh probably a couple hundred thousand is my guess for like how much they had to spend on engineering time for this which was never an issue shouldn't have been an issue in the first place because you know first of all these none of these issues were new like you know the first one it wasn't unknown it was already known uh so you have to actually like look for things not just things that are popular when it comes to your network i think that applies for a lot of people here

like you know if you just see log4js in the news don't just look for that issue i'm sure there's other issues don't just pick what's in the news to focus on you can also communicate with the security community i love that everyone is out here today you know when i talked about the you know firewall rules that they just needed to apply it the same between the both if they just asked me on twitter what would i what i recommended i would have told them that and i would have saved them probably a lot of money uh so yeah you can communicate i've never found out who the rogers employee is who follows me um yeah they're welcome

to reach out if they ever hear this talk and also don't rush so from my guess of the situation is that you know they saw that oh no the port is open on 23. we just want to make that go away let's just fix that so they fixed that i used the word fix and quotes so yeah it's no longer accessible on 23 but now it's accessible on 23 23 to the whole internet so you've gone from the solutions much worse than the original problem you got to think things through even even if you just want to like go to bed and push it out maybe you know deal with the next day uh when it comes to security if you're just

constantly putting out fires you're probably creating new ones behind you and also just don't panic like uh from from what i've seen from this whole situation i don't think there was any sort of disaster type plan uh there i don't think there's any rollout testing you know once the once you push a fix for security i'll use log4js as an example once you have rolled it out to your environment how do you actually are you actually checking to make sure it is rolled out and it is solving the problem that you want to solve if you don't have a plan uh yeah you're probably going to panic and break thousands of your customers modems yeah that's my talk

[Applause] i think we do we have time