BSidesWLG 2017 - John Grant - Mistakes were made

hey guys I'm John I'm a conference virgin and feel like an imposter to InfoSec things I don't do cyber sick things I think that's just people trying to trick execs to get predictive wrong on their phones I'm from Tech Ron we make a GPS Hardware product a little bit boring but it is in critical infrastructure and a lot of power and why that's important is you need to know the time everywhere in the grid so that we don't lose power that's to do with phase angle measurement this came from a thing that actually happened there's a bit of a development for talked about we made this product and we gave it a 10 year warranty to compete in

the market for an electronic piece of hardware and to anyone who does hardware development that's insane but we actually did make a pretty good product and it does stick around a long time the problem with this is that the employment at substations doesn't stick around as long and we had someone I feel like after the BM og talk that I should probably either thrown cats or my team actually said there was a lack of cats in my talk so maybe I'll just like switch this up and say cat engineers which result in substation engineers they have some problems and their problems are in Vietnam who buy a clock they heck through a whole lot of Bush to

get to a substation to a clock that's been working for five years and not because the clocks not working but we've decided to add a new second and this leap second is a big huge problem for everyone who has timing problems so to do that you have to upgrade the clock and to have any piece of equipment and critical infrastructure it needs to be encrypted firmware and to upgrade the firmware to include the update sleep second you didn't need to know what the administrator password is unfortunately we had some catch engineer ring us up and cat engineer said mistakes were made and we have lost the password you haven't had to touch the clock for

five years so we I'm gonna talk about quickly how admin password is lost and that's pretty understandable how do we actually implement a reset on something that's so critical to infrastructure and do it safely and this is a bit of a kind of what we had to go through and funny story long way the thing we do is we take an aerial GPS time stamp and we distribute it on an old 1950s standard called IRB and IRB is a pulse per second with a bit code in it and a pulse modulated width and NTP which should be really familiar with what you might not be familiar with is precision time protocol or PT P and that's like

broadcast NTP with hardware stamping and error calculation through the network and it's really fast it's amazing and sure learn about it because it's coming and ieds here aren't things that explode called intelligent electronic devices and they were been around for a long time in substations yeah one of the biggest problems of why we lose passwords and substations and critical infrastructure is that clients usually kind of set these things and they kind of forget about them this is because the really big million-dollar pieces of equipment that live in substations kind of get set once that configuration kind of gets saved somewhere and then they don't touch up for years and years a lot of assets and critical infrastructure

and this goes for protection control of you know traffic light systems and big water gas power they all kind of rely on standards and once those things are set and how the company wants to use them as security policies they kind of don't touch them to change anything you need administrator privileges and a lot of people don't give that out willing it will be a few people at the top maybe he locked in a safe somewhere rather than that is usually a supervisor or engineering permissions control things because the device is encrypted traditionally what they had to do like our pork engineer was seen the unit back from wherever it was in the world and

manufacture than hearing he's on there to ship them back to New Zealand to get reflashed which is a big problem for them there was one case where they literally had to have a substation powered down until we reflashed the reset the admin password and sent the clock back urgently Freight just so that we could get things running again time sink is really critical to protection and control and anything we do change in this is really heavily scrutinized by a client so they'll send them to testing and they'll try and break them so so there's a bit of a talk but one of the things that we did was we were deciding on how to do this like one

of the ideas here was maybe they could ring up and we give them a special code and would reset the device but what we did is we did a little internal test we rang up our admin to try and get some information and much to their disgust our cat engineers were very upset that someone was trying to ask for a really sensitive information and we were just like yeah there's a chance there that you know if someone could just kind of hack us through a social engineering we talked about maybe downloading a special arming device on arming software for the device and then we'd have to deal with all the security vulnerabilities with that and we're just like now we're not

going a lot of these systems from talking about a usually air-gapped hence why they kind of need this time synchronization there are some dos data centers who use this in saying that that market and industry is changing a lot as everything goes to IOT and that's so what we did find out about our clients is to be on-site in front of some power stuff you had there's a lot of things that have happened they know you're on site you've probably made a phone call and there's a lot of alarm bells ringing as soon as you open the door we decided that having physical ask access to the device was a really good way of kind of handling this and so we

decided that we wanted three things we wanted to be able to do an arming and have an opportunity to not waste your power station or at least alarm everything in it do a power cycle and very importantly nuke all the RAM we use static Ram in our devices and that's changing an industry a lot of devices are using app on chip which means a lot of denote dynamic RAM and there's a lot of issues of that but at least in our fortunate case we could just nuke the whole static section and be like sweet we've lost everything and the reason for that is because our clients would be buying multiple of our devices going into substations they put one in and

then in about five years they decide to move it somewhere else so they'll put another one in there grab it out put it in the back of the van someone could grab it someone could hook up our software and start if you did a reset and still saw the configuration you learn a lot about how that power utility or water utility used their internal network structure so we decided that had to be wiped so we came up with arming the reset took you to be in front of the device you did a power cycle which raises a whole bunch of alarms in the system already raises everything and the client then reconfigures it with their

new admin password which is amazing for our clients because they now no longer have to send the device back to us to get reset yeah so the things we learned probably is the best thing at this point the things we learnt in protection and control is that there's already a lot of layers of security and what we're seeing right now is IT rooms are getting really involved in the design process of critical infrastructure not so much in the US but the rest of the world and things are getting really connected and smart probably about 10 15 years ago NTP and the substation was unheard of but we're seeing that really common now and with the event of PTP precision time

protocol and redundant protocols like PRP we're seeing that fiber is going to be everywhere you guys are probably going to be touching stuff that is the exact same thing inside protection and control systems which is really interesting at the same time for us it's a huge question of how we develop around there sorry that was a pretty small but yeah hobo is interesting [Applause]