VivekPonnada

foreign

[Music] I thought this was interesting if you look at the left side this is part of that recommendation from Snyder as to what they're supposed to do for this or in general in the other I see a specific attack they mentioned device protection parking workstation Harmony Network Harmony private heart and I looked at it and I thought come on even if there was no end controller even if you go back 10 years ago back to stock snap what would be different in our recommendations not that's what prompted me to talk about this right we focused so much on zero days we focus so much on nation states attacking each other but all said and done the recommendations are no

different from yesterday it doesn't matter what the attack is tomorrow or the day after what we recommend doing is exactly the same but you can take the OPC UA recommendations right so the toolkit that we talked about that has both the generic PLC attack Vector as well as the OPC UA same thing look at the right conditions first up proper segmentation of itunot this is what any O2 practice you would recommend on day one proper segregation allow listing it's fairly common in ICS networks a lot easier compared to it to do allow listing you go down the list and that shows firewalls monitoring and blocking the track enabling and aggregating audit logs if you go down the list every

single one of them is a recommendation with only practice students have been issuing or talking about for a long time so nothing that incontroller provides or any other attack the other tool here is going to change these recommendations right let's put all these in perspective project-based Camp how many of you have heard of this one back in 2012 it's one two I strongly recommend you watch the video from Ray Whitman and Dale Peterson on the S4 Channel I put two links here I'll share this with whoever wants but it's amazing that back in 2012 they looked at a few controllers and said let's do an evaluation Let's do an investigation of security of these controllers they found uh in fact they

had to stop in a couple hours because it was so bad this is 2012. and they said ah it's not really worth it because you know proper credentials hard quarter credentials be used credentials all basic stuff High authentication access control it was pretty bad in their list of things so they had red check marks against every controller it was so bad that they stopped and let all those oems of the controllers you know they said hey your systems are so bad go fix it and 10 years later they actually did another video recently it's not on YouTube yet but it'll it'll show up soon enough nothing might change folks older Control Systems nothing much is involved

more modeling Control Systems you can go to the same vendors GE Siemens Honeywell if you buy a modeling control system today they're much better they have a lot more security features built in um so they're capable now of incorporating a lot of security into the infrastructure into the project right but the older system still remain in play none of these systems are replaced overnight so unlike in it right the traditional OT systems are not replaced every three to five years they replaced every 15 to 20 years so something that was bought for and paid off in 2000 is still up and running and it's still insecure quite a lot because it doesn't have access access controls it doesn't

have authentication it doesn't have any security features no encryption for example there's a list from Temple University they published this you have to sign up with your email and they'll make sure that you're not a malicious attacker right to be able to give you this list but this list has more than 1200 records uh all know or disclosed incidents right most of them are ransomware but they have a whole school of items and they map everything to the minor attack framework and almost every one of you have heard of the minor attack framework um there is a section for ICS as well so that's something you might not have looked into but there is a specific

section which includes some unique things for ICS right like loss of views lots of production that's not directly into it so they have a specific section for it so I recommend you take a look minor attack for ICS the key is if you look at the overall picture again it goes back to my list where targeted nation state attacks on Industrial control systems do exist but they're not the vast majority audience of those 1263 you can move these six or seven the rest of them are not I see a specific attack right they were from I.T they were from insecure mode access they were from some kind of collateral damage not necessarily targeted but I see us

so they extended last year why is it an issue in OT right so most of OT control systems are proprietary Hardware with unique protocols because these were developed when there was no equivalent I.T protocol I.T Hardware to just leverage for coaching purposes so back in the day control systems were hydraulic pneumatic previous to digital right ecosystems existed before there was anything electronic or digital so when they were adapting control systems with digital they had to come up with their own so the ge's and Siemens and honeywells of the world designed their own control systems and at the time safety and reliability were the requirements not necessarily secure all right so the protocols were developed with no security

yes security no so it's also called this insecure by Design some people hate this because well it was never requested of the engineers to incorporate security so how could you call it security by Design because we never asked as Engineers without something with security and then black networks this is a pretty big issue pretty much every industrial site you go to has a pretty platform more common in some applications like manufacturing where you walk on the factory floor touch a device you can log into any other device across the network across the floor across the company across the line across the whole company in the OT world because it's one network right it's usually it's like one or two guys

that manage it and maintain it a lot easier for them if it's all insane now copying files transferring content connecting to billing systems for example connecting to the sap World connecting to the tool shop inventory system whatever that might be a lot easier for them which is a flat Network so it's a big problem so that's why the collateral damage right the effect one system you have access to the Whole Net patching is not always possible this is again a big problem compared to it an idea what do we do right as soon as there is a known vulnerability an exploit or whatnot you're like what's the patch the first thing you do is you'll find out from the vendor from

Microsoft whatever the case may be find the patch apply it so you're good that's your mitigation in the OT world that's not practical at all number one patch might not be available because yeah 20 years ago this system has been obsolete for 15 years so there's no way a vendor patch is available on the vendor himself or herself who's out of business all right because that's the vendor that you bought from 20 years ago 30 years ago it only existed anymore testing and validation there's no lab there is no demo there is no place to test and validate and protest in a representative environment and you just can't do it the only time you can test or validate is during an

outage or a shutdown that only happens every year every three years every five years every 10 years so you might not be able to validate there is no redundancy so that system is down to validating your testing your production is impact no outage window available that's the other thing right so yeah just because you have a vulnerability in the I.T world because everything is connected to the internet by default we assume that it's exploitable if one exploit exists out there it's exploitable everywhere right in this world if you properly segment and isolate it might not be exploited maybe it needs admin credentials maybe it needs certain access at that particular HMI or workstation doesn't have

um so there are many other things that make the exploit maybe not relevant to your environment right that needs to be valid the last one not always relevant to risk production this is something very difficult for a traditional I.T person to understand going back to insecure by Design taking modbus protocol industrial Control Systems it's hardly ever used for control these days but at least you've heard of it right OPC UA we talked about OPC before uh it is a secure version there is a secure Pathway to implement it there is an insecure way to implement it as well so those protocols especially if you have no authentication no encryption they will just follow your command so if you have

access to a fully patched Workstation it works this itself dispatched but the protocol taught into the control system is not encrypted so you can command it to do something just like in the Florida Oldsmar case where that HMI or workstation could have been 100 patched the controller that it's talking to could also be 100 patched but if you were able to log into the HMI and just add the content of live change the set point from 10 to 50. it'll execute so how is patching relevant in this case it's not patch but our other systemic issues in this world right awareness this is increasing but we still have people let's say we're not a Target

well yes you're not a deliberate Target but you are part of the ecosystem right collateral damage that's fairly common the other thing you used to hear more is your air gap comes this is a very difficult Challenge and the first thing that comes out of someone's mouth is your air gap you know for a fact they're not considering their their plant um as connected or Network their gaps don't exist anymore in OT they're always connect like in the colonial example even though you think you're physically disconnected removed your networks are separate the fact is you had connections you had connections to your internal systems Network internally uh to your inventory systems networking totally to your

management interfaces it's like one engineer I mean I'll give you an example if you look at it finally you can find you might have 100 different plc's you'll have a few Control Systems Engineers that are supposed to manage the whole set of plcs or DCS and everything and for them to go in front of the HMI which is in the plan you have to put on their coveralls on their boots hard hat and trip to you know the say pour it over and get to the plan it might take half hour to get to that workstation however they've connected it to their computer on their desk they could still perform their job safely and you can

think and whatnot and not have to waste 30 minutes to go to the HMI right so more often than not they have connected because there are other reasons for this network even if that's not established those connections were not documented and nobody authorized it they've already done right compliance remote lenders access right these things exist many remote access options exist especially if you have a compliance regimen where you have to maintain certain emissions so you have to maintain certain quality of product you have external connections for sure because how do you maintain right whether it's connecting to your ratepayer rate basis or connecting to your emission system you have external connections that are probably not documented

and then backups I saw a meme earlier today about you know we lost our server Hey where's the backup on that server that's fairly common in the OT space where it's not really thought through right where the backups are going have you tested validated backgrounds greatly ever anyone else management support this is hard because for lack of all the stuff right for lack of inventory for lack of visibility for lack of truly understanding what connections exist what your inventory is in your OT world how could you ask management for support like what would you ask you go in front of the marriage and say oh I need to improve my O2 security posture well where do you start

right now they find it difficult to evaluate the risk in the IT world there's a lot more documented processed with a lot more risk awareness risk mitigation process where you have a clear case to say these are my assets this is the risk this is the budget that could resolve a certain amount of risk right so you have this cap conversation frequently in I.T not so much emotion the resources and funding are a big problem because it's very hard for management to make the decision for lack of disability legislation non-existent in most cases and when they exist like in the power sector nerkship there is a compliance regimen it's limited to medium and high critical sites and it's also very

prescriptive it's not very risk based right as in what a has to do versus B has to do is exactly the same it doesn't matter if a is on an island where you know if you impact that particular site a whole island doesn't have power versus you're here in the middle of um let's say Calgary where if one side goes down hey nobody cares because there's other power producers on the network right they try to uh managed address somewhere they have slightly different rules but for the most part it's very descriptive it's not risk-based that's a problem because um again compliance is not security but even if you follow the compliance procedures sometimes it's just wasted

dollars wasted effort because they're not actually fixing the risk but you're just doing something because you have to follow legislation in many other cases other credit infrastructure other than power um except for the most recent oil and gas and the TSA came out with a certain regulation for TSA pipelines after Colonial uh the rest of the Industries or binaries petrochemical there is no legislation so again when you go in front of management and ask for funding you can't really tight back to I have to do it right you can tell them what the risk is because you have other problems you can't say I have to do this because of legislation so you're kind of stuck but

you don't know how to make the conversation happen uh this is a game you need to OT it's a long project Cycles some of these projects take a long time to build so how many of you have started a project on the IP side and finished it in six months several I would think other projects might take two years three years whatever right but you finished phase one in the first two months you went through multiple phases you got you know the whole company covered in maybe a few months or a couple years in the OT space um let's take the worst example nuclear power plant but first of all you can't even build anymore these days you can't

get permission to build a nuclear plant but even if you did it takes 25 30 years to get to the finished stage in a new tripod less so in other industries by petrochemical plants from the initial design stages all the way to build and production I commissioned several LNG plans admission several refineries several pipeline applications pipelines might take anywhere between two to three years some of you have heard of Keystone XL where it didn't happen for decades right then there are other pipelines that actually got through the permits and approval process it might take 10 years to get there so long project site so the decisions that you made are security 10 years ago 15 years ago are

relevant today and that's a problem because you didn't have any security requirements 15 years ago right so when you're building projects that's one of the problems we have and then the construction engineering firms that are building this they're not the ones operating that's another problem right where they are forced to think about their current requirements they're not thinking what could happen five years from now because that's another company altogether that's operating and then upgrades on capex driven this happens every day so control systems like we mentioned you know are not replaced for 15 20 years and they're typically capex right so you invested a million dollars two million dollars and an upgrade and your Roi is over 20 years

you're not doing anything year over year so no maintenance budget for cyber security because that was never a big 20 years ago right and then public good versus private cost of security and this yeah critical infrastructure whether taking a water treatment plant or employing a pipelines case it's a private company right they make their own risk-based decisions so Colonial might be 100 okay within one week downtime for ransomware attacks but as a country the US the southeast and some parts of Northeast they suffer because gas prices wrapped up people didn't have gas on time you saw those pictures or videos of people waiting line to get gas right so that's a public problem but the cost of security is with a private

end all right so that's a conflict of interest where critical infrastructure people depend on it but the expenditure has to come from a private organization and then nation states yeah I mean the whole talk today I'm I'm trying to tell you that don't focus on nation states but that one-off chance that nation states do attack a sector or a particular company that is not a fair game right you know how are you supposed to protect yourself against nation states when your whole OT system was never architected for security you're not having to deal with nation states uh one character in this case is a combination of security cyber security insurance so in the past this

wasn't a deal but these days you can't even get insurance without some level of cyber security hygiene so in terms of sticks you know legislation but there are some benefits as well but this is the extended landscape I wanted to put this in perspective that we have so many other fundamental systemic issues that needs to be addressed that need to be addressed without having to focus on the nation states right so the nation state part is just a tiny aspect of it and then the reality check industry Trends and everybody needs to be aware right digital transformation Cloud analytics the previous talk was about Cloud right it has made so much progress in Cloud because you see the

scale you see the cost improvements you see so many benefits so that will slowly percolate to OT as well but that increases the attack surface quite a bit for OT especially because we have other problems iot projects that are happening every day so iot as you all know I guess a benefit overall because you can scale projects quickly you can Implement I always use an example years ago on a Time phone I needed a level transmitter and to get that properly commissioned the level transport itself was like 500 bucks but the cabling the power supply the PLC the connectivity the fiber to the PLC to that location for a couple of Miles all the total upwards of 50 Grand

just to get a level indicator right level transfer but if you have an iot level transfer that could be a thousand dollars but you got the signal right away for a thousand dollar investment you got your letter so iot is pretty popular in the OG space as well not just in it a lot of projects now stock with iot because it's cheaper faster so that's going to continue to happen right but think about it we just talked about how OT has as a whole isn't secured by Design and now you're connecting parts of it to the cloud director so when you were thinking it was isolated you were safer and now you have direct connections to internet every day

because of this increased iot use cases unlimited experience professionals this is a problem every year I would think but more so in OT because the traditional OT knowledge those people have retired or are retiring more and then the younger folks are not getting into OT right they're interested in I.T not necessarily so I always pitch people especially if you're not super excited about it but you have some I.T knowledge and consider OT because your itemology is already probably Max required you're only good you already know what needs to be known in the OT space with respect to the it knowledge but then you can learn industrial control systems and maybe that's your deal risk analysis

um coming to OT risk and I.T risk again let's go back to Colonial so George asked the ciso or CIO off Colonial what's your risk they're not going to say my it risk is this my OT risk is this because it's irrelevant for them planned outage right safety of the Personnel something being down something really impacting the environment these are all Enterprise level discussions not necessarily OT or I.T right so why are we now not dealing with OT like we completely forgot about OT for so many years just because we know how to handle things in it we know how to work out budgets we know how to manage or mitigate risk and I.T so we've

been focused on it we will never focus on OT for so long right but it's meaningless to just think about it risk on its own you have to think about the whole thing and then ransomware for example just affecting it systems again the connectivity from it to OT has tremendous increase so OT systems are higher risk because they're not perhaps they're insecured by Design they have so much collateral damage and so just focusing on IQ systems is Irrelevant for the organization so what can we do right the solutions are the same just like in the Mandan recommendations just like in the Schneider recommendations the solutions are always the same identify get asset Discovery Tools in

place get your network monitoring in place right assess the availabilities for your network figure out what is truly at risk and then follow the monitoring protocols right so once you're able to put some asset Discovery and network monitoring place you then can detect anomalies detecting anomalacing OT is a lot easier than an I.T because OT is very deterministic you know exactly what all things need to happen to get this process right there are 10 15 20 30 variables that specifically work together in a certain fashion it's a physics right chemistry and physics are not going to change think about a chemical reaction you need to have certain temperature certain pressures certain combination of molecules to get this reaction in place

that is not going to change all right so if it varies if your sensor says I got 10 bar pressure when you know for a fact that chemical reaction cannot happen at 10 bar because that's the gain chemistry and physics combined you can detect anomalies quickly they say hey the sensor is faulty right so normal detection emote is a lot simpler a lot more straightforward than an ID and then the last one reducing impact this is another very OT unique thing that is an advantage so think about the situation where we talk about attacks right nit I'd say email server or any traditional IP server it's vulnerable but then there's a vulnerability out there the only way to mitigate is to patch it

right away right as soon as you get the patch you update the patch lock for daily good example you can't do anything without a patch you have to patch it and that's the only way you can mitigate the wrestling in OT actually that's not the case I have hmis and workstations that have all kinds of malware and that has no problem running the planet because as long as they're able to block that malware from talking to a schematic control server as long as I'm able to isolate that device I can still run my process that has nothing to do with the number number one and number two it is the impact this is another unique thing

think about a tank for example overflowing right let's say you have a dike and let's assume that it's not poisonous it's not a major chemical that can cause other environmental issues but let's say you built a die for a wall around the tank so even if it overflows the worst case situation is it's contained right so your PLC your HMI whatever is under attack even in the worst case somebody calls the tank to overflow you're reducing your impact by something that's physical something that has nothing to do with control system nothing to do with digital so that there are things in OT that we can do many many examples where you can build the system in such a way that even in the

worst case scenario nothing bad happens kind of like the nuclear wall with this containment area right that didn't exist in the Chernobyl location but everything in North America is massive closed Contender same thing in Ukraine now you're all seeing all this news about constant shelling but most of these places have this massive little Dome that limits everything right so nothing comes out of that so here are things in O2 that we can build for a lot less expense and that's the key right the building that containment Zone costs you 50 grand for supplementing basic cyber security practices cost you five thousand you're gonna use five thousand dollars to do this but in the OT World

there are many situations where fixing the whole system from insecure by Design all the way to monitoring might cost you hundreds of thousands of dollars versus a containment or some kind of backup that might only cost you ten percent off right so that's another possible and then these other things and these are the common threads across the world right Personnel training rot people are not cyber security over there it's improving but as a percentage probably less than 10 in my experience are aware they're not networking people uh they're not authentication encryption people so a lot of training required in the O2 World especially if you're using more it like Solutions right training is a big deal leveraging

industry groups or Intel sharing it's happening more and more pretty much every vertical oil and gas mining electric and a power sector there is information sharing analysis centers ice hacks all over it's a lot of Intel sharing it's happening quite a bit but more needed Public Safety candidates pushing a lot of common thread common information sharing as well and then faculty type establishing Disaster Recovery policy it's kind of interesting how like you walk into a family for example you'll see their safety plan pin the first thing you'll see as you enter is as part of your safety orientation you'll see that this has recovery plan they said you see there are five protection plan their safety who to call

something would go wrong but that doesn't include security crime that has to change the recovery time objective again taking Colonial example if you're Colonial and you're recovery time objective is six days or seven days you're good you just needed to tell people that that's your recovery time right you say hey I built my network I'm resilient for X Y and Z but my RTO is six days so then people know that yeah for six days maybe you won't have this pipeline or this company but it'll be back up in six days come back to tabletop exercises instant response same thing I have not seen um companies do this regularly they are improving these days anyone here in a

tabletop exercise that includes ICS one two three that's increasing right slowly more and more people are trying to do it but they need help so I.T definitely leverage your best practices right include ICS as well as part of the exercise and then finally it's not possible to 100 Outsource OT security because it we've seen that right IP as in more and more systems of network more and more Cloud utilization you could Outsource to an mssp most of the systems but in OT because no matter what you need the plan Personnel to be involved so they are the ones that can do forensics they call it troubleshoot them they can troubleshoot the alarms they can tell you if this alert really

is a problem or if it's a false flag so you need to have the constant involvement of the black Personnel so you can't read 100 Outsourcing but you can have an IR retainer right you can have external help for it like this right so there are many things you know too that are very unique so you still need the plan person to be involved or for other things you can leverage IPL like tools so the Crux of my messaging folks is that media and in general the new Cycles Focus so much on zero radiation and nation states attacking so you hear about Ukraine you hear about Iran China you know our is the name is the nation's

power grid at risk you know are we at risk for oil and gas consumption you hear so much about that but our problems are a lot more basic yes at the highest level you still need to think about those things but we have fundamental issues in OT that are very basic but still you can solve any questions folks

I must have been crystal clear but in reality we have standards now whether those be the TSA directives whether that be your exempts right that as you said can be very prescriptive and the reaction of of these nation states is in the media and of course all of the C streets see it and so what what do we do as as companies who put in an IEP solution an it or or an OT now solution so now we're taking for example allow listing or right and we're trying to from an I.T side and now we're trying to force it into ebot operations and the IT people usually you know now they're Security Experts but if they

come from the I.T world and now they're trying to learn to OT and there's that whole Clash right which as a cyber security VA which is me I'm in the middle of it and I've I've been on the OT side doing documentation for certain steps and now I'm I'm putting in an IT package into the OT world so how how do we bring these two worlds together and I know you're talking about training and all that but it's these standards now are pushing us on OT and is actually providing more risk to the OT environment because of these this I.T software right so great Point let me unpack it a little bit because for the

context right we need more of these marriage counselors number one so we need more people like this right the the problem remains that OT is unique right I.T and I.T security have done things based on what they know and a lot of knowledge a lot of um established processes in I.T that are really good but not everything fits well in OT so it's uh it's a push and pull so OT on the other hand can't always say we're so unique that don't let it touch our stock right we used to do that I did that for many years right because I I can give you examples of where I needed this laptop because that laptop that

particular OS version that particular tool worked for my PLC and that laptop it tells me that I need to patch it if I patch it I'm true I cannot talk to the PLC so for years OT did not let it touch our systems because it didn't understand our work right things are changing we're using more i.t-like tools at Purdue level two Purdue level three every tool there is it like in fact they're exactly it tools the purpose for OT applications the only difference between this Dell server this HP server that I'm using in OT is that I've used a config tool for the PLC on my server but the server itself is part of the mail no different from what it

used on their side right so we use more and more it tools networking switches routers are very similar it's not exact same in OT compared to it the only difference might be that some of this Hardware might be ruggedized for the industrial environment other than that the tool the interface everything is the same so the bottom line is OT leverage most of it equipment in the past 15 20 years to become more functional more efficient so they can't now say I.T don't touch because it is the one that knows how these tools work on the other hand the process right the one thing that it doesn't have to deal with in the IT world compared to the OT world is the

process safety efficiency productivity reliability things in a Refinery you don't care for data as much you care for the process safety you don't want the process to blow up right that's the part that it has to learn so what we're seeing increasingly is this collaboration when a few years ago the CIO and ciso had nothing to do with the OT board things changed these days The Cisco CIO they're responsible the system especially is responsible for the overall security of the company so that is how Virginia the second thing that helped was forcibly putting people in places so take an I.T guy and put them in the production environment it also helped training OT people on it tools go get

certifications go get traded so then at least you understand where they're coming from then you can differentiate between focusing on email security for example that has nothing to do with my reactor or my Refinery process right so the training is mutual not one way yes it has a lot of process a lot of knowledge a lot of capability however they lack the or discussing knowledge right that recognition has to come from the IT world OT on the other hand has to also recognize let's not read in the wheel let's not learn how to segment or architect my network because it did this 15 years ago let's leverage the best practices for MIT so as I put this in

resource section is not only nerd sips and Merk zip so taking one piece apart from what you said we don't have standards everywhere we have certain regulations in certain industries and certain protocols but not uniquely everything so ifei ac62443 is the standard that is horizontal can be used in healthcare power oil and gas so we have developing standards that are applicable across multiple verticals and industries so you're seeing a lot of growth in utilizing and leveraging that right that'll talk about zones and conduits segmentation in fact they have security levels where security level one would be I'm trying to protect my plant against a wallaby hacker a script Kitty that just found my IP address on Showdown to security level

four where I'm trying to mitigate risk against the nation state so that is a whole one through four where you're developing your program because on day one no one can get to level four right if you're a company that is looking at investment versus risk you can suddenly start thinking about protecting yourself against nation states that's a step that you'll get to eventually but there is a security level of concept as well and then others so minor attack we talked about it I'm part of that top 20 secure field supporting practices programming PLC securely CCE methodology consequence driven cyber informed engineering that's another engineering not security but it's for engineering helping you build Security in the orchestra so collectively folks

itnot are talking have to talk even more and the bridging has to happen in a company in certain sectors in water for example there are no OT security folks it's one I.T security person that's responsible for the whole plan no matter where this is there they're having to learn it the hard way right there you can't Implement an I.T tool like multi-fact authentication very simple in the it wall it's it's a give I like when would you ever say don't do MFA like this that's not a thing in it but in OT that could be a thing because think about an operator in front of a computer that has to adjust something or shut down a piece of equipment because

something is wrong you do not want them to forget their password you do not want them to enter their second Factor you do not want any any delay not even a millisecond Delay from their action so that screen is always open it's unlocked no password no nothing there is a use case right so that's the learning that needs to happen so are there any associations where you are pretty much all these are all combined so we're seeing a lot of otmit people participating in all these exercises and learning from each other