Raising the Bar of ICS Security

welcome uh stephen master welcome to besides vancouver 2022 um track two and day two uh steven's gonna talk about raising the bar of ics security um welcome steven and um take it away all right thanks very much welcome everyone i know we're getting towards the end of day two so hopefully uh you've been enjoying lots of good presentations and uh hopefully i can add to that a little bit as well we've been talking i think more and more about industrial control systems and ics security and there's lots of stories some of which all in the news some of which i'll talk about um but i don't know how far we go with practical actionable advice out there

for folks who want to do better for security and industrial control system environments i know uh in my experience everyone acknowledges that it's a problem and doesn't want to touch it with a 10 foot pole because of mysterious bad things that might happen if we touch those systems that have been running untouched for the last 40 years or so so i'm gonna try and introduce the landscape a little bit here and then talk about some of the uh things that we can do to uh raise that bar yes once i get focused there we go i think we can all agree that when we're talking about uh ics security we're talking about protecting the real world

right um i pulled a bunch of words from various definitions of uh industrial control systems uh these are the words that popped out and uh on the left we've got some of the definitions or at least fragments of some of the definitions around industrial control systems just so we know what we're talking about here that we're we're all on the same page uh as i mentioned we're talking about computers or devices or systems cyber systems electronic systems whatever you want to call them i already said cyber there you go that can influence the real world or bridge the cyber world with the physical world another way to look at it is we have systems or technology that can manage

command direct and regulate behavior again kind of tying into the real world very very often we are talking about systems of systems we're not talking about a single device in isolation we're talking about systems that manage systems of systems of systems it's it's very much a tiered uh environment uh other other definitions i've said is the company or i've seen is the combination of hardware and software and network to again influence the real world and of course what are we trying to do here this is the actual goal which is automating an industrial process which could be anything from you know water to electricity to oil and gas to manufacturing to uh to pharmaceuticals to any number of

other uh sectors as well in fact one of the first um topics or sectors that comes to mind is critical infrastructure right that is really our primary focus i would argue when we talk about industrial control system security because we have industrial control systems in a lot of sectors that are critical to our daily life the 16 that are on the screen uh right now are the 16 critical infrastructure sectors as defined by the u.s government and some of them might be a little bit surprising right we see financial services in here but again if we can't access money we might have some problems uh day to day right we've got health care in here

we've got manufacturing as we've seen with with a lot of supply chain issues we've had over the last few years um you know we can do without certain things uh arriving on our doorstep the next day but uh there's a lot of products uh that are critical to our everyday life that are that come out of that manufacturing sector and there's a whole bunch of other ones here that you can see um and and probably make a whole lot of sense in the context of critical infrastructure the definition um from the us government is sectors that are so vital that their incapacitation or destruction would have a debilitating effect on security national economy or economic security

and national public health or safety so if we have disruption in these sectors it's going to have a real material impact on our daily lives and not just an individual's daily life but everyone's um you know nationwide or or in a in a geography that means obviously we need to do something to protect these sectors because every last one of them has significant automation in some way shape or form and that's what we want to protect so when we talk about a control system this is the forgive my my powerpoint artistry here but this is a very basic model of what we mean when we talk about a control system we have a loop we have a process

it's doing something it could be manufacturing it could be transmitting electricity it could be anything and we have a controller some piece of technology and that controller is going to send output to what we call actuators but that can be anything that could be switches or valves or or motors or burners or anything that we can use to interact this is our bridge to the real world um so a controller is going to send commands or direct actuators which will then have a change in the process which we measure using sensors providing input back to the controller and ideally it works in a relatively anonymous fashion in a loop here keeping our process healthy doing whatever it's

supposed to be doing on a daily basis or our daily basis minute by minute second by second we have other pieces here though we have engineers who help design and implement we have operators operating day-to-day technicians maintaining and they may interact directly with the controller with the sensors with the actuators and make sure things are operating as they are supposed to and we also have supervisory systems that maybe those people might interact with or that might interact with that controller or with many controllers again systems of systems and try to coordinate multiple controllers or provide that high level oversight in terms of what's going on so when i talk about a control system at its most basic

this is what i'm talking about we have sensors that are watching or measuring a process and providing data and input to a controller that then has a program and allows it to determine what needs to happen next it's going to send commands or output to those actuators which will then adjust the process as necessary so what can go wrong what what is an attacker trying to do here well an attacker wants to get in a position where they can disrupt this process in some way shape or form that can happen lots of ways if um an attacker is sitting we'll call it inline or on the network or has an ability to act to interrupt

this communication um that normally the the humans would be having with the actuators well they can change the actuators maybe they can interact with the controller directly or indirectly and that can change output and maybe that can change the view of input that is sent up to supervisory systems to conceal an attack or just uh adjust the sensors you know the old uh um tv show with the the bank robbers or whoever in the building and looping the cameras right maybe maybe an attacker can do that or they can reach right in from this top and affect multiple systems an attacker wants to get in get to one of these positions so that they can have whatever impact they want

to have on the process shut it down gain an economic advantage blow something up whatever it is this is what we're trying to prevent we're trying to prevent attackers from getting all the way to the point where they have access to control uh the actual controller control the process control the sensors the actuators and and take an action that impacts that real world that's what we're trying to protect against which is what i want to talk more about ics attacks you know prior to 2010 there probably wasn't a lot um i'm sure there were things going on but nothing at a um at a scale like we see today nothing that's that was truly remote and

automated and and widespread as as i'm sure many of you know stuxnet came onto the scene in 2010 um and on the slide i've bolded the um attacks or the incidents that had either did have or potential to have a direct impact in the real world some of these other attacks were um had either um they were gathering information or they they weren't manipulating the process directly but they certainly had impacts to industrial organizations um dooku and flame were more uh intelligence gathering they collected information that could be used for future attacks but shimon took out some 30 000 odd endpoints uh in saudi arabia but not directly affecting the process they took out the vast majority of the systems within

the organization dragonfly was a campaign havex was a tool it was again mostly just looking for information but it had impact it was able to overwhelm systems intentionally or or accidentally um this was was one of the earlier supply chain attacks that we saw as well it was embedded havex was embedded in legitimate software that people were downloading onto their ics networks german steel mail was just run-of-the-mill malware but as as we think we know anyways the operators ultimately decided to do something about it and brought their own mill down in the process black energy there's multiple versions but black energy too specifically attacked um controllers uh always the ge simplicity and uh some siemens controllers or hmis as well

sorry not controllers hmis in destroyer one uh is now called in destroyer one because we have destroyer two that was the blackouts in ukraine in 2015 three hours without power to some two three hundred thousand people that was truly real world impacting a lot of people malware uh and they came back um the very next year and attempted uh something similar that was less successful but was still uh directly impacting control systems 2017 back to saudi arabia attacking safety systems our last line of defense to make sure nothing bad happens well they almost came were taken offline enabling significant attacks and then i don't know there were no major news worthy attacks i'm sure lots

happened between 2017 and 2021 where we had oldsmar the water treatment um and colonial pipeline attacks again obviously major impact but not direct manipulation of the real world and i'll talk about in destroyer 2 and pipe dream here briefly on the next slide so we are seeing more and more attacks that are i don't want to use the word advanced but they are more advanced and they are having they are more and more um targeted at the real world at having those real world impacts the new new folks on the scene in destroyer turn in destroyer 2 also from sand worm who are behind several of these attacks it was used against a ukrainian energy

provider and it was talking iec 104 which is an industrial protocol used to communicate with controllers in primarily electric um in the electric sector and there were some additional pieces like wipers that came along with it but again um getting more and more sophisticated able to actually talk um ics protocols there's some modularity it could be used it wasn't set up that way but can be used with other protocols as well and then we saw pipe dream which takes this even further was discovered prior to deployment so it hasn't been used against anybody that we know of but it was tailored to impact specific uh controllers specific um servers and opc ua is a protocol that is a

multi-vendor multi-environment it can be used to impact um a whole bunch of different environments and it had all kinds of capabilities was deemed a swiss army knife the good news was it was discovered before it could be deployed which allowed us to get indicators of compromise and and some awareness out there to potentially mitigate an attack all of this goes just to say that we are seeing significant attacks against ics and we'll talk a little bit about what that means to us in a minute how are attackers going about doing this i talked about a few slides ago you know being able to influence those sensors those actuators those controllers and so on but it's not that simple

and and i think that's something we sometimes lose sight of there is um sans put forth an ics kill chain this was quite a few years ago now uh obviously loosely based on the the lockheed martin kill chain but ics networks are different there's often very very often two stages the attacker has to get to the point where they have access to an ics network and that may happen many many ways but often that is by coming in through the internet or through fishing etc into a corporate a business network and escalating privileges moving around the business network um exploiting vulnerabilities installing c2 and getting to the point where they can commence an attack against likely the

ics boundary or the edge of the ics network if there's some segmentation there and then get into the ics network and learn about what's there at that point uh i'll i'll get on to the the next slide but you've got an initial point where you just get to the ics network there's a lot of work that just happens at that level just to get there that's what we call stage one of the ics kill chain it allows the attacker to get to the point where they can uh perform reconnaissance against the ics environment now to develop the attack against that i just i'll i'll bring this up quite a few times uh throughout this this talk

but that it ot boundary in stage one the attackers are gonna cross that boundary into ot or just get right up and through that boundary often very often from the business network sometimes directly from the internet although we hope not but remote access and business to business partner connections vendor connections those are a huge source of access for for malicious actors that that's how they're going to initially get across that boundary into ot and that's going to speak to some of the things we want to do to protect against them in the future now what this means is attackers tend to land on familiar systems on windows systems because that's what we tend to have at the boundary

uh and attackers like this because they're familiar they're well understood we have a ton of windows attacker tradecraft out there and the attackers they don't need a deep understanding of process control if they can just click buttons on a windows system right in destroyer the first one in 2015 ultimately the attackers were controlling a window system with remote desktop assistance no less and pushing buttons to shut things down like clicking things on the screen that didn't require any sophisticated knowledge of control systems really i mean understanding that you're on a system that can open breakers and cause an outage sure but that was it so this is this is going to start to inform our

defense at least the initial layers of our defense so an attacker lands probably on this windows system at the itot boundary what's next well now they're going to perform reconnaissance in the ics network so they're going to want to maintain access to the extent they've got it but now figure out what's there on that ics network and develop a target specific attack if you do you spend any time in ics you'll know that yeah we do a lot of the same sorts of things from organization to organization and yes there are common vendors and common tools but every single environment is different so an attacker has to develop some target specific knowledge and capability and they probably are going to want to

build that in a lab and test it offline and validate i mean we're going for real world impacts here you don't just want to you know throw a hail mary and hope it works once it's tested and validated then they have to deliver that payload to the ics network maybe over the existing command and control through the existing foothold but that payload has to get there and then be executed and then if everything lines up then the bad thing happens so this means something else for us if we can stop that attack anytime before this point basically before the delivery even the delivery maybe we could argue is is okay but before the installation and execution of the attack

we're actually winning so we need to keep that in mind from at least a starting point on the defensive side right we want to stop them anywhere in this this range obviously as soon as we can or as early as we can but we've got a lot of opportunities here to detect and stop attackers and and that's what we need to be thinking about we win if we stop them before that line and remember again most attacks i told you i keep saying it originate on the business network or the internet yes physical access usb bunch of other risks yes but vast majority come through that business network this helps us inform what we want to do

for defense before we get to defense though let's talk about our risks what are we trying to prevent well we kind of talked about it we don't want stuff to blow up but we have our standard business risks and these still apply we don't want to lose access to business applications that make the business work we don't want ransomware data corruption loss that kind of thing and we definitely don't want brand or reputational impact those are things that can happen whether we're talking about an ics attack or not but in an ics world we have much more critical certainly impactful things that can happen as well if we lose visibility or control the process again bad things

could happen things can um explode leak burn whatever whatever it is obviously a manipulation of the physical process could cause those same things to happen could cause uh risk to to people to environment to both and of course if we're talking about ics unless we're talking about kind of government operated or um you know like i think water treatment for example it's not a profit center for for a municipality for example it but it is absolutely critical but most in in the corporate world most industrial control systems they are the lifeblood of the company they're the economic engine of that company and therefore any loss of production or operations can have massive financial impacts as well to the point where a

company can easily go out of business so these are risks these are big stakes that we're trying to mitigate or protect against what are common exposures i've touched on a lot of these that business to ics connectivity that itot boundary right there that is one of our biggest risks our biggest areas of exposure as i've been saying but remote access and vendors the nature of these systems is they're old we might not have people who understand how they work anymore they could be decades old they're very specialized in a lot of cases vendors don't always like to share information about them so there's a ton of remote access we want to be able to operate them 24 7 so just for that

reason a ton of remote access connections to vendors as we know bad things can come across those connections that's a huge exposure as well internet connectivity whether authorized or unauthorized we want to minimize it obviously but sometimes it if there's unauthorized internet that's a huge exposure and once an attacker has access a lack of segmentation an inability to contain any sort of [Music] pivoting or movement through the network by an attacker and of course if we don't know that something bad is happening well that's bad in and of itself right insufficient monitoring we don't tend to have it or cyber security monitoring in ot networks to the extent we do in it networks anyways we might monitor the process but not

necessarily the technology around it these are the risks we're trying to mitigate so let's raise the bar a little bit that's that's why we're here right let's start with the sliding scale of cyber security this there are a lot of things that we can do to protect ourselves and we start on the left here with architecture right now you're going to tell me well we've got an environment that's been operating for 40 50 60 years who knows how long maybe not that long but some of them are it's we're a little past architecture yeah maybe you are but that's still something we need to um that's that is still where we need to start it's just we can't rip everything

out and start over all at once it we have to work our way backwards obviously architecture is fundamental if we architect our environments appropriately it makes all the rest of the security much much more straightforward to implement passive defense is where we are at uh this is where you know pro um stereotypical cyber security looks like right this is systems added to the architecture to provide reliable defense or insight against threats without consistent human interaction humans might set it up but we're not going to be there you know interacting on an ongoing basis at the passive defense level this is all of the tools you can go the toys you can go to market and buy right

start with firewalls just as an example right any of those toys you can buy plug in and magic happens hopefully but no really even if you know usually we're talking about humans configuring them um that's our passive defense we buy a firewall we put it in place we segment the network that's part of our architecture and we put firewall policy in place to control access and as long as we don't screw it up that defense is there the firewall's there and it's continuing to provide that defense without human interaction as we mature we get to active defense which is the real people part where people are monitoring responding and learning from the adversaries it's more

of a dynamic you know um situation where where we can actually detect and observe and see what's going on and react to that and it doesn't just stop at the detection it's going back and um interacting with the environment to make it more resilient to get rid of the attacker to get them out to section them off and so on and as we mature we can also augment that capability with intelligence that is ideally specific to us or our industry and then offense not hacking back but legal countermeasures self-defense actions that could be involving law enforcement that could be doing something for attribution to figure out where the attacker is so that you can do something in the real world

or it's these are the the counter measures that you are taking to protect yourself it's not it's still protecting yourself it's self-defense it's not hacking back let's look at this a different way bang for the buck everything in cyber security has to do in some way shape or form with bang for the buck right at the bottom of the blue pyramid anyways we have architecture lots of value in architecture done well and done well and early relatively little cost it doesn't cost a lot of money to draw fancy pictures or not so fancy use your napkins and then again if we're starting from that point implement technology or tools to um align with that architecture

and even passive defense there is a cost in purchasing all of these fancy tools and blinky lights and so on and so forth but again we get significant value compared to the cost of that passive defense and as we move up the value chain here though active defense i mean there's still absolutely value there but not quite as much we don't get the same bang for the buck our cost is increased for slightly less value that doesn't mean we don't do it it's just let's start with the easy things and build up from there right and of course intelligence and offense have lesser value as we move up and higher costs to do well and to do right so we

want to focus on i'll be talking today about the bottom two levels of the value proposition here that's where we can have the biggest impact and we can have an impact in an ics environment no matter how difficult it is to get a toehold in there for as a defender there are ways we can have impact for sure so what should you do this list actually came from the drago's urine review uh report and i i thought it did a really good job of bringing up to a high level and summarizing um the uh steps that you should be taking as a defender and the first one shocker is building a defensible architecture so keyword there

is defensible though we this is the point with architecture this is where you get the value if you can architect for example your networks with strong segmentation then that gives you the ability to control communication and that segmentation also gives you choke points where you can monitor so that defensible architecture it's implementing your networks and your systems in such a way that it's easy to apply additional security controls right network monitoring is key especially network monitoring we have a whole variety of systems in ics a lot of them old a lot of them turn key we may not you know they may be commodity operating systems we may not have access to them they may not be

commodity operating systems uh they need to work in a deterministic fashion so we may not be able to borrow cpu cycles to do anti-malware things that leaves us with the network where we can have the biggest impact so we absolutely need that network monitoring because we're not going to be able to touch all of the endpoints in an ics environment no matter how much we try control of remote access and authentication that's kind of two two pieces because we care about authentication throughout not just for remote access but obviously the all of the contractors and vendors and staff who are coming in via remote access that's a huge exposure that we need to really control

authenticate and authorize appropriately we want to manage key vulnerabilities and that word key is important vulnerability management is a a tough conversation in most worlds i.t let alone ot so we need to be selective but we absolutely do need to manage some vulnerabilities and know what's going to happen when something goes wrong when we're under attack or you know maybe it's not malicious but something's gone wrong a lot of ics environments have pretty robust plans for when something goes wrong in the real world you know a leak a fire uh whatever right and they think they're prepared but very often these safety assessments and the incident plans and so on don't take cyber into account number one they don't account

for the fact that cyber could cause the incident i keep saying cyber i'm trying to stop but that they could cause the problem in the first place but also that um the the electronic systems can be a part of the solution or can be the thing that's actually impacted and that we need to recover there's so much there that isn't necessarily taken into account uh in a lot of instant response plans so you absolutely need some capability to gather information to figure out what's there to contain the damage to respond and and well respond to an incident these are the top five things uh five categories of things to keep in mind this is the starting point for ics

security for everyone this is where we need to begin but but but i can't i've been here for 40 years and nothing has gone wrong yet why do i need to change anything now well i'm making a trillion dollars every second i can't stop now for you to do some security stuff i'm short staffed do you know what it's like getting people to work come work at the top of the mountain and up north and the snow and rain and sleet and whatever i don't have the people they don't have the time they have other things to do they're busy making me money we don't have time for security i guess that's the other priorities as well we need to

make more money do more of the thing manufacture more of the thing whatever it is um yeah you know we can put that in we could do that security thing you want to do but how do we know a year from now it's not just going to blow up and cause a major outage um well you know this we've along the lines of the it's been operating for 40 years well what tools do you have that that can work with my 40 year old stuff right or maybe you can you've got some great capabilities over here but guess what this is a plant it's not just one system it's a system of systems of systems from

multiple vendors installed and managed by multiple different people and if you touch one thing you touch them all and we just can't do that or or maybe the vendor says you're not going to touch my stuff we got this we got this you're not you're not authorized to make changes we're not going to support you if you do anything to change our default deployment which of course is deployed for safety and operations and reliability and making money not for cyber security so there are you know if you uh go out and talk to an engineer and operator there's lots of reasons you can't do anything that this list is is actually a short one and this is what we have to overcome

that's part of what we have to do so what can you do you know what i know it's it's it's uh a bit trite but it's a marathon it's not a sprint right let's get better let's let's do what we can to get better everything in ics is planning for the future right on the extreme end you think nuclear you may be in planning and design for 10 to 20 years before a nuclear reactor facility goes live so it's okay to plan for five years from now i'd like to see some incremental improvement along the way but it's okay to take that gradual approach and plan for the future architecture i didn't put the air quotes

in there is free you can certainly start again building those best case templates and so on and so forth knowing what you want it to look like in the future so you can start building towards that virtualization is giving us a lot more tools to reduce impact to reduce interruption to reduce downtime to recover if something does go wrong snapshots and so on vendors are being forced and are choosing to come to the table we just have to get on their latest and greatest or newer stuff you've got new initiatives there's always an acquisition or a new plant or a new well or a new city or a new substation or renew something start there right

it's it's something and use that as your model and move from there the c-suite is certainly much more aware of cyber risk to ics uh so there's there's you know it may be misdirected at times it may not it may be hard to actualize and to realize in the real world but there's certainly more buy-in from the c-suite there's more visibility and you know what we don't have to touch the process if you really dig down there's lots of stuff that's critical don't get me wrong but it's still this stuff over here that's critical this is the stuff that is cranking out the electricity or whatever it is on a 24 7 basis this stuff over here it

helps us it helps us charge people money it helps us monitor it helps us see as human beings what's going on it helps helps us um track efficiency and tune and so on but it's not part of the process directly um maybe we can start there and start securing at the boundary because remember i made a big deal out of this the attacker is coming in through the boundary your core process is not at the boundary so let's start there and of course visibility monitoring in and of itself is not disruptive so start use that monitoring to get your asset inventory to figure out what you're dealing with and go from there so i kind of spoiled some of this but

here's where we start architecture right high value low cost let's like anything in it and especially cyber security what does good look like all i want you to do oh and i didn't i forgot i was on this slide before we start here this i'll get back to this we need relationships this is the thing i forgot um before we get to architecture although not really before we can do it in parallel but the one thing that you will not the you will not succeed at bringing any security to an industrial control environment if you don't have relationships with the the people on the ground so to speak with the operators with the engineers with the technicians

if you don't have a good working relationship you will not succeed i had my best success when we spent over a year just helping out in the ot environment not doing security just helping make things better improve technology figure out vmware and storage and so on and learning and meeting people and making relationships and understanding the business drivers and the requirements along the way that's where we start i i even jumped jumped the shark here a bit and got ahead of myself we need to start with the people side of things if we can't speak to why it's important but why we're not going to screw things up we're not going to cause an outage

you might actually be more safe as a result of the work we're doing we need to do this in a nice you know in a collaborative way we can't just come in and preach the gospel right so relationships if you don't get those in place you can't start there so where to start figure out what good looks like again any anytime i talk to anybody about anything in cyber security you know what let's set a target what is our goal where do we want to end up and let's let's start with the easy or gradually how about every decision you make from now on you look at it in the context of where you want to be in three to five years

and you try not to move further away even maintaining the status quo is kind of a win let's try to move a little bit closer though right so let's develop those templates for where we want to be so we can articulate it so when we we bring a new site online we can use that template and we can at least get that site looking the way we want it to so that when we have scheduled maintenance or are deploying new systems or whatever we can we can align with what good looks like and get there over time it's a long game so the new sites the site refresh and again start at the itot internet remote access

vendor connections boundary those systems there i mean everyone will jump up and down and tell you how critical they are and they are important systems but a screw up there is not the same thing as a screw up at the process level right we can start there we have commodity operating systems mostly windows and traditional networky stuff at this layer let's start with that and then move on from there that's that's architecture some of the things to consider hey how about we actually talk to people and figure out what the requirements are both for security and for the business you know we might know okay we produce electricity or oil and gas or widgets or

whatever it is so the business driver should be obvious but they aren't always right what understanding what drives the business what's important to them you know it's it's a little bit of a cliche as well but what that really bad day looks like because you can't protect at all so what are the most important things to protect you know if we show up on site and safety systems are on the network well we start there they should never be on the network those are your fail-safe for when things go bad on the network like like that is number one right safety business operations figure out what the requirements are and then i'll talk about labels a bit more here

slap a virtual i don't mean slap a label on everything slap a label on everything virtually know what it is what it's for what its requirements are you know for for availability for access for security for operation all of that know what you have right first couple critical controls it's just another way of saying it know what you have out there what it's for group them together so that you can deal with them in a priority fashion in groups build out that or sorry you're architecting at this point so design that strong itot boundary with dmz with remote access including vendor connections partner connections all of that remote type stuff file transfer as well a secure way to get files in and

out segment your network so you've grouped everything so based on those groupings and labels you want to architect your network such that things that are grouped together are on a common network but that we've got a segment a boundary um enforcement of communication or flow of communication between those different segments so we at least have the opportunity to introduce some level of control and in doing all this it kind of happens naturally but ensure that you have a way to get visibility whether that's with a tap a span port net flow all of the above firewall logs audit logs ideally all of it so make sure you you have a way to get visibility away to

monitor and a way to respond a way to act you've architected an environment where you can see what's going on and you can act when you need to but architecture only gets us so far before i move on to um the the passive defense this is the based on the purdue model essentially this is the purdue model we've got our level zero through level five systems we've got our enterprise dmz at the top safety at the bottom i'm not going to spend a ton of time going through this but i just put this in here to reiterate this is what i mean by labeling i can tell you i can sit here and talk to this for a while and i'll come back

to it later you know here's where i want firewalls enforcement boundaries segmentation segregation here's what dmz's need to look like i can't do any of this unless i know what my level 1 systems are my what are my dcs and rtu and plcs what are my aggregate historians what are my local historians et cetera et cetera this is again back to the inventory and labeling until i know what these things are i can't i can't segment them i can't architect a network to protect them the purdue model gives us great guidance just for labeling that's all it is in my opinion is a way to slap these labels on sensors and actuators oh level zero rtu

plc dcs level one communications gateways local historians alarm servers hmis level two a lot of the other stuff level three chunked into different chunks and if i have my acid inventory i have my visibility and i've put level labels on everything now i can start to build firewall rules and and written policy that said these this is the kind of authentication or this is the kind of backup or virtualization or whatever it is um that's that's why i'm harping on the the inventory and the labeling because once i have that then i can draw generic pictures and i can build those templates to tell me how to protect the different classes of devices some key tenets from an architecture

perspective we do not want internet in our ot network there are very specific exceptions where maybe we need to download something on a regular basis well then we should have a proxy server that allows that and nothing but that if you need email internet web browsing other things then have a dedicated business computer i've been talking a lot about the enforcement boundaries so i won't won't spend too much time on this but we want separate authentication we don't want to be shared with the corporate network including mfa for all access into ics we need to provide a mechanism to transfer files securely without malware it kind of goes without saying but let's try a deny by default and permit what we

need and these enforcement boundaries are an ideal point to monitor and log communication in both directions use ad as much as it parts of me kind of cringe when i say this we have a lot of windows in ics networks leverage in active directory not your corporate active directory completely disconnected and not managed by an engineer managed by someone who understands a.d but it gives us group policy it gives us authentication a lot of security controls that we can leverage and then manage the antivirus patching all of that stuff on the ot network separate from the business network that brings us to passive defense these are the the five categories they're they're kind of somewhat tied to

the the five things you should do that i mentioned earlier of passive defense that we um i'm going to talk about anyways enforcement boundaries again i'll move on authentication and authorization though is a huge part of our passive defense protecting our endpoints to the extent we can getting our visibility or monitoring our logging and then managing those vulnerabilities patching i will get there that's always a fun conversation so the enforcement boundary it's generally easier as i mentioned to apply security at the network layer start with that itot dmz boundary segment the ot network further though you know put your active directory systems your jump servers your security related systems your antivirus systems uh your your engineering workstations

versus your operator workstations whatever group them together on dedicated networks so you have at least an ability to um control access right even if you're not today at least segment so that you can get there in the future and then enforce that policy at the network level we want also to have a minor enforcement boundary between levels two and three and sometimes within level 3 to help us contain access to stop the pivoting and contain a compromise and as i mentioned manage that remote access and file transfer so popping this up one more time what i'm getting at is we want this red line here is a minor enforcement boundary it might be an access list on a router or something

but i'm separating level two from level three where i have the ability to and i'll come back to that on the next slide maybe i take my engineering workstation my jump servers active directory all the things i talked about and i put these on different segments within level three so that i can have finer drain control of communication back and forth obviously around the dmz possibly multiple dmz so it's one of our primary goals of this dmz without going too far into the weeds is so that we can't get a single bi-directional connection like command and control if you think about command and control we want to prevent that so if you want to move data

from level 4 down it goes through this dmz terminated on a system here we want to move it up completely separate dmz so now your ability to have bi-directional communication is is minimized it's made much harder so we want these enforcement boundaries at multiple points here and this is a slightly different view if you look at a wide area network i might have a plant network some remote substations well pads whatever they might be i also want the ability if something bad happens here i want to prevent that from impacting the rest of my network so by having an enforcement boundary between levels two and three i can then isolate this um this site and

prevent it from being impacted by other sites or impacting other sites i can cut off or if i have an i.t you know a compromise coming in from the business network at the it level well i can cut off below level three or or cut off the field let them operate on their own but again containing the damage continuing to operate in a minimum viable state right we don't want to run in that state all the time but if we have a minimum viable state and we have a process and a way to reduce our footprint to that in order to contain something bad that's perfectly fine and that's something we want the capability to do

all right enough about enforcement boundaries authentication authorization i talked about a dedicated ad and leverage that centralized authentication ldap radius tacacs whatever it is so that we don't because we have lots of contractors we have lots of transient people coming in and out of ics networks local accounts shared accounts shared passwords aren't ideal those passwords go spread far and wide we centralize one we get logs of every action people take because or every login at least because it's centralized and provisioning and de-provisioning are so much easier we want dedicated ot accounts but we want those tied into corporate provisioning so that if someone leaves something actually gets done about it privileged access in ot networks is just

as important as in it networks i've seen many an ot domain where every single operator was a domain admin we want to control those kinds of things we want network level controls so you cannot connect to management interfaces unless you're supposed to as i mentioned limit the generic and shared accounts and secure that remote access so i i have a whole different webcast and blog post on secure remote access but multi-factor dedicated credentials minimum access separate people by roles there's a whole bunch of stuff that we can do there endpoints where we can again close to that itot boundary where windows proliferates we want to leverage active directory group policy for those security controls have a secure baseline

so that we can pull a device off the shelf and see what's changed so we can do instant response we know what good looks like again back to the new site new location new deployment green field type scenario how about part you know these are static environments partner with the vendor and lock these systems down as much as you can before they go live before they're deployed get that secure baseline application control is absolutely doable in that context app locker for example work with the vendor it's a static-ish environment so get that done where you can certainly use anti-malware where it's permitted impossible endpoint firewall so i talked at ad nauseum about segmentation if you can't do it at the

network layer do it at the endpoint layer everything should be fairly static and you can validate that during your site acceptance or pre-deployment testing right there's a lot you can do there and of course visibility monitoring logging passive monitoring that's where we start we're not talking about scanning you can do it very very very carefully um ideally don't but sometimes there's there are legitimate reasons to be scanned but just look at what's coming by on the network we can tell a lot from that so set up your enforcement boundaries your checkpoints so you can get that visibility it'll build your asset inventory it'll tell you about vulnerabilities in many cases it'll tell you um who's talking to who help you

build that those network segments your segmentation model and your firewall model maybe you'll find the vulnerabilities and improve that login maturity logging in ot is almost always much worse than in it and the way i like to look at this is levels of maturity for logging i won't again go into massive detail but let's call xero no login that's the worst one would be i've got logs but they're all over their place they're on local devices you know i don't do anything with them but they're at least logging you know what great at least i can show up and get access to logs and see what happened as opposed to not that's that to me is awesome you

know seven days worth of logs is way better than nothing now we would like it well configured to log appropriate things for instant response that's another level of maturity and then we'd actually like to centralize those logs to make our lives easier and you know what if we're going to centralize them we kind of want them in something like a sim or a database whatever a searchable place and you know maybe we want some use cases and and um automatic alerting against that and then people actioning those alerts and looking at the logs and then correlating between multiple sources you see there's many levels there honestly get logging enabled and maybe collect it centrally i'm going to be ecstatic in an

ics environment and i mean logging from the process logging from controllers administrative access and your traditional windows firewall networking type remote access logs got a few minutes left vulnerabilities in patching i can't do this justice in in a few minutes but first thing to recognize is not all vulnerabilities need to be patched we are dealing with environments where controllers do take action with zero authentication in that environment it doesn't really matter if it's vulnerable if it will do anything i tell it to do so let's start with not worrying about that stuff okay and then let's also recognize that when it comes down to the actual control systems two things one they're going to be really hard to patch uh it takes more

work outages diligence etcetera testing validation and two with exceptions obviously to get to those systems coming back to my attack chain um the kill chain sorry and landing just inside the itot boundary on those window systems you're coming through the window systems to get to those uh you're coming through the level three systems basically to get to the lower level systems to attack them so let's start by patching and and locking things down at that level three level three and a half itot boundary dmz it's also much easier to patch windows systems i was talking to a customer you know about their patching and and how they keep things up to date and you know they had some understandable gaps

where they couldn't bring systems down but their domain controllers had never ever been patched like there's different categories of systems i get it when you can't patch certain things uh or certainly not as much as you like but your dc's you've got multiples like this stuff that is the keys to your kingdom that kind of stuff needs to be done you've got redundancy patch one half then the other half you've got virtualization again i'm not talking about the process itself necessarily but patch if there's a problem roll back and and i'm not saying it's easy so prioritize those things like the domain controllers the jump servers the the stuff around the edges that's where we

need to focus i won't spend too much time on this because i want to leave a few minutes for questions um but here is a an approach to patching right first does a patch exist does it affect me no don't care is it an emergency patch so we need to categorize it yes okay um if not we're not dealing with it right now it's an emergency patch does our operational need outweigh the risk in other words does the impact to the business from the act of patching or from the act of having patched and causing an outage further down the line does that outweigh the risk of the vulnerability being exploited it's a decision we need to make if operational

needs are greater then again we're not going to patch right now if operational needs are not greater we think the risk is significant look for an alternate mitigation if that exists well mitigate and and and exit only if all only if we go all the way through to the end here do we patch immediately and for many organizations there there are caveats to that patch immediately but we're going to try and do something and then of course if we didn't then we patched during the next window either way we document and continue right it's just a way to think about the patching process um it's very much not an all or nothing thing but it's also very much not a

nothing thing we need to do something manage our risk as best we can so patching is important to a point vulnerabilities are important to a point it's something we have to be aware with and deal with um sometimes that might be this next window might be a year from now and that may suck but it may be okay and that's when we focus our efforts at the boundary so that nobody can get there in the first place so to wrap up we want to strive for that active defense i i purposely stopped short of active defense here i kind of see architecture and passive defense as our minimum viable uh for for defense and we really want to

get start getting into that active defense which is the repeated actions for ics defense analysts monitoring for responding to and learning from adversaries internal to the network so it's humans acting on threats that can be through threat intel consumption getting visibility detecting a threat responding and then changing things to um protect ourselves just to to recap again we've got the sliding scale right let's focus on these two here start with our architecture and our passive defense there's a ton of work to do here and we can do it overcome those objections build those relationships start small help where you can and even if it's not to do with security build the relationships and small baby

steps gradually get better and strive to at least get to some of this middle territory and see if you can get beyond that different view bang for the buck again just reinforcing that this is where we have our bang for the buck so we want to focus on that architecture and it's a long game you know we're going to do the architecture um so that we can be better positioned a year two years three years from now that's okay right the worst thing we can do is just say it's hard it's risky i can't take an outage i'm not going to do anything we need to start planning we need to start building that mindshare building that

momentum to do something to get better that's that's where the bar is at right now when i talk about raising the bar the bar it's kind of down here i just wanna let's get the bar off the ground right let's start there and and no offense to anybody who has done that already but by and large our ics security bar is pretty close to the ground no one's limboing under that thing just final slide to leave you with these five core defenses or things you should be areas of work you should be focusing on from a cyber security perspective defensible architecture we want to build a network that we can defend to do that we need monitoring we need to

be able to see what's going on on that network remote access um access control authentication that boundary that entrance point for everybody absolutely something we need to control key vulnerabilities the ones that can really hurt you we have to prioritize we can't do everything so let's manage them and it's going to happen be ready to respond from an i.t from a cyber from a cyber security perspective and absolutely integrate that with your business continuity and disaster and your your incident command structure and all of that but recognize that the i t or cyber world can be the cause of an outage and um its loss can also in and of itself it can cause you an

outage so you may have to do instant response but it may also be integral to responding to a different kind of incident in a different area so all of that or related incident all of that is important from an incident response perspective so i left uh two minutes for questions apologies not quite there but if anyone has any questions please let me know put them in the chat you can find me on linkedin and other places and i will share these slides as well so i hope you enjoyed that and i hope uh to see you out there lifting that bar and hopefully the weights on the end of that bar are not too heavy

and yes don't forget the relationships thank you very much for that comment um absolutely where you have to start if you don't have friends you're not getting anywhere fast

yeah patching is not the only answer there are key patches we need to get in quite often domain controllers as an example but uh yeah it's an insecure by design environment patching doesn't help with that other things help with that so let's be selective use our energy wisely

honestly when so i would the operators demanding testing in a like environment so i ask them where is your like environment because you're making changes all the time and then they scowl at me and we agree to disagree and whatever but putting that aside during there are scheduled outages there's scheduled maintenance and we need to make security part of those schedule maintenance um outage windows because at the end of those windows usually they have to go through site acceptance testing again so that's the time where we can really do a lot of these things is is during especially the lower levels during those regularly scheduled um windows or pre-deployment partner with the vendor there's nothing wrong with

saying the vendor hey you know what we're going to do a bunch of security work for you that you can reuse with other customers as long as you're willing to work with us and build this out and and land in a better place but it almost exclusively has to be done pre-deployment so focus on those brand new sites or those those site refreshes that's we've we can do enough work in those areas um that we can make that improvement over time without having to worry about the sites that we can't touch today that'd be my take it's not perfect but you know there's sites you can't touch today and that's that's going to be the

way it is until you've proven the solution in other locations

all right thank you everyone i'll all hang out to see if there's any uh other comments just in general in the chat and uh hope you enjoyed that no like i said see out there raising the bar

thank you all for coming forgot to say that so thank you

you