Mackenize Morris - Building a securable architecture for interconnected ICS environments

all right so my talk is named building a secure bill architecture for ICS environments it should really just you've been changed by now but it's because I redesigned the talk halfway through but let's jump into it so a little bit about me I a PhD dropout in chemical engineering I was on the Team USA 2012 modern pentathlon and I have a dog named QWERTY and I pretty much do presentations at conferences as an excuse to show pictures of my dog to large groups of people but let's jump into ICS so super easy this is supposed to be a 30 minute talk but you can just do this right here it solves all your problems so you just

take you just take your workstations and hook them up to the cloud you take your servers you put them in the cloud and then you get IOT connected PLC's and web devices and then you just do everything in the cloud and it works perfectly fine and there's no problems there has never ever been an issue with this but but this is not a firewall and five firewalls are not air gaps slice to say there are people who believe that putting a firewall is the same thing as an air gap it is not and I don't know how people got this belief but it's something that I fight with constantly in the field so we're gonna talk a

little bit about the Purdue de model I'm sure not everyone is an ICS engineer or spends all their time in a control environment but the Purdue model of Purdue framework is pretty much the working thought process on how to configure ICS environments and that's wonderful because it's one of the most complicated things ever so this is the original Purdue model and this isn't very descriptive at all so you have for the level 4 which is your business environment so that's going to be your IT infrastructure what people you regularly use their internet connected computers you have your level 3 which is gonna be your manufacturing plant so a lot of your engineering and operation workstations your control room and

basically everything is revolving around a manufacturing facility and then the profits level to 0 and 1 to 1 and 0 on the same level no sense because they're not very descriptive at all and these you'd think like that these three things are level two through zero but they're not they're just three different types of control mechanisms so that's from that's the original mystery framework this is from is a they have a level five in their purview to model so that's great so we're changing it their level five is part of their enterprise zone and and they include a DMZ as well so we're getting a little bit more complicated but they do have a little bit more

features so they talk about the level zero which is gonna be your actuators your sensors for any of you that don't know actually this is one of my favorite facts but the way that a thermocouple like takes the temperature reading is that it converts a temperature reading into a milli amp bolt so they'll have like five milliamps of 20 milliamps and then the temperature range of the thermocouple will be converted into amperage for you to read and that's actually how you go from like a physical process to data on a computer so that goes into the level one which is gonna be some of your basic control there's some of your PCB cards and like

low-level logic level twos gonna be a little bit more PLC's controllers PCMs and some operations maybe an HMI or a workstation layer three is gonna be all of your engineers working in your control room and then the dmz is where you're supposed to put your servers apparently according to this model and then layer four is your business network this one has letters in it so now we have letters on the side so we have a through K layer this level five is external facing so the external internet and they have a level zero so we're getting a little bit more complicated it's still the Purdue model apparently according according to control global and then this one has an A and has an

aimed b-side so they have a Purdue model a and a Purdue model B on the other side of their model and this is from cute sir so it's okay yet there hasn't been a lot of conciseness and the way that you're supposed to build an ICS architecture and that's just because everyone sort of does it their own way and they haven't really stapled out like is how you do things so one size doesn't fit all every ICS environment is unique and you have to sort of build your environment to fit for needs as a production facility so I wanted to originally give this talk and talk like this is the best ICS environment that you can build and

it's cookie cutter and if you just do this over and over again and that's just in insane philosophy I'm like there's no way that anyone could ever do that so we rarely have an opportunity to green fill the project so greenfield means that you build it from scratch so I'm starting with a Greenfield I think I'm building an entire control plant that is the best time to ever put security controls and do an ICS Network because you can do it right from the beginning and you can get into the design process unfortunately most of us are walked into a facility that's been there since 1940 or 1820 and say here's a control system that we built it runs

like ms-dos and windows 3 and you have to now put cybersecurity controls on it and it's really hard to insert taps into a network when you're not allowed to touch any of the switches so and then the last thing is that vendors and managers can sometimes be obstacles because if you you know are trying to sell them an idea or product they might not be on the same page as you and then vendors which make ICS wonderful will demand things from you like direct remote access to their PLC's if GE actually is very guilty of this all GE turbines GE requires that they can remote into your turbine which is insane but if you want a GE turbine you have to

give them that capability so sometimes that can be difficult and the way to get around that sort of just pressing on them you know really asking your vendor to like build a securable VPN or at least bring them in through some sort of firewall or the very least you know put some sort of network sniffing or like a security onion instance on that connection to connect they're not connect so we air-gap a lot of our things now air gaps aren't exactly the biggest security control there's actually a wonderful professor in Israel who his entire PhD focus is how to get past and like the seven or eight different ways that you can do it so there's a

couple ways that you we can connect an ICS system so we can say completely isolated so every single process every single box is just there by itself with no Nick's in it but one of the issues with that is you don't get any visibility into your network and now everything's out there and you can't even see it so if anything you know rides in on a USB Drive you're not gonna get any visibility to that box being compromised until it shuts down one day or we could connect everything and then now it's all internet give everything an external facing IP address and then hook it up to the Internet so that's not really what you want to do either so

there's like a nice little sweet point in the middle where we connect part of our things and then we get you know good visibility but we don't excessively compromise our own systems and in order to do that we've got to like prepare a tool box so rather than say that this is the best way that you could build an ICS network instead we can sort of talk about some of the things you can do while deploying a nice like an environment or things that you can add into your network sort of hot-swap for what you currently have and you know so sores that keep the same security so tools in a toolbox that you can deploy

when you need to deploy them and then per your own environment when you have business needs you can sort of you know reach back and pull out the ideas and then construct them like Legos to build the house that you want to rather than the house that I'm trying to tell you to build so cybersecurity is part of running a chemical plant the other part of running a chemical plant the fact that it makes chemicals so often cybersecurity gets in the way of the actual process of a chemical plant so they're number one I'd focus every day is to produce the most amount of chemicals as efficiently if possible to sell them because that's how they make

all their money chemical plants don't make money running cyber security so you're almost always going to fall behind operations in any sort of instance you're always gonna be second and anytime you want an outage to sort of get in there and start messing with computers they'll always push you back if they haven't scheduled to me and this is one of the biggest challenges is sort of getting in manage both manager buy-in to like support your team but also working with those operators who are just trying to get the job done and trying to explain to them that having a network that works is kind of really important to getting the job done because if all the computers shut off

then we can't run the plant anymore so some toolbox we have VLANs these are useful they don't really add a lot of security but what they do allow is for you to baseline your environment VLANs are pretty easy to hop around and bypass but if you segment your network or specific processes into each VLAN it gives you a really good baseline for what traffic looks and each sort of network segmentation VLANs are example of segmentation but another example is just subnetting or silos so the best way that I found in a process control environment is that certain computers will all be related to a single process so if your plant runs three or four processes you can break your IT

infrastructure into four subnets four VLANs or four areas of control that then I'll have a very predictable network traffic who they talk to what their MAC tables are firewalls firewalls are wonderful especially when you silo and start segmenting your network putting firewalls on top of those networks to allow them not to talk to each other is wonderful because you don't want sort of east-west communication between processes you know we want north-south back to the business network data diodes are wonderful tool I really fell in love with these a while ago basically their network switches that only allow one-way communication so they only allow pushes so you'll send you an agent on one side of that diode

it'll push that traffic through the diode to the end and it physically doesn't allow a connection back my favorite diode is one that actually takes the networking traffic converts it into a laser shoots it through a box to her transceiver and that's physically it's physically impossible to go backwards whitelisting communications this is a really good one so in an ICS network you have the ability to do stuff that you can't do in an IP network IP networks are a lot of BYOD so you can bring your own device you have a lot of changing users and sort of how they interface with the internet you have users just going to you know eBay and Facebook and Amazon all the time

so that's also an issue you should never be doing that in your control network so in a control network if I have 15 controllers and I have 20 workstations and I have four servers you can actually whitelist the individual communication from controllers to their assigned work stations and then from those workstations to their servers and then you can map that all out and you could hard code it into your switches and your firewalls and then any time any controller doesn't communicate properly or in its specific path that it's been given you just send up IDs alerts now this can be pretty hard to do because some PLC's and controllers are very interesting for example some Snider

PLC's they are once when you turn them on and then they are every year at the same time that you turn them on so if you if you spend 6 ty if you spend 6 months learning your process and then you put that into an IDs I map your snagger controller will ARP once and then it will not hit it's a redundant controller because they're tired trying to talk to each other across the network and it will keep our ping your it's redundant controller for every every second until it gets requests back so like you'll come in in the morning and then you look at your IDs and see that you have like 4,000 IDs alerts and you'll think that

the plants blowing up but it's actually your standard controller is trying to art and like buried in the documentation they explain this process so sometimes you can have an issue waitlisting the environment but three months should give you a fairly good idea of what your your environment does and then on top of that some controllers only are that when you turn them on and that's at like ever so in order to get that you have to sort of download all over your Mac tables out of your switches and then put that into your white listing so it's a little bit of a labor-intensive process there's no really good way to automate it but it can be done and then once it's done you

just set it and forget it and then in five years when you have to read or ten years or 30 years when you have to redo your ICS system you just tear it all out and have to do it again integrity checks this is something that I'm really working on right now is downloading ladder logic off of a controller hashing and then doing that every single day to make sure that controller logic doesn't change now when you do updates to a controller you should be going through some sort of rigorous change configuration process with your engineering staff because your engineering and your design engineers and your sign authorities should be checking to see if the change to the

controller will upset the system well you just sort of tie cyber security into that so that when you see that controller change you know that there'll be an integrity check failure and you just tie that back so you make sure people are updating controller ladder logic without your permission ideas hips these are just wonderful regular in networking tools that we use on the IT side all the time you can put them on the networking side so let's look at a typical control topography so you have three processes and they're just completely isolated so they have a domain controller their historian their master server and then there's an HMI some Engineer some engineering workstations to the PLC you have to be

processes this is how most plants run their facilities just three complete three complete systems three completely isolated systems as an engineer or as a cyber security professional to do maintenance on these you have to walk one workstation to one workstation to one workstation if you're really lucky you can work walk one system to one system to one system but sometimes you don't even have that you have to patch each workstation by hand walk around sneakernet all of your windows patches and all your engineering workstations which can make it pretty difficult so you can start to connect things but unfortunately the first deployment of doing things usually looks like this where they get one really big switch and then they just

plug everything into it and plug it into one master server so okay well this is a little bit better I can manage my whole system at once but it doesn't really make any more secure so this is going to be the part where you went too far but we can we can we can start making this better so we could build we can put firewall between our systems so now all of our our level two areas are still connected to each other they can still talk directly from one process into the other process ok well that's ok but but now we can start looking at how the processes talked to the servers and a little more I didn't like a security

onion instance because this was the security onion conference too so I guess we'll just talk about them so well I did some security onion on the layer three and now we have sort of segmentation so we can start looking at how our servers talk to our workstations which is really nice in this firewall if you really want to get in depth could probably do about like 1,700 firewall rules to map each workstation how it's supposed to talk to the servers which will prevent north-south communication pretty well but it won't stop anything laterally once it gets into the environment so we can add a process firewalls fellig's like a ton of firewalls if there's a McAfee rip in the field like out in the

audience they'll be really happy so now we have process firewalls this is great because now you can actually segment or silo off each process so I can it you know process doesn't talk talks to the firewall then talks to the master server history--and it writes this is great some processes require inputs from other processes so I can say you know PLC and process one's supposed to talk to PLC in process 3 and then only allow that connection and then then have that on both firewalls this is preferred during segmentation is to sort of label out each process and build it into a little box and then these boxes can sort of just like fluid around this is kind of

what networks start to look like now so we want to add in a business historian so not only do we want to historian on our engineering network now we want one on the Business Network because our we have to report to some sort of reporting agency or Authority or the Department of Energy or not nice but Newark has requirements for our power generation or something like that so now what you need to report are is or if you're a chemical plant you have to show that you're producing as many chemicals as you're supposed to so now we have a business historian well now we're gonna put a firewall between layer 4 and layer 3 so

that we're not just sort of directly connected to the business Network now there could have been a a different slide that didn't have a firewall here but for the love of God I hope that's never the instance that you are directly connected from your control network directly in your enterprise which is the number one way that every ICS system gets compromised is that there's a direct connection from their control network into their enterprise network and then their enterprise network is compromised by a phishing email so in just sort of talk like toggles over to the control Network and they starts changing things on a on a controller which is actually how the Ukraine attack happened with Russia they

just it is VNC it in and then they didn't change anything on the control level there was just they just opened up the GUI on the workstation just started clicking buttons on their HMI so that's how attacks happen and you have to segment your pissis network so this is probably if you only had one thing that you could ever do it'd probably be to invest in the layer four layers three firewall if you had to have one so if you are if you have no need or no business need for your business network to remote back into your control network but but you still need to send historian data there if you replace with the data

diode this is great oh you're now your historian on your control network and talk to your business network historian and they can't come back in so you're effectively air-gap in yourself people will say that this isn't an air-gap system because there's a network connection which is true you are sending data one way but now there you have sort of a guarantee that no one's coming back in over a network connection so this is adding in adding in some great some great protections but now they want to put a patch server in your business network okay well now we have to go back to having a firewall because we need back connections into our environment so

we added a firewall now we have to put firewall rules in there so now we're once again could be compromised so so now we need a communication it goes from control network to business network and business network to control network and I don't use a firewall well the answer would be to data diodes data diodes are really easy to configure but they you have to hit the agent on them and it's a little bit safer to have two data diodes and have one firewall and data diodes are expensive but so are really large enterprise firewalls so you can easily break even there but one day to die was configured to only take patch data patch server door

patch server and you stand up another patch server in your control network and you do large data pushes from one to the other and then you still do historian pushes from historian to historian so now the hacker wants to get in he has to send yes it connecting to your enterprise network compromised a box jump to the patch server compromised the patch server push through the data diode agent to the patch server on the bottom side compromised that compromised the historian on your control network and then usually story into re go back through the other data diode agent and then connect connect out using a different path so that you can still be done this isn't a foolproof method but

it you start to see how you can add a lot of Hoops for them to drop jump through and then your hopefully checking your logs and you'd see them sort of try to get through this whole process going back another technique would be to add a DMZ so starting to put your patch server and interpret Orion in a DMZ that then once again protects it our just now I just have a thousand firewalls but what your what you are effectively doing is trying to segment all of your parts of your network into highest levels of layers of trust so you kind of implicitly trust your control network at the bottom because it's really hard to

add any sort of cyber security control to physical copper workers and then you know the next layer up is your your engineers and then above that is your patch server and then you never trust your enterprise users because I mean come on have you seen the people that work in enterprise so unfortunately all of this is mitigated when your vendor comes by and says that they need remote access to your engineering workstation or something like that so you can only get so far and then your vendor comes around and just kind of screws it all up and there's no real way to get around that so waitlisting communication this is this is actually you can actually do

this on any sort of Windows environment right off the bat Windows has built-in application whitelisting it's a pain in the ass to configure do it and these are sort of the the communications that you'll see on the network and then you can just sort of allow those you should never see controllers talking to each other really it's not gonna be common in your environment and then how controllers talk to historians stories to really never depending on how they pull controllers shouldn't really come back down and controllers don't download ladder logic every day so at a low do controller should be like a monthly occurrence usually people aren't working in sometimes controllers haven't been downloaded in years because they're very

happy running the process how it currently is so you need to know what normal like looks like and then you just take that and you just sort of press it and steel and then you just check your fact sheet every day so resources and budget for process the goal of a defense is to budget out a sort of a solution that is adequate for what you're protecting so I wouldn't say that you should build Fort Knox around like a mom-and-pop shop so look at what you need to do if you had one thing that you could do and you had very limited budget it'd be any and you had a business reason to have a firewall between your

enterprise network and your control network if you do invest in a good firewall and then watch that like a hawk and build out all your rules for what that traffic should look like and that's gonna be like the one thing you can do really well the other thing that I would say is that if you have the ability to enable span ports on the switches in your control network when you do in I when you do if you ever have to go through a deeper process which I hope that none of you ever have to do it's gonna be $1,000 thousands of dollars cheaper if you already have the ability to start collecting information even if

you don't do it but you have the capability to start collecting information in your control network it's gonna make your it's gonna make the incident response way easier so last tidbits IDs alerts our custom you're gonna have to sit down and figure out what your network looks like and then you're going to figure out what you don't want in your network there are new controller level firewalls that actually are firewalls between controllers they're pretty interesting they're brand new to the market this market is sort of developing constantly especially since 2015-2016 you're AI seus environments yours you're the one that's going to know it I mean it gets sort of build it out yourself so no normal and then just keep doing that

and make sure nothing changes so this is me if you want to send me an email or ever or connect on Twitter or anything like that or LinkedIn you know feel free I actually alright who's given some giveaways and I think it's ironic that one of the giveaways I have is a USB wireless card which is I hope to god is never going to be in your environment but it might end up yeah you have a question

yes so Wireless is well my personal what I'm gonna say about Wireless is that if you have wireless it's not secure so ZigBee is a great example because ZigBee sells itself as wireless encryption right so do you know how long a ZigBee device lasts how long the battery the xapi device lasts okay so daisy B says that ten years and they say that they do encryption so if you turn on encryption in a ZigBee inva in the ZB device do you know how long it lasts then six months six months it goes from ten years to six months they sell the idea that it's a it's a great wireless low-power communication protocol but if you if you actually add

all of the encryption that they talk about it goes from ten years to six months which kind of defeats the product the purpose of that those devices but they're wireless as a whole that report the the best that I can say about Wireless is that everything that's in your wireless domain you would should consider untrusted and then you should bring it back into a single focal point and then you know manage that focal point only because if it's in the wireless area or if you have a ton of our key use out in a field or a pipeline that's going to be all yeah I guess ability to be compromised

yeah yeah it's a it's a it can be an interesting concept does anyone else have any questions it's 60 70 % of vendors want direct access most vendors will void your warranty if you won't give it to them so and and there are a lot of people who if you have a control system in place you can't cancel that you can't just tell your vendor to go by because those systems usually cost twenty-five thirty million dollars to build you had a question back there

oh this one yeah oh well this is going to be pretty close right so this is gonna be your layer 4 this is gonna be your dmz / well this is gonna be a layer 5 depending on who you talk to this can be your gamzee that's gonna be your layer 3 these things on the left are your layer 2 this is gonna be your layer 1 and then I don't have any layer 0 on this diagram and then your vendor is gonna be seven or eight or something like that alright way I had things but I forgot that I was gonna supposed to give them away so don't have any questions yes yes he is

his name is QWERTY okay do you want a blue team book and then I guess I have a wireless thing that should probably go to the guy that loves Wireless in the back any other questions

yeah that is the best way that I've seen it done is to force the vendor to come in through a jump box or something like that the other way that I've seen it done is that if you have to allow this single connection down is to literally just pull the ethernet cable out of the wall when they don't need it and when they call you you plug it back in which is better so that's about all I got all right well thanks for coming to the talk [Applause]