Vivek Ponnada: Top 20 Secure PLC Coding Practices

[Music]

[Music] hey folks good morning hope everyone is enjoying the conference so far i certainly enjoyed it yesterday so um appreciate you being here and we're going to talk about the top 20 secure plc coding practices or programming practices um let me just uh next slide a little bit about me i've been around the industrial controls world for quite some time been doing ics or ot cyber security for the past eight years uh i'll go into a little bit more detail about what ics and ot is uh just for those people that uh are as familiar but um we'll spend some time on the basics i currently work for a company called ici and we provide various cyber security

solutions specifically for the uh ot community in the past i was with ge um being a service guy being an engineer and a sales and service manager for quite some time and you'll see this quite a bit in the ot world a lot of people work in many countries just because you get a travel it's a smaller community than traditional it my contact info is on the bottom left and the project contact info and i get these folded as well on twitter and the plc website is right here all right so let's jump into the agenda we're going to talk a little bit about the industrial control systems the purdue reference model how ics and ot is different from i.t

the impacts that we've seen so far from the various threats the vulnerabilities that are well known in the plc world and then secured by design system and of course more details about this specific project why this list what's in this list and what is this top 20 plc coding practices list and then a few examples to get you a good feel for what they are and i'll be kind of monitoring the session chat here so if you have any questions just put them in the chat that way um let me go to the session chat all right so all right so next up basics of industrial control systems if you're familiar with it maybe this is a good rehash control systems

existed for a long long time even before electronics before digital uh they were based on pneumatic which is air or hydraulic you know which is oil so steam turbines and equipment that ran as control systems exist for a long time hundreds of years now of course everything is electronics based you'll hear these terms scada which is supervisory control and data acquisition programmable logic controllers or plcs or distributed control systems or dcs these are all different things they kind of developed from different use cases but over time maybe the usage is kind of blended because a lot of these systems are now more than capable of being what the other was intended for originally but you'll hear these terms in the

industrial controls world quite a bit but the key in the industrial uh control system compared to our traditional i.t system is there is a cyber physical aspect to it that is something that's rotating spinning at the end of the electronic series or there might be a valve that's opening and closing allowing for some kind of fluid in a water or oil to move so there's always a cyber physical aspect or component to it which impacts safety right the traditional ics vendors are for example standard electric rockwell automation honeywell ge all these different vendors that provide industrial control systems are kind of different from your traditional i.t providers like cisco or fortinet or palo alto for example

now these industrial control systems are used in multiple industries most of them are critical infrastructure so power generation oil and gas pipelines for example mining that we have quite a bit in bc and uh in alberta for sure oil and gas and then air rail and road transportation and my company does a lot of these water and waste water treatment utilities but you know even healthcare right so a lot of use cases in different industries now the purdue reference model this is kind of like the basic to understand how the industrial control systems are typically organized right or how you can get a better understanding for what systems are out there so it's a hierarchical structure from zero to five

you typically see the level zero as field devices sensors for pressure or flow or temperature and then the valves that i mentioned before or any kind of electromechanical components at the far end uh these are considered level zero and then level one would be the contours we'll talk more about today the program of logic controllers or the scada or dcs controllers and then level two would be these human machine interfaces the hmis now traditionally this was what considered ics and then of course you have this level two or level three where you have these same kind of human machine interfaces but for a wider area network so maybe this covers more than one site uh maybe

there is a historian that covers more than one site and maybe these days even a domain controller that has you know multiple sites under it and then these engineering workstations are those that are used to program these controllers and so you download logic or code from these ews to the field controllers so you'll hear this term ot as well so it's kind of blended these days so what ot or ics means most people use otn ics synonymously so they mean the same thing or you could perhaps think about ics as traditional industrial control systems with these sensors controllers and user interfaces and maybe ot as those that are what historically were it hardware like a dell or hp computer running windows

that is used in the industrial world as this hmi or this engineering workstation or even this historian but you know it doesn't really matter as long as you understand what the other person means when they say ot or ics or on the same page now you all understand i.t quite well now the enterprise zoom the uh internet dmz the firewalls and anything that's in this dmg is typically you know ending the it kind of logins or any kind it access here and then have a separate jump server or something to get into the ot lance this is typically what we expect people to follow not that every site is like this right a lot of sites have um

you know very flat networks we'll talk about that in a bit there's also this uh current um use case where a lot of iot implementations you know directly accessing cloud services that are directly getting to the ot network at the very bottom you know level zero through uh two uh in in our world it's typically dubbed as industrial internet of things as opposed to just traditional iot but you know the same concept now ics and ot is uh quite different from i.t in a lot of ways so in the it world you know we're trying to focus on the cia triad because the idea is to protect information from an authorized disclosure or you know losing

integrity or availability however in the ot world there is an alternate triad that's more important or more relevant so safety reliability and productivity safety obviously is paramount because like i mentioned cyber physical there's always something at the end that's moving that's rotating or impacting uh physical process so you need to be safe you don't want to hurt people you don't want to damage equipment so safety is paramount so a lot of cases have backup systems backup relays for the plc for the controller so that if the plc fails for whatever reason you have something else that can stop the damage or the accident from happening similarly safety functions which are specific applications specific uh logic and

hardware that kind of has a backstop to the basic processor which is in the plc similarly the hardware even the traditional plc is developed for reliability um for example operating in higher temperatures you know out in the deserts or out on the deck you know much higher uh weather conditions compared to you know being an air conditioning room right and also for longevity many of these systems operate for 15 20 years or more um so they're they're built in a different way okay now like i mentioned uh flat networks are much more common in the ics and ot world again because of tradition because of history because you know many of these things were not connected to the

internet so it made sense for a common architecture in a site so that one person can manage from a sample location they don't have to worry about segmentation they didn't have to worry about isolation and we'll talk about access and authentication in just a second so because of this the software and protocols that were built for this world for this ot world uh way before i t right these were built before um you know many of the it protocols were available uh they were built with functionality in mind so there was no authentication no encryption it wasn't required it wasn't part of the puzzle plus when you needed time sensitive um delivery you need less less overload right so you less

anything overhead so that you know you're not impacting anything for loss of package for example so these systems are dubbed insecure by design because they were never intended to be secure in the first place so no authentication like i mentioned historically for level zero and level one devices level one device is what we'll talk about you know plc's the iot devices like i mentioned with new connections directly to you know the level one devices or level zero devices um they're bringing on an additional risk on top of the fact that you know these about structural problems exist right now it solutions don't always apply to ics or ot problems even if the hardware is very similar or

exactly the same for example the you know if you're talking about iam you know the hmis cannot be logged out so if you're trying to authenticate somebody if you're trying to identify somebody and expect them to log in and use multi-factor authentication for example you don't want that to be a problem and you know the operator has to adjust something for process reason and they're fiddling with their um you know two-factor authentication key or they forgot the password or you know anything like that so typically those user interfaces are uh open they're always on there's no password to get in right similarly the dlp data loss prevention uh if you're trying to restrict access to the plc for example but that might be

the only way to download uh controller code or you know have an update on an alarm or something like that so you can't use the traditional it kind of solutions here similarly if you're trying to use a firewall even at that level one you know where it's an industrial protocol maybe you have a special firewall but if you're trying to you know configure something too strict and drop packets if you suspect something based on an acl but then that drop packet is critical for the process to run safely and that's a problem right so the configuration and the rules will be slightly different there and maybe you don't want to drop out packets in case something is wrong

similarly patching on the hmis or even on the firmware level for the plc might not be available at all um because this is an older plc 15 years old maybe out of support um but you can't replace it uh obviously for availability reasons or maybe you know you don't have the budget for it um and and and other times even if there is a patch and you apply it it might not reduce the risk at all because like i mentioned before these protocols themselves are insecure by design so you know yes you patched the hmi yes you patched the plc but someone used a legitimate port legitimate protocol to access something because they got into

the it network and they're still able to impact the plc so what have you seen because of that right so the threats and impacts we've seen consistently over the many years uh moving from right to left you know we had the um shimon and an arpeggio that came in from the ite world you know destructive malware that caused a lot of havoc in saudi arabia and then of course ukraine and also ransomware now most recently of course the jbs and colonial which kind of put this in the spotlight but even before a lot of ot systems were impacted because the ransomware came from i.t uh wannacry for example and then we've also heard recently about

the old summer florida instant where someone was able to remotely access uh you know because the remote access was insecure using a published password and were able to adjust the concentration of sodium hydroxide 100 times more than what the available limit available or the allowed concentration was so there was another one here in kansas as well so these things you know are from the ot side and many of the ransomware attacks happen from the i.t side but then there's also traditional industrial control systems uh being affected you know struck snap was the first famous one from 2010 all the way to triton which was in 2017 affecting safety plc's in saudi arabia so these systems were affected with a

physics payload essentially someone was using a replay attack on the plc or infiltrated the supply chain uh even though they were air gapped in in iran for example so the threat actors obviously you know ranged from apt nation state all the way to criminals and activists and some malicious insiders so there's a whole spectrum here but if you think about uh you know what gets the most news is this side right so most of the i.t ransomware or the ot insecure remote access takes uh most of the precedence in public news but there's also you know this aspect which impacts the plcs or the controllers themselves so like we discussed before the plc's are vulnerable and they're insecure by

design and you know you see some news items here talking about how a certain project mentioned the plc was just open very easy to exploit and then all these vulnerabilities so why are we talking about plc vulnerabilities or or you know plc coding if everything is just so bad right that that's the main question that we're gonna address today so why is this list relevant so let's take this sample example this scenario where ransomware has affected some of the i.t systems and you took a precautionary outage this is no different from what happened at colonial right so colony had some i.t billing systems affected and then they shut down the whole pipeline so you're trying to mediate the eit systems

right but before restarting the process it's only logical to ask have any of these ot systems being affected right and traditionally these ot like hmis or ews is that we mentioned were windows based so the traditional endpoint protection or maybe some kind of network monitoring can help you answer nothing fancy right i mean just an endpoint maybe antivirus would flag something or maybe your um cisco router you know gives you some feed you know some some information on what happened in packets right uh that's possible at the um level two or about but how about the level one if somebody asked you this question are the plcs affected the controller is affected can we restart the process safely

then this question is it possible to answer using native functionality in the plcs we think so so back in s4 is a conference in 2020 january jake brodsky did a presentation about uh all the different good things we can do within the plc's without installing third-party tools without having to invest in uh separate i.t appliances there are things that we can do to kind of answer this question so that's how this top 20 secure plc coding practices project was initiated we had more than you know more than close to a thousand people register on the website several dozens contributed we had regular meetings for a year and then this june we were able to publish this version 1.0 uh it's

available on the website for free to download and you can use it as you see fit because it's a community effort uh where you know you can use it for improving your program and practices now what's in this list right so this is the level one device right the plc of the communication uh controller and then you have level one which is the hmr the user interface and then the level zero um sensors and the field devices so we try to restrict our um our plc coding practices list to just as plc because there are other things you can do at the level zero and level two uh level uh to improve security but we try to focus on the native

functionality in the plc itself so any code that impacts the plc i will try to come up with items to improve the security posture there and then potentially in the future we'll add this environmental practices that kind of covers the others now this is the list we have 20 of them uh you can go to the website that's on the slide tool you know the link is available so you can download the pdf for free you don't have to enter any information we're not collecting anything it's just a free pd after download it's about 40 pages has these 20 practices and all the details regarding the practices but we're going to talk about a few of these so that you get a

good feel for what these practices are and how they're relevant to the controllers and the process safety let's start with something very simple pretty straightforward the practice number 16 by the way these practices are not ranked so even though there are 20 of them it's not that number one is more important than number 20. all 20 are um for various reasons uh important for your process and you can kind of look at what's applicable to your particular situation your particular process or plant and then adjust accordingly so practice number 16 says summarize plc cycle times and trend them on the hmi the user interface so cycle time is the time it takes to compute each iteration of the code or

the logic in the plc and mind you these programmable logic controllers these are all deterministic meaning you provide them certain types of inputs you expect certain types of outputs and everything is known everything is deterministic so the cycle time should be constant on the plc unless there are changes to the code or logic or something else in the process right so if you trend them for example as this bottom right shows you'll see some minor variations but let's say someone is trying to execute some malicious code you get the spike right so you're able to see that every so often there is some additional code execution so they can you can do some investigation we don't

call it forensics we kind of call it troubleshooting in the industrial world but you can kind of look at it and say oh something is wrong right now traditionally plcs have a limit the cycle time once it reaches a maximum cycle time it'll stop the plc so that uh you know the hard stop comes into effect so malicious actors are well aware of it right so they try to keep the additional code clean or lean so that you have some variation but it won't reach the maximum which is why i think this trending and logging is important and so you can have this kind of band where it's allowed but then once across a certain band you know

you can set an alert we call it an alarm in our world so you can set an alarm for the operator and of course you know if you have this on the hdmi as a trend the operator can easily see that in every so often there is something going on that they can alert the engineers the controls engineers to come and investigate so visualizing values over time draws attention to anomalies right now next up is practice 12. um uh i'll test oh jason yeah thanks for saying that uh yeah uh pretty high yeah you're right that most of the the threats in the public um you know so far that we've seen came from the i.t side usually

ransomware or some kind of easily accessible internet based attack right that's why if you go on short and for example you'll see a lot of uh internet facing controllers and then most of the attacks come from that way but there's also you know the supply chain attack um like we mentioned the stuxnet is a good good example nation states have other avenues obviously um to attack um you know these process plans all the way to triton these uh physics-based payloads these supply chain attacks are few and far between you know only five in the past 10 years but that's significant and uh if your threat model involves that for sure need to be considering that

but uh next up is practice number 12 uh validate inputs based on physical plausibility this is an interesting one because if you look at the practice it has a few examples so one monitor expected physical durations for example a gate on a dam uh would take a certain time to go from fully close to fully open and you know what that is right so if the gate is suddenly taking too long then there could be a mechanical problem right or it could be that someone is trying to force it close or force it open or force it to be not fully open right it's possible that someone is trying to affect it so if you alert on this that

hey the gate is supposed to open in you know two minutes but it's stuck that should be definitely an alarm right and it would be in a critical loop anyway because they're trying to make sure that the gate goes to fully open but there is also a cyber security benefit to tracking this uh you'll see that team across the board in all these practices that many of them have reliability and maintenance benefits but we're also trying to as part of this practices list focus on the security benefit of it in case you didn't configure it for a maintenance benefit consider configuring it or programming it for a security benefit similarly in a monitor expected physical

repeating activity for example the wastewater treatment plants they have this diurnal cycle so what happens in the morning versus what happens in the night or maybe in a batch process what happens at a certain time during the day and then in the afternoon versus evening they all should be consistent right so you're trying to get a particular batch out or you're trying to get a particular type of quality in the water but if for whatever reason that different you know someone is trying to influence it uh by changing something and changing the treatment methodology or dosing methodology or something like that you should be able to highlight or alert it based on this you know not being repeating you know it

should be repeatable but because it's not the alert and then limit operator entry for set points that are practical or physically plausible or possible as an example for this old florida that we talked about before yes you had insecure remote access yes somebody was able to explore that to access your operator interface yes they looked at sodium hydroxide levels and then they entered some value right but if you think about it that value that they entered was a uh order of magnitude higher 100 times more than what the typical dosage is um you know or maybe it was thousands of times so why is that like if you typically need that dosage to be 10 milligrams or 10

something parts per million and physically the valve can only open so much in a certain duration it'll take an hour to get to maybe that additional 10 extra parts per million how would you or why would you have an operator input that's thousands of times more allowed right it should not happen so if the operator can enter only between 10 and 15 or 0 to 15 and the valve can only go from 0 to 15 that's going to align right so that way even if somebody makes a mistake or if even if someone malicious is trying to affect it they can only go to that maximum level that's allowed and physically possible in the process

it cannot be anything more so that you're trying to remove one avenue where someone can affect it right similarly this other practice practice number two i'm showing a picture here it's an example of a plc this is how now bradley plc looks um you have the this is the safety section this is the input i input and outputs called the i o power supplies at the top and then the controller and the different modules for the i o processing and you'll see here that there's a key switch so some plcs have this key switch that you can put that in a different mode based on your needs so if you put that in run it runs the plc code

if you put that in program it's still running the plc code but you're able to download code to the controller and if you put that in remote in some plc's it has multiple options so remote means you can only program remotely or in other cases you can actually run or program it remotely so it just depends on the plc but the key is if the plc is in run mode it typically cannot accept download to the controller so people cannot impact by downloading any code to the controller so if the plc is not in one more therefore this practices alert the operator right so there should be an alarm if the plc is not in run

mode now you might be doing something an engineer might be actually doing something he's you know put out his coveralls went all the way and you know turned the switch to program mode because he's doing something still the operator is aware of it because there's a permit open somebody's authorizing this download so he can ignore that for the shift for that eight hour or ten hour period um so exception for example for this testing or the maintenance window but at the end of the shift when the permit is closed when the operator is aware the testing is complete it should go back into run mode so that you know it it's kind of like a

backstop where no matter how how or what access was allowed to that level one device if it's in run mode obviously you can download to the controller so you can't impact or change the code for the most part right uh there are examples of that being bypassed and that's a different scenario that's a different level of nation state attack uh now the plc does not have a hardware switch again this concept is to track operating mode so if it doesn't have a physical switch you can still use some other software mechanisms maybe use a checksum feature we have another practice that talks about that or some kind of password mechanism to avoid you know someone being able to

download the controller from remote um next up practice 13 now disable unneeded unused communication ports and protocols these two seems pretty straightforward right most controllers and network interface modules support multiple communication protocols unfortunately many of them are enabled by default so whether it's telnet or ftp obviously you know we don't use them in the it world anymore but because historically they were using the industrial world they're enabled they're they're available and like i mentioned enabled by default but every enable port and protocol adds to the plc's potential attack surface so attackers can't use them if disabled right they have to enable them again to use them but you can you can alert if uh any of

these ports are enabled or protocols are enabled and of course you can detect if a certain you know protocol or port is in use depending on traffic monitoring and if you have a firewall if available but even natively you can set an alert or an alarm if a protocol is being enabled which is going to be your your path to recognize that something is changing in the environment so the best practice that's recommended as part of this practice list is to develop a data flow diagram that kind of shows all the required communications about the physical ports and the logical networks and protocols so you're kind of aware you know even at the plant level

or if you happen to be a much more sophisticated company organization that has even a at the stock level too can easily recognize that a certain port or you know protocol is now enabled uh when in fact it shouldn't have been because it's not used in the system right now the other benefits besides security you know it reduces the potential for malformed traffic on this protocol for example whether it's malicious or not which decreases the chances of the plc crashing mind you the plc typically has limited uh cpu capability especially the older ones so there are more ways it can crash in case it started expecting some traffic but it gets it right so by reducing this malformed traffic by

dissolving those ports and protocols you're reducing the chances of the plc crashing and overall the plc complexity is also decreased right you don't have to administer some code that's not used you don't have to update that firmware for example if that particular vulnerable cord is not in use you hear a lot about these s-bombs these days you know software builder materials um let's say you do in the future have an access to all the different code that's used in that particular firmware version and you recognize that you disable that code because you're not using it that is probably you know less of a headache for you because you don't have to update that vulnerable code you

don't have to keep it keep it up to date we have many more examples like this but i wanted to end this here because uh this project is still ongoing uh we have a lot of contributors to this we have been doing a lot of presentations about this different engineering companies uh going to different vendors and and trying to uh improve their security posture by you know programming the plc's better right but we would like some feedback we would like some collaborators to uh spread the word essentially um we had a lot of collaboration with the mitre you know both cwe and the attack framework are mapped in the plc practices list uh the cw team is trying to

come up with a plc section uh they currently don't have it but they're trying to work with us and uh you know include that in their cw list a lot of plc vendors and user groups have been approached and we're getting some feedback but you can be part of this and if you are a plc programmer or if you're in the ot security uh you know give the list you know download the list uh spread it to uh the plc engineers the ones that are programming this currently right give it to them tell them that these are the best practices that the industry is telling each other to follow through so uh it'll certainly help uh we're also doing a lot

of articles and podcasts so um if you have any more questions you know by all means reach out to me uh let me know um you know we have more than me in the community project the secure plc twitter handle is uh is a good place to start you can also go to plchyphensecurity.com there's a formal feedback form there that you can give us feedback on the practices themselves or the implementation so i will end there and let's uh take any questions um just you know putting it out there so that you have this plc handle on the website um if you all need any uh these lights or the slide deck just let me know on this chat or on the

discord channel and then i'll send that to you any questions folks

thanks marcy plc security.com that's right let's see i'll go to the q a section in case there's anyone there not yet um yeah let me know if you have any questions folks this is uh i guess an interesting development that we did in the past you know probably 18 months or so because a lot of the ot security discussion or the narrative is top down coming from internet based level five down below not so much at the level zero level one so this is definitely an interesting development

grant sure um if you don't mind just uh let me drop you my handle yeah just message me or something or if you go to this plc hyphen security.com you can email me there as well and i'll reply and give you the slide deck i don't know if there's any other easy way to do it but if there's a way to upload i can do it here but this is probably just as easy

yeah this is an interesting portal that we have i wish we can have a conversation so if you have any questions or comments we can share but it seems to be one way so i'll stick with the uh time duration maybe spend another few minutes making sure if you have any questions that i can answer on this particular platform voice is limited to people who are moderators so i will i will get if you have any questions please do drop them in the chat thank you marcy

yeah in general i see 13 people um could you all maybe just mention if you're coming from the i.t world or if you are already an ot practitioner could you all maybe put it in the chat to see what the background is for those that attended just interesting for me to kind of see you know where we need to go next right so we're again this is not a commercial product or anything you know a lot of people that contributed to this programming list uh work for all these different companies but uh it has nothing to do with the companies right so we're all trying to uh come together and then put a list for

community and public use so thank you jason i appreciate it uh itnot i um well that's interesting so i guess it came from the it world and uh are now responsible for ot as well we see that uh quite a bit um um like jason mentioned and a lot of threats are coming from the it world so we're certainly appreciative of the it people that are willing to learn the ot world can understand the nuances you know why you can't do a certain thing the same way you do in it in the ot world and and work with the engineers to obviously keep the process up and keep this you know people safe right so we

appreciate that we have a lot less people coming from the other side coming from the ics world and learning i.t though we have some increasingly so but we need both right we need the engineers that do the programming we need the technicians that work with the hardware uh everyone to learn the importance of security learn what's going on and you know improve the security posture for the organization in total

grant started out in electronics and control systems and i t networking and i t security oh that's amazing uh you that that's one of the few uh kind of superstars we have going from the controls fall into it so that's amazing to to know roger couldn't do how to join the community my suggestion is to start with that website plc iphonesecurity.com there's an ebay email address published there as well send send a note to that email it comes to about three people one myself as well so depending on how or what you want to participate in if you want to be on the monthly calls we'll send you that calendar invite or if you want just the

team's channel access where we put these documents where we are doing a lot of translations these days maybe you want to assist in one of the translation projects or you want to do the presentation yourself to maybe a local engineering community we have the powerpoint slides over there for you to customize uh it's a free-for-all kind of thing because you know we're not trying to be gatekeepers here we're trying to encourage more people to get involved so yeah the easiest way is to send me an email which is published on the plc hyphen security.com website or you know follow a secure plc on the twitter handle and dm me or you know send a note on that

twitter feed

and rajit what's your background are you an i.t person or an engineer in the ot world or what's your background

okay

all right i think we've been almost five minutes in here let me answer this question engineer starting from control system ot and moved into network security well that's awesome so grant was uh something similar as well this is rare folks i mean you're one of the chosen few i came from this world as well right i started off as a controls engineer and then you know learned i.t we have uh it experts right i mean jason for example is such an amazing speaker uh we're so glad to have him you know watch this presentation we we definitely can't take a lot of help from rit folks because they solve a lot of problems um already

so we don't have to reinvent the wheel but we should bring our perspective to it as to why certain things are not applicable or should be done differently in our world in the ot world but yeah anyone that came from the engineering side in the ics world or and i started off as a controls engineer in the ot world and turned the learning um you're probably like a third to a quarter of the total ot focused uh folks these days so uh pat yourself on the back that you're one of the few uh interested people and and and trying to do the right thing for uh the greater community i really enjoyed talking to you all um

and yeah look forward to collaborating further via the emails or you know twitter or whatnot i have my contact info and let me know if you need the slide deck bye folks