How I Pwned the ICS Data During My Internship

[Music]

okay so let's get started here so hello everyone uh first of all i would like to thank the besides philly committee for inviting me here to talk and i would also like to welcome all our audience here uh for this presentation and today i'm gonna be speaking on how i found the ics data during my internship so a bit about my background before getting started so uh my name is shail patel and i'm also known as venti cv uh and i'm a security researcher and an application security engineer at form assembly uh previously i have also worked at inrel back in 2019 there i focused on the ics and scada security uh i've been in a cyber security space

for about two years now and occasionally i like to participate in vulnerability disclosure programs bug bounty programs and i also like to do ctfs so the first of the presentation uh was a part of my internship research from or a year and a half ago back in 2019 as i mentioned so while i was interning here uh there uh this was performed in a controlled laboratory setting with all kinds of capabilities so now the project team has devised data enhanced hierarchical control architecture so this dehc architecture was developed to solve some unique power system challenges such as a field load balancing reverse power flows or voltage stability so with that in mind i i performed the testbed security

assessments from various perspectives including the communications and transit site specific device level security assessments and when you devise security analysis as well so the last half of this talk will be focusing on the notorious findings in the in the world that i did for this one and just for educational purposes so uh let's get started here with the basics of industrial control systems so in simple terms industry control systems are basically the computers that control the world around you so uh man responsible for managing the ac in your house or maybe in your office or uh turbines at the power plant or an electric power grid so there are there are some weird examples of uh

industry control systems so some common terms associated with the ics are plc's programmable logic controllers distributed control systems remote criminal units and hmis so this these are the various entities involved within the ics so uh now before diving deeper into specifics it is uh really important to understand the difference that uh operational technology which is ot uh systems vary in a lot of aspects then uh traditional enterprise iit based implementations so the way it works is operational technology monitors and manages industrial processes assets and manufacturing so essentially it's a hardware and a software that keeps the industrial systems running so i would say that the common way of looking at the security requirement of an id system is to acknowledge that it's

data centric so the key requirements being the confidentiality integrity and availability of the data being processed but for operational technology for ot systems um let's say that an attacker might be able to see the furnace on or off or at what temperature is your you know ac is operating is isn't really a concern so but hey what if there is no availability of power supply so just imagine that we are in the middle of the winter and uh there's a there's no backup generator which is even if it is there it is compromised and let's say the greatest compromise so what do we do in that case right so uh there are some safety concerns as

well that we are going to be talking about in detail uh associated with that thing uh so for another reason uh ot is less concerned with data and more concerned with physical processes that i mentioned so the primary pro concerns of ot systems are reliability and safety so it is important to note that not all the id security testing methods and common findings and recommendations can be used in the ot space as well but i'm very sure that i'm going to be talking about a lot of overlaps uh as well uh and along the in this presentation down there so it is important to note that organizations leverage lessons learned securing enterprise id but they also adapt those lessons to the

unique characteristics of ot so now everyone is well aware of the ukraine uh great attack example back then in 2016. so this is this thing loads uh moving beyond the perimeter-based security in a facility and then adding security controls to the assets that matter most so uh let's let's talk that thing that ics cannot be easily taken down for security updates and so often they therefore remain uh vulnerable and unpatched so uh furthermore because these systems have uh very limited computing resources right so they often lack the capacity to run anti-malware softwares and various kind of monitoring tools and devices so but in recent years operators of the critical infrastructure have increasingly deployed recommended technology controls for

predicting their systems but there are not enough people to man them so there is a shortage of enough people to back up the icas so uh another thing is that if we don't have any clear visibility to figure out what what we have installed in the field and uh basically we are dead in the water right so biggest mistake that is being made is comparing id security with control system security but uh they are still fundamentally different and they're also overlaps as we are going to be talking about them in detail so regardless of our experience no matter how well trained we are we we all make mistakes and it is okay but sometimes this human mistakes uh

leads to severe consequences and causes harm so in order to avoid this implementing industry control systems was essential but uh later we realized that this industrial control systems they this itself has its own problems so now uh in this project uh this project had a hardware in the loop setup in the test bed with different devices interconnected in the ics topology so now a hardware in the loop was implemented for the development and testing of this complex power system here so uh basically our task here was to capture the communication between the beaglebone device and the pv inverter now pv is nothing but photovoltaic anything that is to do with the sunlight so note that the communication protocol

used in this case is modbus so modbus is an ics protocol as i mentioned earlier so modbus runs on default port which is 502 so in in this case here we are capturing the modbus traffic between our beaglebone device and our photovoltaic inverter so but before diving into the details there are certain things to be taken uh note off like say for instance legal burn uh it acts as a local controller for stabilizing the inverter set points now unlike tcp and udp uh modbus is a communication protocol as i explained in the earlier site so it's extensively used in this data systems uh and which was also used in our our case so modbus yield protocol

is nothing but a master sleeve protocol now one master control modbus data transactions with multiple slaves that responds to the master's request to read from or to write the data to the slaves so there are various read write and uh operations that can be performed within the modbus topology so uh it's also important to note that anyone can implement modbus in their field device architectures and since it is free and an open standard available so now each device communicating on a modbus protocol is given a unique address so as you can see here on modbus rtu so there are two types of modbus but rtu is nothing but modbus remote terminal unit and another being as uh modbus tcp ip so

uh in this case only the modbus rtu uh can initiate a command so it's always the case that okay the master is listening to the slaves and slaves are sending or asking for some requests so whereas in this case of modbus tcp any device can send out a modbus command and thus they can act as a master so but or as i mentioned although normally just you know one device acts as a master so uh with that in mind we can say that okay we are initiating the request here okay so modbus system we have the master and we have the slaves so the uh to the master we are initiating all the commands to

the slaves and uh in that the action is performed through the receiving response side for the mod for the master so within the function code and also it has a data response so the sleeve has a specific format just like a header and we have pdu that is packet datagram so in this case as well we have function code and data response which is sent back to the master so we can we have to note that you know read only as well as redirect operations can be performed on the input sets so as you can see here that modbus commands can be instructed to change the values in one of its registers which is nothing but

writing to the coil and holding registers and it can be also used to read an input port to read the data from a discrete and coil ports so you can see here there are four types of functions here your there are okay various use cases here that we can use for read-only rewrite and uh we also have the bit bitwise operations that can be performed within the object types so within this table we can see that to command the device to send back or one or more values containing its coil and holding registers so modbus can be used for multiple purposes within the registers so the command contains the modbus address of the device it is intended for so this time i

mentioned a few slides back once where each modbus has its own device address so about beaglebone uh it is nothing but a low cost and low power open source development platform for running and creating your own experiments so this legal bone here was programmed by the power systems team to control the photovoltaic inverter so for the purpose of simulation in the test bed they had two things in the loop so one of them being open dss which is nothing but distributed power simulator which was used for grid modernization and another thing was opel rt simulator which we're going to talk about in detail in a while so now inverter was used for converting the direct current into alternating current

and in this case we also used for continuously monitoring the grid for optimizing the power along with the active and uh passive monitoring of the controls for temperature so there are various things that are going with the inverter as well so now that the testbed is set up uh i all i did was i was in the lab so i jumped to the same subnet and i used the network tabs from with multiple monitor ports for uh monitoring the ethernet links so you can see here that dual comp is a company and it manufactures network tabs that we can use for uh plug and play in the network so now once i was connected to the network then

uh with the help of wireshark i filtered out the protocol uh which was modbus in our case so i quickly filtered out modbus in our case and allowed the package to be captured simultaneously through the tap so here you can see that all the communication that happened between our beaglebone controller and our inverter is disclosing the plain text so this is nothing but the pv set points the photovoltaic inverter has its set point values from the coil register so this is not nothing but the values from this register so now now we got we need so what we can do is you know with the help of this tool called cache modbus scanner so chipkins is a

company which makes uh this tool uh to play around with the modbus values so we can perform false packet data injection uh play around with the values and we can rewrite and we can manipulate the coil values as well so note that they support both the modbus method so uh serial modbus rtu as well as modbus tcp uh which was uni used in our case so all you can do is you can quickly provide your target ipad this year uh for modbus protocol uh of course it's a default put 502 and you can connect to their modbus system and your and of course this is a remote connection so you have a long lasting stability so

we can start discovering every detail from within this such as register values so this is some kind of a confidential information that modbus register holds the addresses so we can start looking for such registered addresses to discover for and another option it provides just for pulling so modbus pulling allows you to monitor several slave devices simultaneously so which is indeed a time saver so with the help of modbus pooling we can monitor all the sleeve devices together and we can we can save time with that yeah so uh another thing is another tool lighting is being used here is levia modbus so this is also used for polling but from the previous finding here of

our register values uh all we can do is we can input the ip address here and we can also provide the register address from the previous finding here and connect to the list of all the slaves and just passively sniff through all the traffic and listen to all the slaves so yeah there are some amazing functions that this tools provides for monitoring the network traffic and as well as changing like reading as well as writing the data to the registers so another interesting packet capture scenario here was sniffing the uh sim and udp traffic so sim is nothing but computer integrated manufacturing so this scenario consisted of rt opf so which is nothing but a short python code

for setting up the power flow balance sequence uh voltage as well as the load parameters so this python script holds various voltage values as well as uh power flow equations values so unlike beaglebone our tech is nothing but a real-time automation controller it's an automation platform which was used in our case to act as a tv controller so this was used as a photovoltaic controller for connecting to the other substations which sends command and control to the field devices so rtx is different than rtopf which is also different than the inverter so all these devices are performing different functions and each whole own unique values and confidential values that we can look for so in this pcapp here if we pay close

attention to any of the response packets you can see that there is a 25 or 24 bytes data string here so that kept me thinking this is the string that we should look for so what i did was i further followed the stream line from the settings and i found that one of this hex ring and another was a binary stream so i know that there was some uh juicy data accident behind this so binary stream was nothing but a clear clear clear text stream so there was no decoding of any kind so i did not have to use any kind of decoding tool to find what's there any is there any hidden data behind it so

that was a that was just for the clear text stream that i found for so now another thing that we can look for is for the hex stream all i did was i converted the extreme to the little indian format so uh this can be done with any online utility or any other tool that you can look look around for so this were nothing but the analog decimal communication values here that uh that were used between our itac and our rtopf that we were looking for to test so uh this is a this was an interesting use case scenario where we were able to dig up uh some analog values that were used in the communication between

our rtag and rtopf so one of my one of my favorite isis protocol is dnp3 distributed network protocol and the npc runs default on port 20 000. so the communication is completely in plain text in case of dnp3 so no need to worry about register calls and input values so all we can do is just re-read and read directly which is different than modbus so an important thing here to note is for adms it is a software platform advanced distributed management system uh it supports distributed management and optimization such as finding fault location or isolation as well as restoration right so uh we well we covered our tech already in our one of the previous slides so uh we can

keep moving forward with uh another finding so what is the data that we are looking for here uh within this screenshot so within this pcap filter uh all i did was i looked for dnpg i filtered the packet 14 p3 in the results and in the request response here we can see zeros and ones as binary streams so these are nothing but capacitive bank values disclosed when our rtac and adms are communicating and another set of values captured from the same test are all tech values sent to the adms in either volts or kvar nothing which is nothing but kilowatt amperes which is a reactive power as well as a unit of capacitor in our case so just like modbus we can

perform spoofing attacks false packet data injection or replay attacks or we can also manipulate time time colonization but unfortunately for me none of these dnpg-based attacks were in scope for me for testing so now now i have completed the packet capture analysis now it's time to move on to the device level security assessments so in order to test the beaglebone uh i i ran i ran the device through some open source other vd scanners i i ran the device to find some open ports and services associated with that so i did some dynamic analysis i also ran it through nmap i ran it through smart sparta and observed some interesting results so some of the open ports here where a dns

uh which was on 53 node.js running on 3000 and some unessential services which i was not interested to look for so it but it also appeared that one interesting thing was cloud9 ide and when i checked that repository it appeared that it had no authentication so i could quickly get into the beaglebone environment uh and hit the web interface but just by going to the web panel so it was totally lacking authentication and which allowed me to read all the image data within down this image so uh also i noticed the outdated version of apache here and it also had a direct listing enabled on the root directory that allowed uh unauthorized listing of files and directories listed on the

image so i i ran it through various tools i checked for various indexes i checked for indexing and i found out that yeah it was a missing parameter and so i confirmed that manually and since uh open open west is no longer maintained project but it was one of the good tools that discovered shell shock valvety in in in my beaglebone environment so shell shock happens when you are running older versions of bash which are vulnerable uh and that allows arbitrary code execution so after enough findings uh i ran the avashi and shell shock proof of concepts to measure the impact what happens in the vehicle bone and it was a successful pen testing so i

reported that to the team and they applied the fix to that thing so another in scope assessment for me was for checking the rtopf python code so since this was in python and it was it was allowing me for static analysis so all i did was i ran this through various tools like band-aid uh d-link prospector p lint and since this was short and sweet and simple code so it was under development so this limited my testing techniques for source code review here but uh this this this code nothing this this tool is nothing uh check for uh source code redundancies so in our case nothing was discovered from security perspectives except for white spaces and indentation problems

which was a normal expectation so moving on to the vendor device security assessments there were lots of pieces involved and i'm not going to name specific vendors here for obvious reasons but uh grid edge management system which is nothing but gems uh it's a web portal that can be leveraged in different ways for field device control and data visualization uh but since cryptographic attacks are very only present since quite a long time so i checked for the open uh openssl version it was running on the outdated openness version on this platform and so i was able to find out uh hardly uh proof of concept from this link and i ran the exploit and it worked so i was

able to confirm uh heartbeat attack uh within this utility and i reported it to the team so another another utility that i was testing was uh advanced distributed management system so while auditing adm software i can relate the flaws from our packet capture scenario so there are no signs of encryption here there is nothing such as uh there are no signs or there are no uh tools or import functions for cryptography so as you can see here either while feeding the electric values here or either while data is in transit for dnp3 so distributed approach for adms management would have been a better approach than this for managing these assets instead of centralized management like this

in terms of security so yeah that was an understanding [Music] so another day another assessment but this time i did recon on the inverter but did not find anything concrete that i was looking for so the next thing that i did was i scanned the whole subnet for this test bed so for asset discovery i found the web interface for inverters cluster controller so this was nothing but the hdmi which had a prompt box for two default users so i got excited i had default users so immediately i looked up for the data sheet for this model and it had default credentials user 000 so i immediately entered one of these credentials and my finger crossed

uh it fortunately got me in so once again i had a complete control over the hdmi that not only allowed me to view the powerplant information but it will also let me edit instantaneous values and change some of the privileged settings within this panel so yeah there are some fun stuff that that can go around with this panel here so last year as a red team lead i was involved in [Music] building and creating vulnerabilities and exploits from scratch as part of one of the cyber security defense competition for students so my job here was to help the red teams achieve their goals so in various scenarios here i miss configured raspberry pi's i left the hmi for one of the solar panels

unauthenticated on the books i enabled django default i ran the chrome jobs to reset the firewall connections implemented sticky keys vulnerability i left the ftp server open and guest users enabled so basically think of any system network or application level flaws that will uh help the red teams for exploitation so for most but my most favorite thing here was to uh that i wrote a logic bombed back door uh and go to spawner reverse shell in one of the hdmi boxes uh in the middle of the competition to create cures so what this backdoor did was it opens it opened up random ports at specific timing so as soon as there was a competition time

boom boom it opened all up the network ports and with the help of that red teamers can connect to the sports and they can have access to the back doors so the proof of concept was also available on github if you can if you want to check that out so uh enough of the serious talk now so from now the later part is uh about more about my finding in the wild using showdown so as most of the people must be already familiar with showden it's it's an iot search engine where you can find any specific model devices including endpoint devices as well as servers so you can literally search for anything on show damn so showdown is available in

free version as well as it has its speed features so some of the some of the good people here are from uh scadas changelog group who keeps on posting the screen devices on the internet as assets so one might want to check that out as well so now the way that student works is it's also similar to census which is also another uh spidering engine so all they work is by inspiring all the ipv4 addresses and maps all the banners and host names and other records to that particular ip addresses so you can also do targeted search for cvs addresses or ports and services for a particular target on shorthand but apart from direct search shouldn't

also provide api version to search with command lines so the interesting thing that it allows you to check live state of the device uh so you can filter out quickly on what to focus and what to not focus so it clearly shows what is the device model and from what organization or company so there are some fun findings that i'll be presenting in coming slides for default credentials and along with the banner state information uh the search will also show you the configuration parameters for certain type of services so it can help us in finding misconfigurations in this particular case now many domains and subdomains have little no authentication where you can visit them on the banner uh

which you can see in a few examples in the community slides now this is an obvious statement here so you recon the weather are your findings so you can use google as well as showdown and all the other types of search engine down the road this is this is a perfect example of no authentication where showdown offered me this electrometer so if you have a close look at this hmi here you will observe that this meter has a lot to offer now you can read the voltage all the power related information you can check the status of the substation here from the settings and you can do many more fun things uh so yeah this this uh electrometer was

openly available on shoulder and without any authentication here so this is this is a particular thing to take a note of and i also looked for network components such as routers that were openly exposed on the internet now there's a login page for sierra east manager router so uh i did was i entered username user and password12345 which were the default credentials i just guessed it and it worked out so now if it is a router and you're logged in now that so you can know that the quality damage that you can do at clear fee so you can change the ip addresses input addresses break the connections disable network authentication and basically you can destroy the whole

network there are multiple default credentials available online for this particular model but uh since it was allowing me to log in with user at one two three four five so i did not look any further uh hp printers are very interesting things to look for uh so again it did not ask me for any login information you can check the device sheet here you can check that from the configuration settings you can check check the usage event logs who painted what at what time from what printer so all the metadata that we we call it we can switch along from this machine model uh you can also give the print commands anonymously and just print and waste the

whole cartridge right so that is something that uh that can also be used by an adversary now another hp model but uh same old mistakes here zero authentication now if i was a bad guy like i mentioned earlier i would have emptied the whole cartridge and printed random pages and changed the access control by kicking the admin out i would have done some exploration get to know the printing history for their important documents now it could be it could be anything from pay steps to any uh private employee information which can be stolen easily and quickly so there are many fun things you can do here and mess around with them within the printers as well

this is a controller unit from index company that runs on iec 60875 which is a common transfer protocol standard for power systems automation uh so here i did a reverse lookup for this standard on showtime and it brought me up here now again there was no authentication here uh so it's even scarier here since it allows unauthorized file upload here as you can see to this hmi so this is a serious issue now i could have i could have uploaded a backdoor or a ransomware it also also displays the firmware version here so an adversary might also look up for our known uh proof of concept here by checking the version number so this is a cobam satellite which you

can see here and it is an active slayer so you can also observe that it has the receiving transmission uh signal strength here so you can see that is an it has an active received strength as well as active transmission strength the dashboard gives a variety of satellite parameters such as where it is located frequency elevation gain antenna information you can also go to the settings and manage the administration and you can also change the admin access so if satellites are compromised then chances are yeah the space technology is going to be ruined up in the future so this is another thing that needs to be taken care of in terms of iot and ics exploration

this is a live solar log inverter in production which is exposing the network device configuration system and plant details uh this is almost everything to spill out the beans for so the next time when you see this device by yourself you know you need to do you need to know what kind of uh exploitation techniques that you need to apply to extract the data out of this model so yeah and i would say you can research this model and then look for detailed parameters to further exploit it now this is the tesla's power powerwall battery for controlling monitoring and optimizing which is showing an active status here with load unit measurements now isn't it fun

to read someone else's better status and measurements so this is displaying everything like variety status how much power how many horsepower how many kilovolt ampere at what kilowatt is your batteries battery running how much percent is remaining so an attacker might have some fun by modifying the parameters or might further brute force the directories and look for more information from this now these are unauthenticated webcams from houses schools or localities it can be from anywhere so on on that bottom right here this is an hmi for the wind portal now this was the only hdmi amongst all my findings where my default credentials did not work but it allowed me to fire up some xss now

this could have been further utilized by me or any adversary for credentials stealing or cookie stealing with xss so what does this all imply here uh with all this everything so they just want easy access to get in and they did not bother to change the credentials so therefore there was no authorization there was no authentication of any kind except for local backups now you don't need any transcripts or frameworks for finding sensitive information you can just go to the showdown or any other search engine and do open source intelligence and find some sensitive information uh like this so all these instances should not have been opened available on showden or any other search engines as well but uh

like i said earlier it's the laziness that is thriving it happens on accident so every day on twitter i read someone finding github or aws secrets or keys and whatnot now this is something that is not done accidentally now as developers we are just lazy that is very hardcoded information is easily available there now security through obscurity is nothing but hiding the underlying security mechanism now even if you are hiding your uh security mechanism that will not help you at all in this case to protect these devices because people are smarter they will find alternatives and yeah they will dig up and then uh they will use all the extradition vectors and attacking techniques that

they gather and then launch all the attacks so it also appears that there were there were no firewall rules that were implemented to block this connections from accessing this device remotely now of course i cannot uh stress enough the importance of the world default you have already seen it by now and of course this device should not be there on the internet right so there is no harm in implementing modbus if it is done correctly but uh no plain text so regardless of shield modern personality or modbus or tcp uh i suggest that add an extra layer of transport layer security tls for encryption and udp itself is unreliable but even if you are moving forward with udp

and if you have sensitive information uh you can use secure protocols like dtls datagram transport layer security for maintaining the privacy now like modbus do not just use default dnvg instead use the newer implementation which is nothing but dnp3 secure authentication dnp3sa uh so this dnp3s will provide role-based access control authentication and authorization for these devices so check out the new version for dnp3 and i think the newest version is version 6 and another interesting protocols that can be implemented for additional security or either as a replacement for egr step 2 which is nothing but smart energy profile iec 623 c5 but again it all depends on your applications and needs so you you might want to work that out with

your developers and your power systems engineers and then decide the protocols that needs to be implemented and same problem here they could have firewall beaglebone they would have disabled unnecessary direct listing they could have passed their own but their abilities now vendors make mistakes too so that that's why hybridity is a serious trouble in some cases now so that's why you don't want to read more than what you're allowed to from the memory and so i'm glad that the managers fix this problem now it is important to take other defensive actions other than having a barrier of networks like in this diagram to control what is flowing through locally and externally so most of the people are focusing on

enterprise level security now but we also need more people in ics security as well there should be an equal distribution there should be equal distribution of manpower uh if you are if you are a power systems engineer it is essential to know what you are placing in the production field it is if if the device is not uh needed anymore disconnect it from the internet and take it offline before you forget so we cannot just be like hey i had my plc in the field and i cannot i just found out today that i cannot log in and they are just compromised so yeah just take down the device from the internet and uh before you forget uh

always start from the zero trust model for this devices and obtain the right documentation for the systems before you're deploying because assumptions do not always work and it is it should it should not be just secure design and implementation for any grid or other ics related model but also having correct incident response keeping bts in place is equally important to defend from live attacks we can't we cannot be just like we have we have our pci compliance we have our uh all the checklist ready and we are good to go but we also need to be prepared because when the right time comes we need to fight back the attackers from using incident response so with all this in mind i hope that

there were lessons learned from many angles and you won't fall in any of these categories so feel free to dm me on twitter dms are open you can follow my github stay safe thank you very much

you