SecurityWeek has once again reached out to companies that offer products and solutions for protecting industrial control systems (ICS) against cyber threats, and asked their experts to share interesting stories from the field.
Their stories make for a good read, and can also provide useful information and insight for practitioners.
And the stories begin…
Ron Brash, Director of Cyber Security Insights, Verve Industrial Protection:
“At a site that was largely industrial automation equipment for the manufacture of packaged consumer goods – there were 3 controlling workstations running EoL Windows and needing copious amounts of electronic maintenance. As we walked through the risks with the site owner, we asked about what would happen if you had a failure on one of those workstations, and the answer was a surprise. In case of failure, there was a 4th system that plant staff would rebuild on the spot from memory; there were no other spares. Clearly, the recovery strategy was very optimistic, but in this situation – having a good night of sleep might be difficult.
Most horror stories center around the creativeness of site users & third parties – here are 3 of the more likely ones. While visiting a remote pump site, we found a device serving Wi-Fi without any security mechanisms; the network it was connected to would have allowed remote access directly to the facility plant floor. At another site, we found USB wireless dongles and TPlink software installed on HMIs that were supposed to be behind several layers of security and without Internet access; they were connecting to the corporate wireless, and the site owners were unaware. Lastly, for multiple sites, the majority of “bad things” reside on transient technician laptops AND they move freely within most facilities; truly the most important chess piece in a real OT attack aiming at complete domination.”
Dor Yardeni, Director of IR and Threat Analysis, OTORIO:
“We were contacted by one of our customers who needed help during an attack that was already in its advanced stages. By that time, Most of the servers in the IT environment had been compromised, but the OT environment was considered safe. First, we identified a malicious c&c connection initiated from a malicious document on the remote desktop server. We observed several privilege escalation techniques that gained the attacker privileges of a service account, which was a dedicated account for running scripts only on a specific machine. Still, because of bad configuration, it had local administrator privileges on all IT servers.
At that point, the attacker was executing a malicious loader on hundreds of servers and computers. We detected some connections and artifacts that indicated that the attacker found a way (due to bad firewall configuration) to execute code on one of the engineering stations in the OT environment, which was accessible to some PLC’s.
After a couple of days of investigation using one of our advanced threat intel tools, we discovered that the attacker was trying to sell access to the OT network on a famous hackers forum on the darknet. Luckily, by that time, we had already been in the remediation and mitigation phase and were able to thwart the attacker’s efforts.”
Jules Vos, Head of OT Security Services, Applied Risk:
“For a new mooring system a cyber security risk assessment was held as part of Management of Change (MoC) process. The mooring system was replacing an obsolete system on a very large vessel, which was permanently moored with 16 mooring lines. The old mooring system, after initial start-up, had never been operated satisfactory and had been switched off after 6 months. In the 6 following years the system had been out of operation.
During the MoC risk assessment the question was raised how business critical this system is. Everybody including the marine authority, operators and engineers agreed that the system was not critical. It had been switched of for 6 years and everybody felt with the regular inspections they had done that they managed risk very well. So they could do without. Then the following scenario was brought to the table; Assume the new system, after successful commissioning and after 2 years of satisfactory operations generate an alarm that 3 mooring lines are failing. A quick inspection does not reveal if the alarm is true or false. What do you do? Everybody agreed that in that case the emergency procedure would be triggered which could potentially lead to abandoning the vessel. So based on this scenario the criticality of jeopardising the data integrity of the system (e.g. through malware) changed from low to highly critical. This real life example demonstrates that during cyber security assessments participants shall be challenged to look at risks from different angles.”
Damon Small, Technical Director of Security Consulting, NCC Group:
“NCC Group was hired to perform an architectural review of a production OT facility. This assessment included a physical examination of all network and computer assets. The consultants found an undocumented cellular modem. The operators believed it to be related to the closed-circuit TV (CCTV) security system. Further research showed that the modem allowed another facility to remotely monitor CCTV after business hours while no operators were onsite. Although it served a legitimate and important function, this device was installed without the knowledge of IT or OT leadership and could have provided a vector for unauthorized access. Such shadow IT projects are common and are typically the result of OT trying to solve a peculiar problem for which IT may have limited experience. It is the consultants’ job not to say, ‘No, you can’t do this,” but rather, ‘Here is how you solve this problem in a way that minimizes risk to your organization’.”
John Cusimano, Vice President, aeCyberSolutions:
“We were conducting a cybersecurity assessment of a medical device manufacturing facility. As is often the case in manufacturing facilities, there was not clear segmentation between the business (i.e. office) network and the plant floor (i.e. production) networks. However, an attempt was made, through configuration and group policy to prevent manufacturing computers from accessing the Internet and to handle patching differently. For example, IT was responsible for patching all Windows computers in the facility but were specifically requested not to automatically patch some specific computers in the inspection lab. These were older computers running outdated OSs because the application software required was not available on the current OS.
During the course of the assessment, the lab manager angrily complained to us that, even though they weren’t supposed to, IT was continuing to try to patch their computers. They knew this because the computers would ‘blue-screen’ every time a patch was installed. Upon investigation it turned out that IT was not deploying patches to these computers. In fact, what had happened is that the lab staff had reconfigured their computers to access the Internet (presumably for personal reasons such as email and browsing). When they did this, they inadvertently enabled auto patching because Windows was configured to automatically download and install patches.”
James Green, Director of Field Services, Verve Industrial:
“We were hired to do a security assessment after deploying our solution to a large manufacturing facility. One of the first things we do is interview site employees to get an understanding of their current processes. In multiple interviews we were told they do have a procedure for an “emergency” patch for new “high profile” vulnerabilities, and asked if it had been used recently. Once we started digging into the data we found a number of devices that still had the vulnerability alive and well although they had deployed an emergency patch throughout the environment. These devices were sitting behind another firewall in the plant which is why they were missed, there was no visibility into this system from the outside, or so they thought. On top of the Vulnerability still being present, we found a number of devices that an outside connection, completely circumventing the firewall directly to IT. We see things like this all the time and they’re almost always caused by inexperience or a rush to get things done. It just highlights the importance of having a full asset inventory and making sure you have continuous visibility.
During an initial site walkdown we noticed a few rogue devices which were unknown to the site staff. This is becoming less and less typical as companies continue to highlight the risk of external drives but we do still see it. Most of the time it’s caused by needing specific files for upgrades, downloading log files, configs, etc…. In this particular case, after doing discovery we actually found a hotspot plugged directly into the controls system that had been physically hidden on top of a cabinet. Things like this aren’t necessarily surprising, we constantly find shortcuts that have been put in place to make work easier/more accessible for Engineers/Technicians without considering all of the security implications. Again, highlighting the need for full asset inventory and continuous visibility.”
Ron Pelletier, Founder and Chief Customer Officer, Pondurance:
“We were brought into a manufacturing company to help them with an investigation into a ‘rougue’ IT administrator that was threatening to hold them hostage. He knew all of the passwords to the systems and was not going to share them if they terminated his employment. We deployed our network sensor in a clandestine way, under the auspices of a compliance audit for GDPR, and we were able to gather the admin passwords through password cracking. Interestingly, when we enabled the sensor we immediately found two bitcoin mining operations in progress without their knowledge. Manufacturing companies usually have a rich level of bandwidth and system processing due to their massive supply chain systems so this was not a surprise for us.”
Dmitry Darensky, Head of Industrial Cybersecurity Practice, Positive Technologies:
“While servicing the equipment of the Network Traffic Analysis (NTA) system, engineers with the developer company drew attention to an incident registered by the system. It was not taken into operation by the engineers of the first line of the SOC and carried only the ‘detected’ status for two weeks. The system found that SCADA servers at sites in different regions had been seeking to establish network sessions with servers within the armed forces in several states.
Curiously, the firewalls and IDS systems blocked all attempts to establish network connections, but did not send alerts to the SOC. During the subsequent investigation, it was discovered that the firewalls also did not log the unsuccessful connection attempts. The servers were restored, but the source of the compromise could not be identified.
Most industrial enterprises today cover corporate segments of the IT infrastructure with SOC services, but OT segments remain in a blind spot. In addition, corporate SOC specialists don’t have the necessary expertise to analyze events and respond effectively to security incidents within OT networks. This clearly affects the level of protection of critical production assets within industrial companies.”
Krish Sridhar, Director, aeCyberSolutions:
“At a manufacturing plant in the south east a ransomware propagated from the IT business network to the Process Control Network (PCN). Thanks to a firewall rule the malware on the infected machines was unable to contact the main command and control server and hence the impact of the infection to operations was contained. However the incident prompted the plant personnel to review their current PCN configuration. Due to poor network design and lack of documentation the plant personnel struggled to determine what Industrial Control System (ICS) machines where on which network. The issues were exacerbated due to installation of ad hoc network devices and connections over the years with very little documentation available to their purpose and functionality. The risk assessment team along with the plant network engineers started to whiteboard a network diagram to better understand the current configuration and device connections. During this effort, the plant network engineers often disagreed with each other on the as-built network configurations and functionality. While the whiteboard effort was valuable, many questions remain unanswered and unclear.
This experience underscored the importance of deploying robust and resilient process control networks to limit malware propagation and help troubleshoot and respond to network related incidents.”
Dennis Lanahan, VP Sales, Critical Infrastructure Markets, Owl Cyber Defense:
“The critical infrastructure sector has always been security conscious, and even more so in the last year. Recently we’ve seen operators taking their security postures even further, to the point of air-gapping entire facilities, or the facilities have been air gapped all along for security purposes. In the last month we’ve spoken with a chemical manufacturer and a large oil and gas operator who now need to share the data from the segmented OT environment to the outside world. Sharing data to either their own data centers and even to the cloud infrastructures. They want and need the best of both worlds: Hardware-enforced, air gapped and isolated secure OT networks, and the ability to securely share the data between network domains. We’re talking with these companies now about a way to send data from a secure segmented facility to a corporate network (or a cloud service), without making a network connection between the domains.
We’ve also heard from critical infrastructure organizations who are worried about unsecured connectivity that is starting to come built into their IOT/IIOT devices. Plant operators are finding out that their IOT/IIOT assets have cellular or Wi-Fi capabilities that they didn’t know existed. Owners are understandably worried about the data that might be sent out over those connections. In the short term, this means security teams have to sweep their facilities with specialized RF scanners, looking for rogue connections. In the long term, we think the answer is to build security technology into IOT/IIOT devices alongside connectivity – a small, embedded data diode inside the IOT/IIOT device that allows the owner to control what information can travel over those connections.”
Related: ICS Security Experts Share Tales From the Trenches
