The Need for Integrated ICS Risk Management

In 2018 and 2019, Verve conducted dozens of ICS risk assessments for new customers in OT (operational technology) industries ranging from power generation, transmission and distribution, pharmaceuticals, CPG manufacturing, water, and gas distribution.

During these risk assessments, we found five commonalities, solidifying the case for integrated risk management for industrial control systems.

5 Key Findings from ICS Risk Assessments:

1. The average industrial site has over 1,000 critical vulnerabilities and hundreds of missing critical patches. Interestingly enough, even the companies regularly applying OEM-distributed and approved software patches still had hundreds of vulnerabilities present because the distributions did not include many important third-party application patches such as Adobe and Java.
2. Only 15-20% of the companies leveraged rigorous network segmentation for increased protection. While almost two-thirds had hardware capable of providing some level of network protection, many of these were improperly or incompletely configured and designed, reducing the security effectiveness of these network protections.
3. The vast majority of embedded OT devices such as PLCs, relays, RTU’s, VFDs, meters, controllers, etc. had very limited or no published vulnerabilities. However, most of the embedded OT devices showed unpublished risks (i.e. buried in their firmware such as un-published VxWorks) or insecure configurations that cyber attackers could take advantage of to negatively impact critical operations.
4. The average workstation/HMI contained multiple dormant or unnecessary users and accounts and failed roughly one-half of the configuration hardening checks conducted by the Verve Security Center.  Together, this creates a significant opportunity for inappropriate access without ample protection.
5. Approximately 80% of the companies did not have tested, prioritized incident detection, response, and recovery tactics in place. This included a range of gaps including the lack of tested, automated, and up-to-date backups, the lack of asset criticality view, and the lack of plant-based personnel awareness into the types of operational issues that might trigger an incident escalation.

In almost all cases, the task of remediating all of the identified risks is overwhelming for a limited staff who do not have significant training in security operations or systems management.

Furthermore, as we all know, many OT systems cannot be patched immediately, and certain older systems do not have user-level access control, making traditional IT-based remediation strategies inappropriate in OT. Additional resources are required and often allocated. Over time, most systems can be patched and hardened appropriately.

However, the central question is: how do we achieve the greatest risk reduction for the time and money available? Across all of the findings – not only at the broader site level but at an asset-by-asset basis – what remediating actions applied to what systems or assets are most critical to reducing the risk to operations? And, how do we sequence activities so that we continue to increase our maturity over time?

What is 360-Degree ICS Risk Management?

Our answer to these questions is what we refer to as 360-degree ICS Risk Management. This approach has two broad themes:

First, provide true risk prioritization and remediation planning by taking a “360-degree” view of each asset in the environment. “360-degree prioritization” implies looking deeper into the asset’s attributes than simply the OS and known vulnerabilities and where it sits in the network.

“360-degree” implies identifying those things plus: all user and accounts both dormant and in-use, all current endpoint protection and its status and recency of update, configuration settings to understand whether those configurations are hardened or not, the criticality of that asset to the overall process, the recency and accuracy of backups, whether or not devices have dual NICs that may allow for routing around network protections, etc. This 360-degree view allows one to then calculate a true risk score for that asset, relevant for that process.

With a 360-degree view and a calculated risk score, operators can effectively prioritize their remediation plans. Which assets need to be patched first? Which are protected by compensating controls and, therefore, may be the lower priority? Which have other risks such as user/account/access risks that would not be seen by traditional vulnerability scan? Which can be further protected by locking down application whitelisting rules which are too lenient?

Analyzing each asset for its relative risk and developing remediation actions by asset may sound complex, but the alternative is an overwhelming complexity not in the analysis but in the operational execution of these initiatives at the plant level. Through centralization (what we call “think global”) and automated scoring, this analysis can be streamlined and made significantly more efficient. But the second theme of 360-degree risk management is also necessary.

Second, integrate the 360-degree risk prioritization with “closed-loop remediation management”. Current approaches for ICS vulnerability assessment and remediation are too time-consuming. After conducting a vulnerability assessment, either by scanning or manually checking vulnerabilities, operators tend to implement OEM-approved patches in addition to testing any non-supplied patches on their systems and rolling those out individually.

The alternative in many cases is risky when pushing patches from WSUS or other tools. As for other risks, such as user access, configuration hardening, etc., each requires manual intervention on a device or linking to an active directory which may not be available. Finally, monitoring the risk involves looking at multiple different screens and tools such as whitelisting, AV, network detection, etc.

We encourage operators to adopt closed-loop remediation management to link data from the assessment function directly to automated remediation capabilities. The vulnerabilities identified by the unnecessary risky software, the dormant user accounts, and the misconfigured network appliances are linked directly to the ability to take action to remediate those threats.

A “closed-loop” system is important to ensure there is no gap between the “Identify” component of the NIST Cyber Security Framework and the “Protect” component. This approach significantly reduces the required labor to mitigate the risks identified. To be clear, we are not encouraging centralized/global actions.

We recommend a “Think Global-Act Local” approach to scale risk analysis and remediation planning while placing the automated tools into the hands of local technicians most knowledgeable about their plant and its systems to execute the final actions in an automated manner.

3 Ways to Reduce Risk for OT/ICS Security

From asset/device visibility to OT system inventory 

A holistic 360-degree perspective of the entire system and all of its hardware, software, connectivity, users, etc. is necessary for comprehensive risk assessment. To do this, it’s important to move beyond the basic knowledge that a device is present on the network and what OS version it is running. Although that is a necessity, it is not a sufficient capability. You’ll need to gather data on all application software such as its versions, the user accounts, configuration of networking gear, secure settings on Windows equipment, etc.

From vulnerability identification to ICS/OT risk assessment

Identifying known vulnerabilities is necessary, but, again, it is not sufficient in industrial control systems. Understanding the full risk of an asset is critical in OT. Risk includes things such as unnecessary and risky application software (even if up to date), dormant or shared user accounts, insecure configurations, gaps in network protection, devices that have insecure design “built-in”, etc.  These risks may be partially offset by compensating controls such as firewall protection, application whitelisting, very tight user access control, etc. True 360-degree risk assessment takes these different components into account to form a risk score on each asset to assess its relative risk score. When combined with the process criticality of that system, a true risk assessment is possible.

From OT asset identification to true ICS risk remediation

As we all now know, resource constraints are one of the largest challenges in making significant improvements in ICS cyber risk maturity. We must move beyond just identifying potential risks and vulnerabilities to prioritizing specific actions that drive the greatest risk reduction most efficiently.

This includes two elements. First, we must prioritize actions based on the most efficient means of risk reduction using the 360-degree view above. Second, we must drive automation into the process to allow limited resources to deliver maximum impact. We must move from stand-alone tools that only provide detection or assessment, to tools and processes that then allow for automation of the remediation actions – but done in a way with local control over those actions.

By combining these three steps, we have the chance to drive significant, rapid maturity improvement even in a world of limited resources. The good news is that we have lots of opportunities to improve and practical steps we can take. However, it will require a change of mindset and methods to get there.