Technology-Enabled Vulnerability Assessment in Chemicals Production

Summary

A global specialty chemicals producer needed to assess their fleet of 60+ manufacturing facilities. After reviewing alternative approaches, the company chose to partner with Verve using its technology-enabled vulnerability assessment approach.  What was originally projected to take at least 18-24 months and cost millions of dollars, was completed in less than half the time and cost. And perhaps most importantly, the organization immediately began the remediation of risks because of the use of the Verve Security Center for the assessment effort.

Verve’s technology-enabled vulnerability assessment significantly reduces the time and labor requirements while enabling real-time, ongoing visibility to track progress and gain continuous visibility into your risks. This case study describes how we applied this approach with a leading chemicals company to quickly prioritize security gaps and remediation efforts.

The Challenge

As threats to OT continue to increase, a global chemicals producer embarked on a cyber security maturity effort. The initial stage of the company’s program was to conduct a robust cyber risk assessment across each of its 60+ global sites. Conducting vulnerability assessments is often a costly, manual effort requiring on-site resources which, as a result, only happens infrequently. The approach offered by the company’s traditional IT consultants included a manual approach to visiting each site, conducting surveys, reviewing network diagrams, and eventually summarizing this into an overall roadmap. The company was concerned about the cost, duration, and depth of such an approach would take.

The company was looking for a cost-effective solution that would provide a detailed assessment without the significant cost of manual processes. They realized that traditional IT tools are risky to deploy in OT, so they needed a cost-effective, detailed, rapid, and OT-specific approach

The Solution

The company chose to partner with Verve to leverage the Verve Security Center and our 30 years of ICS/OT experience to develop an assessment and roadmap for each site as well as the overall organization.  Verve’s solution leverages the Verve Security Center, a truly unique approach to conducting robust OT/ICS security assessments. Verve brought together the Verve Security Center platform and our Verve Industrial Protection (VIP) services to provide a rapid, detailed, and “closed-loop” solution leveraging technology-enabled vulnerability assessment.

The Verve Security Center includes comprehensive asset visibility and risk assessment across all elements of an environment’s risk.  This includes the ability to see detailed information from PLCs down into the backplane and the devices behind the backplane, gather visibility of all software, patch status, vulnerabilities, users and accounts, ports, services, configurations, etc. The Verve platform then analyzes the software and firmware for known vulnerabilities and insecure or non-compliant configurations. The platform also allowed us to analyze network gear for both misconfigurations as well as inappropriate security rules.

Verve is a comprehensive OT Security Management Platform as shown below:

It begins with a core “engine” of asset inventory. This view of inventory requires no span ports, taps, or network collectors. It goes directly to each asset to gather data. Therefore, that data is detailed and complete. It does not estimate information based on traffic interpretations. It provides IT-like information without any risk to OT systems.

For example, the platform provides vulnerability visibility down into the embedded devices and the cards in the backplane of OT devices.

There are 5 key benefits of this “technology-enabled” approach:

1. OT safe:

Verve Security Center was designed by ICS engineers for ICS security. In other words, it was designed for operational reliability and creates no risk to operations from its deployment. One of the challenges of conducting technical assessments in OT is the risk to operations. Verve’s 30 years of control systems engineering work and capabilities are built into the platform. We have been using it across almost every industrial industry sector and with all brands of OEM equipment with no disruptions or impact on operations.  In fact, operational personnel often find the insights generated can help with the reliability of their system because it can identify potential network or system functionality issues that may plague operations.

2. 360-degree risk perspective:

Verve provides that full 360-degree view that enables comprehensive risk assessment and planning.  Verve captures a view of all of the site’s asset inventory and pulls that information back into the centralized reporting console. Unlike a “pen test” which may discover one path into the environment and highlight the need for better intrusion detection, Verve Security Center provides an assessment of all the potential threat vectors and the risks to the system.

Key information gathered includes:

• Rogue asset discovery of assets that aren’t included on the current inventory
• All installed software to review for risky items such as TeamViewer or other remote access applications, as was done by attackers in the Oldsmar water incident.
• Firmware on embedded devices
• Software vulnerabilities/CVEs to identify potential attack paths
• Patch status to determine whether critical patches are missing
• All users and accounts to determine the presence of dormant accounts that may enable attackers to leverage older credentials available on the dark web, as was done in the case of Colonial Pipeline.
• All configuration settings to compare to standards such as DISA-STIG as well as Verve OT-specific standards we have developed
• Status of key security & reliability software such as backups, Anti-virus, application whitelisting, etc. to determine whether attacks can be defended and responded to quickly
• Network connections and configurations to identify potential paths for external unapproved access as well as weak firewall and switch configurations that may allow the spread of malware or APTs

As a result of this comprehensive view, Verve provides a series of scenarios of potential penetration paths that an attacker can take.

Importantly, however, this “test” does not only provide a grade such as you flunked the penetration test and we got in in the following way (frankly, in almost every OT/ICS environment a true “pen test” is both risky as well as not very insightful because there are usually so many vulnerabilities and attack vectors that getting in and causing disruption usually is quite simple). The Verve approach, instead, provides very specific recommendations of what you need to fix – e.g., remove these 35 dormant accounts from the OEM vendor that haven’t been used since an outage 2 years ago, deploy these specific patches to address critical RDP vulnerabilities, harden the password settings to require changing them every 30 or 60 or 90 days, etc.  The granularity of the assessment enables a much more practical and efficient remediation roadmap.

3. Enterprise visibility across all sites:

Verve Security Center aggregates this data into an enterprise reporting console so that the assessment analysis can scale and provide the corporate security or process control team visibility into the risks across the environment.  One of the challenges to a multi-site assessment is embarking on is tracking all of the risks and assessment outputs.  Verve automates this entire effort. The technology leveraged at each site aggregates back to a central reporting for analysis, planning, and remediation play-booking. The benefits are improved efficiency and effectiveness of the assessment.

We call this approach: Think Global:Act Local. The ability to see all of the risks across the fleet centrally, but enable local control over remediation actions using the power of the automated tool.  The enterprise reporting console allows a smaller, common team to review the risks across sites around the globe. This drives efficiency in the analysis of all the various threats to the environment.  Further, instead of site-by-site analysis, the whole process goes much more quickly when the data is automatically aggregated centrally. This centralization also allows for improved consistency in the assessment. Because the same group of individuals is reviewing the data from all 70 sites, the consistency of risk prioritization and remediation planning is significantly better.

4. Accelerated time to security – assess & remediate in the same platform:

The Verve Security Center is not only an assessment tool, but it also enables the remediation of risk from the same console.  One of the biggest benefits our clients find from the Verve Tech-enabled Assessment approach is that it not only allows them to get very specific on asset risks but also to quickly move to remediation so they can demonstrate rapid improvement, rather than just pointing out problems.  One of the reactions we often hear from operations/plant personnel who aren’t part of the security team is that the assessment is just going to point out problems, and identify things we aren’t doing, most of which we already know because security has not been a focus in operations in the past. For instance, you aren’t conducting appropriate network segmentation, you aren’t monitoring for intrusions, you aren’t updating or patching older systems, etc. By leveraging the Verve platform, the assessment provides very specific information which may be valuable operationally – e.g., Did you know that the server you rely on for a key part of your operation has intermittent CPU issues and is almost out of disk space? Or that your vendor installed a 5G device in the backplane of a controller so they can monitor things without informing you, etc.?

But more importantly, the platform can then be used to fix the problems identified without waiting another 6 months for new tools, etc. to be planned and installed.  If the security and site personnel discover dormant accounts that should be removed across hundreds of devices, Verve can immediately automate removal – obviously with approval from the operations personnel. When you discover unapproved remote access software or DVD burners, iTunes, or even worse software running on HMIs and OT servers, Verve can immediately remove that software at scale.

5. Ongoing monitoring and assessment:

Many assessment methodologies are one-time, providing a point-in-time perspective of the risks of an environment. The day after the assessment is completed, the risk picture could look different if new vulnerabilities are released or changes are made to network configurations, or devices are upgraded in a typical upgrade process. Verve’s Technology-enabled Assessment approach provides an ongoing, real-time view of the risk in the environment. So, as an organization takes remediation actions, Verve’s Enterprise Reporting console is constantly updating information for all sites. The team can monitor as sites move from “red to green” on key metrics such as dormant users or insecure configurations in networking equipment or missing critical patches, etc. In addition, if changes are made or new external vulnerabilities are released (hundreds of which happen every week), Verve updates the risk scoring of each asset and site in the company.

Traditional manual or one-time assessment approaches provide limited lasting benefits. The Verve technology-enabled assessment provides ongoing value to maximize the investment made upfront in the initial effort.

Roadmap:

The key outputs of the assessment include a comprehensive view of the risks, prioritized for each site as well as across the enterprise.  The below “defense-in-depth” diagram shows representative types of risks we often find, although are not specific to this client setting.

Verve’s services team then leveraged this tech-enabled visibility to develop a prioritized risk ranking based on our proprietary asset risk score and the client’s disaster and physical-hazard analysis of asset criticality. This led to a clear prioritized road map of remediation initiatives – again a representative example is provided below, rather than anything specific to one particular client.

And this detailed roadmap can then be synthesized into a broader set of initiatives over time.

The Outcome

The overall time to completion of 60+ sites was approximately 9 months. This included not only the deployment of the Verve Security Center but also a review of all assessment outputs, prioritization of risk, and remediation roadmaps.

The initial phase involved deploying Verve and conducting the assessment at 4 pilot sites in different global geographies. This pilot effort accomplished several tasks:

1. The teams aligned on objectives and key work plans and approaches of how to work together. This is often an overlooked effort as partners and clients come together across OT environments. This was a critical phase to gain buy-in to the effort, understand the status of security efforts at a few sites, define detailed processes and work plans for future sites, etc.
2. The joint Verve-client team also used this to define a set of expected outcomes using these four initial sites as models.
3. The team deployed Verve at these 4 sites and gathered the detailed risk analysis, creating an agreed asset criticality score model.
4. And finally, this phase included a review of available IT and OT policies and procedures to identify gaps in current documented policies and procedures.

During Phase 1, we also kick off the governance and standards development process. This phase is portrayed along the bottom axis of our process chart above. The reason we extend it along during the entire assessment and roadmap process is that we find that the governance model and standards will evolve as we get deeper into the assessment processes across the sites. In addition, as we conduct the technical assessment and design the roadmap, the site-specific feasibility exceptions and requisite compensating controls need to be considered in the governance/standard model.

Then the team accelerated rapidly achieving 2-4 site deployments and assessments each week. The key to this is the speed of deployment of a software-based solution rather than one that relies on spans/taps/fiber runs/collectors/etc. Verve’s endpoint-based approach is unique to the industry in its ability to rapidly deploy, gain deep visibility, prioritize 360-degree risks, and then enable rapid remediation.

The eventual impact was a comprehensive risk view, but also a very specific roadmap for each site. For instance, using the Verve Security Center technology-based assessment means that the organization was able to get down to very specific patch strategies, specific accounts to be managed, asset-by-asset configuration hardening, etc.

The result is to be able to demonstrate a baseline security maturity against the given framework, such as NIST CSF, and to measure and track improvement over time.  Verve can deliver both the assessment and begin to remediate within a month. Within 30 days, clients can have a robust assessment and road map as well as make progress on remediation using the closed-loop platform. They are able to immediately demonstrate progress to their board against the specific gaps identified in the vulnerability assessment and now contract progress, update on new risks you know real-time basis because of the ongoing presence of the Verve Security Center vulnerability analysis software.