2020 and 2021 were watershed moments in the cyber security risk of industrial organizations. This was most evident in the power industry where cyberattacks increased at an alarming rate and new government regulatory efforts such as President Biden’s 100 day plan and the new TSA pipeline guidance created an even greater focus than in the past. Energy was the third most targeted industry in 2020 (from 9th in 2019) and has increased even greater in 2021.

Power and other industrial organizations are under what we call an “AIR-RAID” of OT cyber security challenges:

  • Attacks: OT systems are under increasing attack not just from nation-states, but from commercial hackers who threaten industrial operations and profits in return for ransom payments.
  • IT: OT systems are becoming much more dependent on IT systems – and their inherent vulnerabilities
  • Regulation: regulators are increasing the focus on industrial operators both in the United States as well as around the world with increasing implementation and reporting requirements
  • Resource-constraints: OT is already resource constrained in the day-to-day operations, and the 50-75% increase in ICS vulnerabilities in the past 12 months makes it almost impossible to keep up with the threats
  • Access: the “air gap” is dead…if it ever existed. This increased connectivity significantly adds risk to these “insecure by design” networks
  • Insurers: cyber insurers are now requiring much more robust OT security reporting to maintain coverage. This significantly increases the pressure on OT security practitioners to demonstrate consistent maturity
  • Directors: Boards have realized the significant threats from ransomware to financial operations and are now placing greater requirements on CISOs and operations leaders to demonstrate improvement in OT as in IT

And just at the end of 2021, the Log4j vulnerability dramatically increased the challenges for OT cybersecurity personnel. This is an insidious vulnerability as it exists within libraries used by a very large number of software vendors. Finding that library on sensitive OT systems is incredibly challenging. Unfortunately, this is only the latest in a series of challenging risks and vulnerabilities which have been arising more frequently.

Most industrial organizations are not prepared to address these upcoming challenges. OT security is often disaggregated into vendor-specific approaches or left to plant I&C or other technicians to manage, corporate cyber security teams often do not have senior representation from the operations side of the business, data is disaggregated into different silo’d tools, and risk information is often outdated due to a lack of automated ways of gathering 100% risk visibility in sensitive ICS environments.

This results in increasing cost faster than budgets allow, slow and disjointed reporting information, stretched OT resources, and boards of directors are getting more annual assessment reports showing significant OT cyber security risks. The current status quo is not tenable as these new cyber security realities emerge.

The good news is there are success stories out there from organizations that managed to get ahead of these challenges and create efficient and effective OT security programs that provide both security advancements as well as robust compliance across a wide range of assets. The following case study is how Verve Industrial worked with one such power organization to provide key technology and talent in this journey.

 

The Challenge: Efficiently secure different asset types

Over the past decade, Verve has worked closely with a leading North American power company to help them achieve efficient and advanced OT cyber security and compliance with cyber regulations. This integrated utility has a range of different types of generation assets – coal, gas, wind, solar and hydro – transmission and distribution, natural gas storage and distribution. Given the size and range of these assets, the organization adhered to a range of NERC CIP requirements to move beyond compliance to true security.

Like similar power companies, they began their OT cyber journey in pursuit of NERC CIP compliance. The company focused on its high and medium-impact assets – transmission control centers, large generation sites, etc. It addressed the core elements of NERC CIP using both people, processes, and technology. It let each business unit pursue its own strategy to achieve those objectives given the differences in each environment. And, similar again to many other large utilities, the low-impact assets received less focus and attention originally.

 

The Plan: Define what “good” looks like

But over the past three years, the focus shifted from compliance to greater cyber security maturity across the enterprise. They saw the writing on the wall several years ago and began a focused journey to improve the maturity of all their OT assets. They needed to find a way to efficiently secure at a higher level the “other 80%” of their assets – generation, distribution, gas, etc. – as well as expand beyond simply compliance on the high and medium impact assets.

This began an OT cyber security program journey led by Verve.

Step 1: Align leadership

The first step in that journey was organization. The CISO and senior team realized that to achieve true IT-OT security and commitment from the operational leaders to a change in direction, OT would need to have a “seat at the table”. Their approach included naming one of the senior generation engineering leaders into a corporate cyber security role overseeing all cybersecurity architecture with a specialized oversight of OT security for the enterprise. This blended role put OT at the center of the IT cyber security team, ensuring that every discussion had a senior representative representing that point of view. It also formed a focal point to align the various OT cyber security initiatives in each division into a common framework and strategy.

Step 2: Set program requirements

The second step was to define a future state set of requirements for the OT cyber program. This list evolved but eventually landed on three principles or requirements which were core the overall program:

 

A single, enterprise, vendor-agnostic 360-degree risk view of all assets

One of the first challenges was consolidating the risk view across the wide range of distributed and varied systems. The organization had dozens of different OEM systems across their estate: Emerson, ABB, GE, Vestas, Rockwell, Schneider, etc. plus multiple types of networking equipment both IT and OT. Different divisions took different approaches to managing these assets – or not managing as the case may be. The enterprise ServiceNow instance was blind to these assets so tracking basic systems management functions was impossible.

The resulting costs and ineffectiveness of this situation became clear as regulators, insurers and the board increased data requests from the teams. There just weren’t enough resources to constantly be checking on the latest vulnerability or the current patch status of devices or whether certain accounts were present on all systems. They needed a single solution to centralize visibility and actionability across these systems to take a consolidated risk view and reduce the cost and complexity of using multiple different OEM tools or manual processes.

In addition, the organization had conducted surveys and manual vulnerability and risk assessments in the past.  They discovered these were always out of date and incomplete given the lack of asset-level visibility. Similarly, they knew they would not be able to patch every vulnerability immediately, especially on many of the embedded devices. They needed to make trade-offs between mitigating controls as well as prioritize how “at-risk” each asset was.  Therefore, the requirement was to provide a real-time view of comprehensive risks: patch, vulnerabilities, users & account risks, insecure configurations, etc. And critically, this must be accomplished with no scanning of the sensitive OT systems.

The goal was to leverage a single platform that could provide a 360-degree risk view on every OT asset and network, and integrate that information as necessary into ServiceNow for IT analysis as necessary.

How COVID-19 is affecting manufacturing cybersecurity

 

A common OT security platform to grow into maturity over time

No organization can jump to “Level 5” maturity overnight, but continuously adding new tools and interfaces as needs evolve isn’t scalable. Critical to the organization was a platform that could expand into new cyber capabilities and integrate the various elements necessary into a single dashboard. This was part of the overall need for efficiency and low-security Total-Cost-of-Ownership.

In the past, the tools and approaches used in OT security left a gap between identifying risks (e.g., manual vulnerability assessment) or threats (network anomaly detections) and the remediation and response to those. The organization wanted to quickly make progress on remediating its risks. Therefore, the approach needed to integrate the ability to make take remediating actions at the endpoint and network. It needed to provide “systems management” not only detection.

It also needed the ability to conduct robust threat detection and response. The company had deployed network taps and span infrastructure to monitor for network anomalies. But it needed to complement this with endpoint detection and response as well as integrate all of this into a common OT-specific threat detection platform. Importantly, they wanted to do this without exploding the costs of their Splunk instance by overwhelming it with more log data.

Enterprise IT Security Management Platform

 

Deep OT expert support resources

Their resources were already stretched thin with outage schedules and maintenance. They needed a partner who immediately understand their control system environment, including the range of different vendors’ equipment. The current industrial controls’ support vendors did not have the level of cyber security knowledge necessary to address necessary security topics, so the organization needed that OT cybersecurity integrated expertise.

Although they did create a dedicated OT cyber team at corporate of about 3-4 people, they needed external support to manage the ongoing security maturity at sites. This team was required to understand control systems and bring the latest intelligence and capabilities from the cyber security community.

 

The Solution:

Delivering on these requirements required adopting what Verve calls a comprehensive vendor-agnostic “OT Systems Management” (OTSM) approach to security that addresses the above requirements. OTSM applies similar security and management that the industry has applied to IT for years.  Think Global: Act Local

The OTSM approach begins with an architecture “fit for purpose”; one that addresses the cost, resource, risk complexity, and operational reliability challenges of OT security. We call this “Think Global:Act Local” (TG:AL).  TG:AL enables scaled visibility, analysis, and planning but ensures operation reliability. The structure is seen below in an example of a global organization.

The first goal of TG:AL is to address the labor and efficiency challenges associated with OT security. According to KPMG, the number one challenge of ICS security practitioners is the lack of skilled and knowledgeable OT security personnel. This was especially true at this power company. Remote locations further added to the challenges of having personnel at a site that could address the wide range of security requirements necessary for maturity. Budget constraints also meant that they needed to have a very low total cost of ownership of any security initiative.

Further, one of their key requirements was to be vendor-agnostic. They did not want to rely on islanded different vendor tools. They realized that to deliver efficiently and comprehensively, the platform needed to provide visibility across vendors. Verve’s architecture is vendor-agnostic and gathers all the various systems into this central database, without disrupting the operations of those systems.

Finally, they needed a solution that would cut across systems from gas pipelines to electrical distribution to coal-generated power plants. The solution needed to scale to meet that breadth of needs.

The client had the opportunity to evaluate many different tools during their years of achieving NERC CIP compliance. They saw many configuration management solutions, network anomaly detection, IT-oriented vulnerability management, ServiceNOW, Splunk, and others.

From this experience, they chose Verve as their partner in large part due to its TG:AL architecture. TG:AL aggregates comprehensive data from every asset into an enterprise database so that a small number of centralized resources can analyze risks, vulnerabilities, and threats.  Without this architecture, each plant or site needs to hire or train personnel knowledgeable about a wide range of security elements, leading to significant costs and gaps in expertise. Again across all vendors.

The “think global” component delivers a 70% lower labor cost than more traditional OT security solutions. However, cost and coverage are not the only priorities.

The solution must also ensure operational reliability. Everyone in OT has their story of how IT tried to scan or patch or make changes from a central point and tripped the plant or worse. The architecture must allow for control by those closest to the process.

The “Act Local” component delivers this reliability. The enterprise database is read-only, allowing analysis but no inbound actions. The playbooks are distributed to the local process controls environments. There, before any action occurs, the operator with knowledge of the process decides on the timing and approach to executing the actions. This ensures that well-meaning teams do not cause unintended harm to the process in the pursuit of security.

Importantly, these actions are automated so that the operator does not need to manually take the actions. He or she makes a few clicks in the software and hits execute to deliver the changes once tests and timing have been approved.

Verve built this architecture working together with global industrial clients that needed this balance of efficiency and reliability.

 

Technology-Enabled Vulnerability Assessment

The TG:AL architecture enables the broader OT systems management program. That program begins with a robust, endpoint level “360-degree risk assessment” to provide a comprehensive, real-time view of the OT risks on each asset in the environment. Generating this 360-degree view cannot be done through packet inspection as much of the important data necessary is not communicated over the wire – e.g., patch status, all installed software, users & accounts, etc. The 360-degree view is used to assess the overall risk score of each asset to provide a prioritized remediation roadmap that addresses the various challenges in OT security, such as the inability to immediately patch.

This forms the foundation of a “Technology-Enabled Vulnerability Assessment” (TEVA). which is an alternative solution to a packet inspection approach. The TEVA provides significant advantages over traditional survey or manual risk or vulnerability assessment approaches.

First, it provides detailed information on each asset in the environment, rather than high-level findings based on individuals’ understanding of the environment. One of the challenges of OT is a lack of data to draw on for these assessments, unlike IT which usually has at least data on patch levels or vulnerabilities. So TEVA gets at the missing data.

Second, it is significantly less time-consuming and expensive because the technology replaces the human labor required to travel to sites, interview personnel, walk down networks and manually capture information.

Third, TEVA occurs in real-time, avoiding the common problem of vulnerability assessments that are aged before you begin the remediation and cannot provide updated progress information as changes are made. Finally, once the technology is deployed, remediation can be immediate as soon as risks are identified. The organization does not need to wait months prior to beginning to execute on security.

Tech-enabled vulnerability assessments identify many risks for quick remediation:

  • A significant number of dormant accounts were caused by leftover vendor-created accounts during the prior maintenance outages. Since the environment had no central AD server, these local admin accounts create access points for attack.
  • Lack of updated Anti-virus signatures
  • Missing backups where the installed backup tool had failed or filled to capacity
  • Insecure, unnecessary software programs. Although OEMs often threaten to not support systems if unapproved software is installed, we always find dozens of these applications, from LogMeIn to DVD burners, iTunes, and others.
  • Firewall rules have weakened over time as increased access is granted, creating openings for attackers to exploit.

In addition to these, TEVA also identifies risks that may be more complex to remediate immediately:

  • Vulnerabilities & Patches. Some of these can be addressed quickly, but others are not approved by OEM vendors or require rebooting of key operational equipment.
  • Cleaning up dual nic’s. In many environments, vendors and operators create separate nic’s to route traffic around firewalls for various reasons. Remediation may require a short or long outage to reroute traffic through reconfigured firewalls or remote access gateways, leading to longer lead times

But the advantage of the 360-degree view is that the organization finds compensating controls that address those longer-cycle asset risks by hardening them in other ways.

 

The 3R’s – Remediate, Respond, Recover

Assessment is only the initial step. OTSM, as its name implies, requires MANAGEMENT of assets and networks. Or what we call the 3-Rs of cyber security: Remediate, Respond, Recover. These 3-R’s separate security from monitoring and they are critical to demonstrate the kind of security improvement that energy and other industrial organizations require. They are not “passive”. They cannot be achieved through monitoring traffic through span ports. They require meaningful interaction with OT systems to protect them and stop threats before they spread.

IT has been managing systems for years, and in fact, over 75% of all the cybersecurity jobs are related to “systems management” according to NIST’s NICE CyberSeek database.

NIST's Cyberseek database of cybersecurity job openings by task

We need to apply these same principles to OT while recognizing the sensitive systems and their unique requirements.

In OT this can sound like heresy – “If it ain’t broke, don’t fix it”. But in fact, OT is becoming more and more dependent on IT systems as virtualization and IIOT expand. OT needs to begin to manage these assets not only for security but also for operational reliability. Organizations can no longer rely on “airgaps” to protect unpatched, insecurely configured assets.

OTSM takes an OT approach to these systems management requirements. It enables rapid remediation through a platform that can patch, harden, manage software and users, update AV or whitelisting, etc. Instead of relying on inefficient and inconsistent manual processes, OTSM automates remediation actions to accelerate mean-time-to-remediation. The organization could centrally identify the risks and immediately develop common remediation playbooks to distribute across sites to fix security gaps. These playbooks are executed by the “Act Local” team on-site using their understanding of the process to test and approve the timing of executing the action. Confirmation of the action is immediately available to the “Think Global” team to ensure compliance and security resolution.

Similarly, when the SOC or “Think Global” team identifies a potential threat, it has the data to dive into that event to understand the context of the assets impacted in greater detail. Prior to Verve, the organization had alert-fatigue and blunt instrument responses because of a lack of deep endpoint and network data. Using Verve, the Think Global team could dive deep into the assets to understand where the threat may move and identify response actions.

This deep analytical view allows the central team to define what we call the “least disruptive response” to any threat. As an example, Verve detected a potential brute force attack leveraging a specific set of accounts.  The 360-degree view of the assets, including all account data on every machine, including dormant accounts, enabled the “Think Global” team to identify which systems might be impacted and to design a playbook to remove those accounts. The “Act Local” resources then confirmed that these accounts were not critical for operations and executed the action within hours. This is just one example of the power of the “Think Global: Act Local” architecture.

Finally, no security program will protect or respond to every threat. There will be the day when the attackers beat the defenders. The last line of defense is the third “R”, recovery, i.e. restoring systems after an event. The OTSM program enables this process by aggregating backups across different OEM vendor systems. It tracks the backup status to ensure the organization is ready to recover when needed. Verve helped this power organization establish an OT-specific backup suite to cover its traditional OS devices, networking gear as well as many of the critical embedded devices.

 

OT Expert Support

Cyber security is NOT just technology. Although TG:AL allows for scaling resources, this organization still needed additional external support for deployment and bringing security best practices to the organization.  One of their big challenges was that the current ICS support vendors did not possess deep cyber security knowledge and were relatively vendor-specific. Key to the success of OTSM is cyber security personnel that understands system differences, but the security commonalities to drive consistency, efficiency as well as operational resilience.

Verve has a 30-year legacy of vendor-agnostic control systems design. Over the past 15 years, we expanded this expertise into cyber security, maintaining that vendor-agnostic approach. Verve’s teams work with customers to help conduct technology-enabled vulnerability assessments, across the different vendors in the environment. They develop risk reduction roadmaps across an enterprise, down to a plant level. They assist in remediating such as network segmentation, patch management, configuration hardening, etc. – across different vendor systems.

Segmentation is a perfect example where a vendor-agnostic approach is critical. These systems interact, and the design team needs to understand how the different systems work so that the right network architecture can be designed. Verve’s team possesses the capability to draw those cross-vendor insights.

Verve’s team supports the client across all their OT assets by providing managed OTSM services using the Verve platform. The team adds additional risk assessment, remediation, and response resources to the OT cybersecurity team and works hand-in-glove with them on a real-time basis.

The Result: Reduced time to threat detection and response action across entire OT environment

The client’s mission was to create an efficient and effective way to bring consistent cyber security to their entire OT environment. Together, Verve and the power company aggregated data across wind, solar, hydro, fossil, nuclear generation, distribution, and gas business units. We provided a single view for risk analytics that updates in real-time, radically reducing the cost of ongoing security assessments. The time to respond to requests from the board and insurers dropped from weeks to hours. In addition, we significantly improved the real-time detection of threats by adding endpoint behavioral data to their prior detection methods to reduce false positives and identify response actions more quickly.

The platform now manages dozens of sites across a wide geographical footprint driving efficiency.  Perhaps most importantly, there is now a common understanding of the OT risks to aggregate with the IT risks into a single risk view for the organization. All with limited cost of additional headcount due to the use of the TG:AL platform.

Learn more

Contact Verve to make progress against your OT cybersecurity challenges.

Contact Us

Related Resources

Case Study

Distributed Generation Energy Case Study

How Verve Industrial serves the distributed generation energy industry with asset management and centralized reporting for effective OT security.

Learn More
Case Study

How a top energy company rapidly improved OT cybersecurity maturity

See how a top energy company delivered measurable improvement in OT cybersecurity within a 12 month period.

Learn More
Case Study

Remediation in 90 Minutes with Think Global, Act Local

Achieve remediation in 90 minutes! A real-life case study tackling challenges of large and complex assets, coupled with scarce OT security resources.

Learn More

Newsletter Registration

SIgn up for our bi-weekly newsletter to receive the latest case studies, blogs and content to your inbox!