How a top energy company rapidly improved OT cybersecurity maturity

2020 and 2021 were watershed moments in the cyber security risk of industrial organizations. Perhaps this was the most evident in the oil and gas industry where events such as the Colonial Pipeline attack and resulting regulations emerging from TSA have raised the profile of these risks. But this is just one very public event in a string of recent increases in threat activity targeting the world’s energy infrastructure. Energy was the third most targeted industry in 2020 (from 9^th in 2019) and has increased even greater in 2021.

Oil & gas and other industrial organizations are under what we call an “AIR-RAID” of OT cyber security challenges:

• Attacks: OT systems are under increasing attack not just from nation-states, but from commercial hackers who threaten industrial operations and profits in return for ransom payments.
• IT: OT systems are becoming much more dependent on IT systems – and their inherent vulnerabilities
• Regulation: regulators are increasing the focus on industrial operators both in the United States as well as around the world with increasing implementation and reporting requirements
• Resource-constraints: OT is already resource-constrained in the day-to-day operations, and the 50-75% increase in ICS vulnerabilities in the past 12 months makes it almost impossible to keep up with the threats
• Access: the “air gap” is dead…if it ever existed. This increased connectivity significantly adds risk to these “insecure by design” networks
• Insurers: cyber insurers are now requiring much more robust OT security reporting to maintain coverage. This significantly increases the pressure on OT security practitioners to demonstrate consistent maturity
• Directors: Boards have realized the significant threats from ransomware to financial operations and are now placing greater requirements on CISOs and operations leaders to demonstrate improvement in OT as in IT

And just at the end of 2021, the Log4j vulnerability dramatically increased the challenges for OT cybersecurity personnel. This is an insidious vulnerability as it exists within libraries used by a very large number of software vendors. Finding that library on sensitive OT systems is incredibly challenging. Unfortunately, this is only the latest in a series of challenging risks and vulnerabilities which have been arising ever more frequently.

The Challenge: Improving cybersecurity maturity

One of the top five oil and gas producers in North America enlisted Verve Industrial to accelerate its cyber security maturity in the face of these rising challenges. Like many energy companies, this organization has a wide range of ICS vendor systems and, in fact, has one of the largest global installations for one of the major DCS vendors. The senior leadership recognized that they were a significant target, not just because of the safety or environmental risks they faced, but also as a ransomware target given the financial impact of a potential shut down.

The OT organization undertook several security initiatives. They implemented network security efforts to add perimeter protections but had little visibility into the ongoing maintenance of those protections or to the risks inside that perimeter. The team tested ‘network visibility and detection’ tools but was overwhelmed with the number of alerts generated by “anomalies” and missed many of the assets in the environment given dependence on the network packet information. Their large OEM vendor patched those systems using manual techniques, but it was time-consuming and did not cover all the systems or the vulnerabilities in the environment.

They stepped back and developed a comprehensive view to drive an efficient and effective cybersecurity program rather than a series of one-off efforts. However, because of the energy price swings during the COVID-19 pandemic, the organization faced significant budgetary pressures, and, therefore, any program needed to be efficient and low-cost to implement and manage.

These five requirements became core to the overall OT cybersecurity program:

• Vendor-agnostic
  • The organization had dozens of different OEM systems across their estate: DeltaV, Schneider, Rockwell, etc. plus multiple types of networking equipment both IT and OT. They needed a single solution to centralize visibility and actionability across these systems to consolidate risk view and reduce the cost and complexity of using multiple different OEM tools or manual processes.

• Real-time, comprehensive risk assessment
  • The organization conducted surveys and manual vulnerability and risk assessments in the past.  They discovered these were always out of date and incomplete given the lack of asset-level visibility. Similarly, they knew they would not be able to patch every vulnerability immediately, especially on many of the embedded devices. Therefore, they needed to make trade-offs between mitigating controls and prioritizing how at-risk each asset was. The requirement was to provide a real-time view of comprehensive risks: patch, vulnerabilities, users & account risks, insecure configurations, etc. And critically, this must be accomplished without scanning the sensitive OT systems.

• Rapid remediation
  • In the past, the tools and approaches used in OT security left too much of a gap between identifying risks (e.g., manual vulnerability assessment) or threats (network anomaly detections) and the remediation and response to those. The organization wanted to quickly make progress on remediating its risks. Therefore, the approach needed to integrate the ability to make remediating actions at the endpoint and network. It needed to provide “systems management” not only detection.

• Deep OT expert support resources
  • Their resources were already stretched thin with outage schedules, maintenance, etc. They needed a partner that could immediately understand their control system environment, including the range of different vendors’ equipment. The current industrial controls’ support vendors did not have the level of cyber security knowledge necessary to address necessary security topics, so the organization looked for a partner with OT cyber security integrated expertise to act as an extension of their own team.

• Cybersecurity platform that allowed them to grow into maturity over time
  • No organization can jump to “Level 5” maturity overnight. But they don’t want to have to continually add new tools and interfaces as the needs evolve. Critical to the organization was a platform that could expand into new cyber capabilities over time and integrate the various elements necessary into a single dashboard. This was part of the overall need for efficiency and low-security Total-Cost-of-Ownership.

The Solution: OT systems management

To deliver on these requirements required adopting what Verve calls vendor-agnostic “OT Systems Management” (OTSM), a comprehensive approach to industrial security that addresses the above requirements. OT systems management applies similar security and management that the industry has applied to IT for years.

Think Global: Act Local

The OTSM approach begins with an architecture “fit for purpose”; one that addresses the cost, resource, risk complexity, and operational reliability challenges of OT security. We call this “Think Global: Act Local” (TG:AL).  TG:AL enables scaled visibility, analysis, and planning but ensures operation reliability. The structure is seen below in an example of a global organization.

1. Think Global – Scale analysis in a centralized cybersecurity platform. Gather data from all sites into a centralized database for vulnerability and risk analysis and remediation/response planning.
2. Leverage regional SMEs with access to the same platform for specific security advice.
3. Act Local – Operations control over actions. Provide automation to plant/regional personnel to enable them to take action in a way that is sensitive to the requirements of the operational environment.

The first goal of TG:AL is to address the labor and efficiency challenges associated with OT security. According to KPMG, the number one challenge of ICS security practitioners is the lack of skilled and knowledgeable OT security personnel. This was especially true at this energy company. Remote locations further added to the challenges of having personnel onsite that could address the various security requirements necessary for maturity.

Budget constraints also meant they needed a very low total cost of ownership of any security initiative.  Further, one of their key requirements was to be vendor-agnostic. They did not want to rely on islanded different vendor tools. They realized that in order to deliver security efficiently and comprehensively, the platform needed to bring visibility across vendors. Verve’s architecture is vendor-agnostic and gathers all the various systems into this central database, without disrupting the operations of those systems.

The TG:AL architecture aggregates comprehensive data from every asset into an enterprise database so that a small number of centralized resources can analyze risks, vulnerabilities, and threats. Without this architecture, each plant or site needs to hire or train personnel knowledgeable about all security elements, leading to significant costs and gaps in expertise. Again across all vendors.

The “think global” component delivers a 70% lower labor cost than more traditional OT security solutions.

However, cost and coverage are not the only priorities. The solution must also ensure operational reliability. Everyone in OT has their story of how IT tried to scan or patch or make changes from a central point and tripped the plant or worse. The architecture has to allow for control by those closest to the process.

The “Act Local” component of the Verve architecture delivers just this reliability. The enterprise database is read-only, allowing analysis but no inbound actions. The playbooks are distributed to the local process controls environments. There, before any action occurs, the operator with knowledge of the process decides on the timing and approach to executing the actions. This ensures that well-meaning teams do not cause unintended harm to the process in the pursuit of security. Importantly, these actions are automated so that the operator does not need to manually take the actions. He or she makes a few clicks in the software and hits execute to deliver the changes once tests and timing have been approved.

Verve built this architecture working hand-in-hand with global industrial clients that needed this balance of efficiency and reliability.

Technology-Enabled Vulnerability Assessment

The TG:AL architecture enables the broader OT systems management program. That program begins with a robust, endpoint level “360-degree risk assessment” to provide a comprehensive, real-time view of the OT risks on each asset in the environment. To generate this 360-degree view it is not enough to rely on packet inspection as much of the important data necessary is not communicated over the wire – e.g., patch status, all installed software, users & accounts, etc. The 36o-view is used to assess the overall risk score of each asset to provide a prioritized remediation roadmap that addresses the various challenges in OT security, such as the inability to immediately patch.

This forms the foundation of a Technology-Enabled Vulnerability Assessment (TEVA) which provides significant advantages over traditional survey or manual risk or vulnerability assessment approaches.

• First, it provides detailed information on each asset in the environment, rather than high-level findings based on individuals’ understanding of the environment. One of the challenges of OT is that there is no data to draw on for these assessments, unlike IT which usually has at least data on patch levels or vulnerabilities. So TEVA gets at the missing data.
• Second, it is significantly less time-consuming and expensive because the technology replaces the human labor required to travel to sites, interview personnel, walk down networks, and manually capture information.
• Third, TEVA is real-time, thereby avoiding the common problem of vulnerability assessments that are aged before you begin the remediation and cannot provide updated progress information as changes are made. Finally, once the technology is deployed, remediation can be immediate as soon as risks are identified. The organization does not need to wait months prior to beginning to execute on security.

A tech-enabled vulnerability assessment identifies many risks for quick remediation:

• A significant number of dormant accounts were caused by leftover vendor-created accounts during the prior maintenance outages. Since the environment had no central AD server, these local admin accounts create access points for attack.
• Lack of updated Anti-virus signatures.
• Missing backups where the installed backup tool had failed or filled to capacity.
• Insecure, unnecessary software programs. Although OEMs often threaten to not support systems if unapproved software is installed, we always find dozens of these applications, from LogMeIn to DVD burners, iTunes, and others.
• Firewall rules weakened over time as increased access is granted, creating an opening for attackers to exploit.

In addition to these, TEVA also identifies risks that may be more complex to remediate immediately:

• Vulnerabilities & Patches. Some of these can be addressed quickly, but others are not approved by OEM vendors or require rebooting of key operational equipment.
• Cleaning up dual nic’s. In many environments, vendors and operators create separate nic’s to route traffic around firewalls for various reasons. Remediation may require a short or long outage to reroute traffic through reconfigured firewalls or remote access gateways, leading to longer lead times

But the advantage of the 360-degree view is that the organization can find compensating controls that can address those longer-cycle asset risks by hardening them in other ways.

The 3R’s – Remediate, Respond, Recover

Assessment is only the initial step. OT systems management, as its name implies, requires the MANAGEMENT of assets and networks or the 3-R’s of cyber security: Remediate, Respond, Recover. These 3-R’s separate security from monitoring and they are critical to demonstrate the kind of security improvement that energy and other industrial organizations require. They are not “passive”, cannot be achieved through monitoring traffic through span ports, and require meaningful interaction with OT systems to protect them and stop threats before they spread.

IT has managed systems for years, and in fact, over 75% of all the cybersecurity jobs are related to “systems management” according to NIST’s NICE CyberSeek database.

We need to apply these same principles to OT while recognizing the sensitive systems and their unique requirements.

In OT this can sound like heresy – “If it ain’t broke, don’t fix it”. But in fact, OT is becoming more and more dependent on IT systems as virtualization and IIOT expand. OT needs to begin to manage these assets not only for security but also for operational reliability. Organizations can no longer rely on “airgaps” to protect unpatched, insecurely configured assets.

OTSM takes an OT approach to these systems management requirements. It enables rapid remediation through a platform that can patch, harden, manage software and users, update AV or whitelisting, etc. Instead of relying on inefficient and inconsistent manual processes, OTSM automates remediation actions to accelerate mean-time-to-remediation.

The organization could centrally identify the risks and immediately develop common remediation playbooks to be distributed across sites to fix security gaps. These playbooks are executed by the “Act Local” team at the site using their understanding of the process to test and approve the timing of executing the action. Confirmation of the action is then immediately available to the “Think Global” team to ensure compliance and security resolution.

Similarly, when the SOC or “Think Global” team identifies a potential threat, it has the data to dive into that event to understand the context of the assets impacted in greater detail. Prior to Verve the organization had alert-fatigue and blunt instrument responses because of a lack of deep endpoint and network data. Using Verve, the Think Global team dives deep into the assets to understand where the threat may move and identify response actions. This deep analytical view allows the central team to define what we call the “least disruptive response” to any threat.

As an example, Verve detected a potential brute force attack leveraging a specific set of accounts.  The 360-degree view of the assets, including all account data on every machine, including dormant accounts, enabled the “Think Global” team to identify which systems might be impacted and to design a playbook to remove those accounts. The “Act Local” resources then confirmed that these accounts were not critical for operations and executed the action within hours. This is just one example of the power of the “Think Global: Act Local” architecture.

Finally, no security program will protect or respond to every threat. There will be the day when the attackers beat the defenders. The last line of defense is the third “R”, recovery, i.e. restoring systems after an event. The OTSM program enables this process by aggregating backups across different OEM vendor systems. It can track the backup status to ensure the organization is ready to recover when needed. Verve helped the organization establish an OT-specific backup suite to cover its traditional OS devices, networking gear as well as many of the critical embedded devices.

OT expert support

Cyber security is NOT just technology. Although TG:AL allows for scaling resources, this energy organization still needed additional external support for deployment and bringing security best practices to the organization.  One of their big challenges was that the current ICS support vendors did not possess deep ICS security knowledge and were relatively vendor-specific. Key to the success of OTSM is having cyber security personnel that understands system differences, but the security commonalities to drive consistency, efficiency as well as operational resilience.

Verve has a 30-year legacy of vendor-agnostic control systems design. Over the past 15 years, we expanded this expertise into cyber security, maintaining that vendor-agnostic approach. Verve’s teams work with customers to conduct technology-enabled vulnerability assessments across the different vendors in the environment. They develop risk reduction roadmaps across an enterprise, down to a plant level. They assist in remediating such as network segmentation, patch management, configuration hardening, etc. – all across different vendor systems.

Segmentation is a perfect example where a vendor-agnostic approach is critical. These systems interact, and the design team needs to understand how the different systems work so that the right network architecture can be designed. Verve’s team has the capability to draw those cross-vendor insights.

Verve provided the client this expert support in prioritizing risks, deploying critical remediation actions, as well as overall cyber security training and incident response support. Importantly this capability was “cross-platform” so the client had a single partner to help close the gaps between different security elements that often arise when a more silo’d approach is taken.

The Result: Immediate and measurable cybersecurity improvement

The objective was to deliver measurable improvement in OT security within a 12 month period of time and build a foundation to push further security advances on a continuous improvement basis. By following their requirements, Verve helped the organization assess risks and develop practical remediation guides and playbooks.  Almost immediately, they improved their cybersecurity posture by removing risky user accounts and access, resetting password and access settings, creating backups that had languished for over a year, and taking patching actions to reduce critical vulnerabilities.

Over time, the team also identified potential threats and used the platform’s “Think Global” component to conduct incident response processes to identify potential root causes of these alerts. They improved their network segmentation by evaluating potential gaps with rule analysis. Finally, they are building on the platform to continually add greater security functionality as the organization matures.

They did all this without adding new headcount by leveraging the TG:AL platform which created the efficiency they needed given their budget constraints.