3 Steps to Reduce Risk and Respond to Threats in Industrial Environments
How to determine the overall objectives of OT security before taking action that may not lead to true security improvement.Learn More
Subscribe to stay in the loop with the latest OT cyber security best practices.
2020 and 2021 were watershed moments in the cyber security risk of industrial organizations. Perhaps this was the most evident in the oil and gas industry where events such as the Colonial Pipeline attack and resulting regulations emerging from TSA have raised the profile of these risks. But this is just one very public event in a string of recent increases in threat activity targeting the world’s energy infrastructure. Energy was the third most targeted industry in 2020 (from 9th in 2019) and has increased even greater in 2021.
Oil & gas and other industrial organizations are under what we call an “AIR-RAID” of OT cyber security challenges:
And just at the end of 2021, the Log4j vulnerability dramatically increased the challenges for OT cybersecurity personnel. This is an insidious vulnerability as it exists within libraries used by a very large number of software vendors. Finding that library on sensitive OT systems is incredibly challenging. Unfortunately, this is only the latest in a series of challenging risks and vulnerabilities which have been arising ever more frequently.
One of the top five oil and gas producers in North America enlisted Verve Industrial to accelerate its cyber security maturity in the face of these rising challenges. Like many energy companies, this organization has a wide range of ICS vendor systems and, in fact, has one of the largest global installations for one of the major DCS vendors. The senior leadership recognized that they were a significant target, not just because of the safety or environmental risks they faced, but also as a ransomware target given the financial impact of a potential shut down.
The OT organization undertook several security initiatives. They implemented network security efforts to add perimeter protections but had little visibility into the ongoing maintenance of those protections or to the risks inside that perimeter. The team tested ‘network visibility and detection’ tools but was overwhelmed with the number of alerts generated by “anomalies” and missed many of the assets in the environment given dependence on the network packet information. Their large OEM vendor patched those systems using manual techniques, but it was time-consuming and did not cover all the systems or the vulnerabilities in the environment.
They stepped back and developed a comprehensive view to drive an efficient and effective cybersecurity program rather than a series of one-off efforts. However, because of the energy price swings during the COVID-19 pandemic, the organization faced significant budgetary pressures, and, therefore, any program needed to be efficient and low-cost to implement and manage.
To deliver on these requirements required adopting what Verve calls vendor-agnostic “OT Systems Management” (OTSM), a comprehensive approach to industrial security that addresses the above requirements. OT systems management applies similar security and management that the industry has applied to IT for years.
The OTSM approach begins with an architecture “fit for purpose”; one that addresses the cost, resource, risk complexity, and operational reliability challenges of OT security. We call this “Think Global: Act Local” (TG:AL). TG:AL enables scaled visibility, analysis, and planning but ensures operation reliability. The structure is seen below in an example of a global organization.
The first goal of TG:AL is to address the labor and efficiency challenges associated with OT security. According to KPMG, the number one challenge of ICS security practitioners is the lack of skilled and knowledgeable OT security personnel. This was especially true at this energy company. Remote locations further added to the challenges of having personnel onsite that could address the various security requirements necessary for maturity.
Budget constraints also meant they needed a very low total cost of ownership of any security initiative. Further, one of their key requirements was to be vendor-agnostic. They did not want to rely on islanded different vendor tools. They realized that in order to deliver security efficiently and comprehensively, the platform needed to bring visibility across vendors. Verve’s architecture is vendor-agnostic and gathers all the various systems into this central database, without disrupting the operations of those systems.
The TG:AL architecture aggregates comprehensive data from every asset into an enterprise database so that a small number of centralized resources can analyze risks, vulnerabilities, and threats. Without this architecture, each plant or site needs to hire or train personnel knowledgeable about all security elements, leading to significant costs and gaps in expertise. Again across all vendors.
The “think global” component delivers a 70% lower labor cost than more traditional OT security solutions.
However, cost and coverage are not the only priorities. The solution must also ensure operational reliability. Everyone in OT has their story of how IT tried to scan or patch or make changes from a central point and tripped the plant or worse. The architecture has to allow for control by those closest to the process.
The “Act Local” component of the Verve architecture delivers just this reliability. The enterprise database is read-only, allowing analysis but no inbound actions. The playbooks are distributed to the local process controls environments. There, before any action occurs, the operator with knowledge of the process decides on the timing and approach to executing the actions. This ensures that well-meaning teams do not cause unintended harm to the process in the pursuit of security. Importantly, these actions are automated so that the operator does not need to manually take the actions. He or she makes a few clicks in the software and hits execute to deliver the changes once tests and timing have been approved.
Verve built this architecture working hand-in-hand with global industrial clients that needed this balance of efficiency and reliability.
The TG:AL architecture enables the broader OT systems management program. That program begins with a robust, endpoint level “360-degree risk assessment” to provide a comprehensive, real-time view of the OT risks on each asset in the environment. To generate this 360-degree view it is not enough to rely on packet inspection as much of the important data necessary is not communicated over the wire – e.g., patch status, all installed software, users & accounts, etc. The 36o-view is used to assess the overall risk score of each asset to provide a prioritized remediation roadmap that addresses the various challenges in OT security, such as the inability to immediately patch.
This forms the foundation of a Technology-Enabled Vulnerability Assessment (TEVA) which provides significant advantages over traditional survey or manual risk or vulnerability assessment approaches.
A tech-enabled vulnerability assessment identifies many risks for quick remediation:
In addition to these, TEVA also identifies risks that may be more complex to remediate immediately:
But the advantage of the 360-degree view is that the organization can find compensating controls that can address those longer-cycle asset risks by hardening them in other ways.
Assessment is only the initial step. OT systems management, as its name implies, requires the MANAGEMENT of assets and networks or the 3-R’s of cyber security: Remediate, Respond, Recover. These 3-R’s separate security from monitoring and they are critical to demonstrate the kind of security improvement that energy and other industrial organizations require. They are not “passive”, cannot be achieved through monitoring traffic through span ports, and require meaningful interaction with OT systems to protect them and stop threats before they spread.
IT has managed systems for years, and in fact, over 75% of all the cybersecurity jobs are related to “systems management” according to NIST’s NICE CyberSeek database.
We need to apply these same principles to OT while recognizing the sensitive systems and their unique requirements.
In OT this can sound like heresy – “If it ain’t broke, don’t fix it”. But in fact, OT is becoming more and more dependent on IT systems as virtualization and IIOT expand. OT needs to begin to manage these assets not only for security but also for operational reliability. Organizations can no longer rely on “airgaps” to protect unpatched, insecurely configured assets.
OTSM takes an OT approach to these systems management requirements. It enables rapid remediation through a platform that can patch, harden, manage software and users, update AV or whitelisting, etc. Instead of relying on inefficient and inconsistent manual processes, OTSM automates remediation actions to accelerate mean-time-to-remediation.
The organization could centrally identify the risks and immediately develop common remediation playbooks to be distributed across sites to fix security gaps. These playbooks are executed by the “Act Local” team at the site using their understanding of the process to test and approve the timing of executing the action. Confirmation of the action is then immediately available to the “Think Global” team to ensure compliance and security resolution.
Similarly, when the SOC or “Think Global” team identifies a potential threat, it has the data to dive into that event to understand the context of the assets impacted in greater detail. Prior to Verve the organization had alert-fatigue and blunt instrument responses because of a lack of deep endpoint and network data. Using Verve, the Think Global team dives deep into the assets to understand where the threat may move and identify response actions. This deep analytical view allows the central team to define what we call the “least disruptive response” to any threat.
As an example, Verve detected a potential brute force attack leveraging a specific set of accounts. The 360-degree view of the assets, including all account data on every machine, including dormant accounts, enabled the “Think Global” team to identify which systems might be impacted and to design a playbook to remove those accounts. The “Act Local” resources then confirmed that these accounts were not critical for operations and executed the action within hours. This is just one example of the power of the “Think Global: Act Local” architecture.
Finally, no security program will protect or respond to every threat. There will be the day when the attackers beat the defenders. The last line of defense is the third “R”, recovery, i.e. restoring systems after an event. The OTSM program enables this process by aggregating backups across different OEM vendor systems. It can track the backup status to ensure the organization is ready to recover when needed. Verve helped the organization establish an OT-specific backup suite to cover its traditional OS devices, networking gear as well as many of the critical embedded devices.
Cyber security is NOT just technology. Although TG:AL allows for scaling resources, this energy organization still needed additional external support for deployment and bringing security best practices to the organization. One of their big challenges was that the current ICS support vendors did not possess deep ICS security knowledge and were relatively vendor-specific. Key to the success of OTSM is having cyber security personnel that understands system differences, but the security commonalities to drive consistency, efficiency as well as operational resilience.
Verve has a 30-year legacy of vendor-agnostic control systems design. Over the past 15 years, we expanded this expertise into cyber security, maintaining that vendor-agnostic approach. Verve’s teams work with customers to conduct technology-enabled vulnerability assessments across the different vendors in the environment. They develop risk reduction roadmaps across an enterprise, down to a plant level. They assist in remediating such as network segmentation, patch management, configuration hardening, etc. – all across different vendor systems.
Segmentation is a perfect example where a vendor-agnostic approach is critical. These systems interact, and the design team needs to understand how the different systems work so that the right network architecture can be designed. Verve’s team has the capability to draw those cross-vendor insights.
Verve provided the client this expert support in prioritizing risks, deploying critical remediation actions, as well as overall cyber security training and incident response support. Importantly this capability was “cross-platform” so the client had a single partner to help close the gaps between different security elements that often arise when a more silo’d approach is taken.
The objective was to deliver measurable improvement in OT security within a 12 month period of time and build a foundation to push further security advances on a continuous improvement basis. By following their requirements, Verve helped the organization assess risks and develop practical remediation guides and playbooks. Almost immediately, they improved their cybersecurity posture by removing risky user accounts and access, resetting password and access settings, creating backups that had languished for over a year, and taking patching actions to reduce critical vulnerabilities.
Over time, the team also identified potential threats and used the platform’s “Think Global” component to conduct incident response processes to identify potential root causes of these alerts. They improved their network segmentation by evaluating potential gaps with rule analysis. Finally, they are building on the platform to continually add greater security functionality as the organization matures.
They did all this without adding new headcount by leveraging the TG:AL platform which created the efficiency they needed given their budget constraints.
How to determine the overall objectives of OT security before taking action that may not lead to true security improvement.Learn More
Achieving a mature level of OTSM is critical to improve overall ROI from increasingly connected industrial systems and to ensure foundational elements of OT cyber security are in place to protect critical infrastructure from targeted and untargeted attacks.Learn More
The Think Global, Act Local concept emerged to describe how to scale vulnerability analysis, remediation design, and audit in industrial control systems.Learn More