Many industrial organizations find themselves in the beginning stages of addressing the cybersecurity of their Operating Technology (OT) systems. Most lack visibility and resources to adequately assess – not to mention remediate – the risks of these environments. And once the environment has been secured, the maintenance and monitoring tasks stretch experienced resources to the breaking point.
Industrial environments are filled with vulnerabilities and risks and don’t show any signs of slowing down. Many of those risks cannot be addressed directly through patching or traditional account and user management, etc. due to the unique operational environments. As a result, organizations not only need large numbers of resources but also require those staff members to possess a wide-range of OT cyber and device experience to address different elements of security – from patching to network segmentation to configuration management, backup and restore, etc.
From recent OT cybersecurity studies, we learned:
- 83% of security experts believe there is a significant shortage of OT security workers.1
- More than 2/3 of respondents said the lack of dedicated security staff is diminishing the effectiveness of their organization’s OT security.1
- 69% believe organizations are having a hard time finding the right resources because they are lacking the right skillset.1
- 59% of businesses reported they would find it challenging to respond to a cyber incident due to a shortage of skills on their team.2
While demand for OT/ICS cybersecurity resources continues to increase (thanks to recent incidents such as SolarWinds and the steady discovery of ICS vulnerabilities), most organizations are not equipped to effectively manage and maintain the security of their OT systems.
The Challenge
A top 10 North American power generation company with a number of multi-unit generation facilities from coal to gas to hydro sought to achieve rapid security maturity and maintain NERC CIP compliance across the fleet. Their team had deep instrument and electrical technical expertise but lacked experience in security and compliance. They wanted a single team to manage compliance and security across endpoints and networking while offering both remote and on-site support.
While this company operates dozens of electricity generation facilities, pipelines, gas storage, electricity transmission & distribution, etc, it had certain facilities subject to NERC CIP regulations including Medium & High criticality assets. It had others with no required compliance at all. It also had a range of different processes – from coal plants to distribution and gas storage tanks and pipelines. It had significant requirements for its data flows especially historian data to set pricing and distribution information to markets. This meant that the approach to cybersecurity and reliability would be different for different parts of the business – some of which had significant penalties should performance of IT systems not operate.
With limited OT resources to address these risks and reliability requirements as a regulated utility for parts of the business, they turned to Verve.
The Solution
This case study highlights that in the OT environment, managed security is more than just monitoring of logs and flows. It involves one-time system hardening and network protections. It involves monitoring of OT processes to ensure uptime. And it includes ongoing risk analysis across a range of security controls.
The solution for this client – like so many of our other clients – has evolved as their security posture and aspirations changed over time. What began with support for ensuring OT systems uptime related to data historians, evolved into deployment of security tools and network segmentation, and continued with ongoing management of all enterprise OT security analytics and reporting through the Verve Security Center to support of local and central personnel to maintain security posture through patching, configuration hardening, backup & restoration, etc.
Verve worked with this client to continually improve its security and reliability posture, provide accurate and timely reporting to executives and regulators, aswell as reduce the costs of maintaining all of these OT systems. While providing a comprehensive suite of Managed OT Security Services, we begin with a Technology-Enabled Vulnerability Assessment, which informs our next steps in remediation services such as network segmentation and patching, through to ongoing monitoring and response to incidents – both security and operational.
Phase 1
Verve’s managed support began with monitoring and response services around the client’s OT systems such as their data historians. Verve deployed a software platform to monitor uptime and reliability of their systems. Verve analyzed traffic from networking devices, flows into servers as well as uptime of local sensors to identify anomalies in device performance. The team then analyzed those indicators to respond immediately to anomalies to ensure system uptime.
Verve improved uptime of these data systems from less than 80% to a consistent 98%+.
Phase 2
The second phase of managed support involved ensuring NERC CIP compliance by deploying the Verve Security Center and its partner components as well as network segmentation. One might not consider this “managed services” in the world of IT security where “MSSP” has a defined notion of ongoing SOC management. However, in OT, the ongoing client support requires a range of “project-based” capabilities. In OT, it is key that the resources deeply understand the client’s processes across the range of operational and security elements.
So in our second phase, our managed security team provided integrated support to segment networks to separate systems for compliance and security, deploy Verve Security Center for comprehensive asset management (inventory, patch, vulnerability, risk management, threat detection, etc.) as well as our partner toolkits such as backup and restore and anti-virus.Key to the enablement of Verve’s OT Managed Security Services is the Verve Security Center and its enterprise data analytics capabilities.
The team included Verve’s unique blend of OT expertise across networking, endpoint management, vulnerability identification, etc. This combination of managed capabilities drove significant efficiencies in the overall security maturity. First, having this breadth of skills in the same team allows a single team to provide a range of security elements without separate teams and separate site visits, downtime, etc. Further, it provides greater effectiveness because the team understands the process at that site as well as the other security controls. This reduces any conflict or inefficiency in duplicated efforts.
Phase 3
From there, the client began a more comprehensive security effort beyond the basic foundations of NERC CIP. This included expanding security elements across the rest of its OT asset base and increasing the security monitoring and reporting integrating IT and OT security. Verve began this step in the journey by expanding the Verve Security Center to manage the OT systems across coal, gas, wind, solar and nuclear generation, gas storage and distribution. This provided the managed services team with an integrated solution to manage all OT devices – from accurate inventory to vulnerabilities, patching, user & accounts, configurations, etc.
Verve’s managed services team provides ongoing analysis of new vulnerabilities and risks. It worked with the IT security team to define different risk criticality standards to monitor on an ongoing basis. The managed services team customized the base asset risk score within the Verve Security Center based on the client’s preferred risk posture and criticality scoring. It then built customized monitoring of risks and threats based on that. Finally, the team provided real time support on areas such as network anomalies, application control, etc.
Verve managed security teams look across a range of data:
- Rogue asset discovery of assets that aren’t included on the current inventory
- All installed software to review for risky items such as TeamViewer or other remote access applications, as was done by attackers in the Oldsmar water incident
- Firmware on embedded devices
- Software vulnerabilities/CVEs to identify potential attack paths
- Patch status to determine whether critical patches are missing
- All users and accounts to determine presence of dormant accounts that may enable attackers to leverage older credentials available on the darkweb, as was done in the case of Colonial Pipeline.
- All configuration settings to compare to standards such as DISA-STIG as well as Verve OT-specific standards we have developed
- Status of key security & reliability software such as backups, Anti-virus, application whitelisting, etc. to determine whether attacks can be defended and responded to quickly
- Network connections and configurations to identify potential paths for external unapproved access as well as weak firewall and switch configurations that may allow spread of malware or APTs
This data all comes together into a 360-degree updated risk assessment which includes the benefits of:
- Prioritize limited resources on most critical assets and risks
- Stop the spread by focusing on compensating controls when protection isn’t feasible (e.g., 100% patching)
- Address “insecure by design” assets regardless of CVE vulnerabilities
- Understand recovery status – backups, network connections, etc.
- Address user/accounts in non-AD environments
- Ensure initial protections (e.g., network segmentation) haven’t experienced “rot”
Phase 4
The next phase of this effort is to apply Verve’s unique advanced threat and risk analytics to the real-time data emerging from Verve Security Center and all of its components. Verve’s software is unique in that it can aggregate data from a wide-range of telemetry to identify both anomalies and specific threats.
In OT, you cannot separate security from operational reliability. Verve builds on our 30 years of OT automation reliability monitoring and response to integrate reliability and security. For instance, one of the challenges of OT monitoring is knowing whether this is a security event or an operational issue not related to an outside threat. Our team leverages the aggregated data in VSC in conjunction with our team’s deep process understanding to identify true root causes as well as what we call “Least Disruptive Remediation”…i.e. what action can eliminate the risk with least impact on operations.
The Impact
This energy client – one of the leading providers in North America – has become a leader in OT cybersecurity. It has matured over a period of years adding various layers to its OT security posture. Verve is proud to have partnered with them using Verve’s OT Systems Security Managed Services to provide continuing and advancing OT cyber security.
The impact of effective OT systems managed services is felt in multiple areas:
Significant improvement of efficiency
Through the Verve Security Center platform, industrial organizations significantly reduce the cost of managing their OT security. In multiple analyses, the approach delivers a significant reduction in both assessment and maintenance costs. This is due to its ability to:
- centrally manage risks across dozens or hundreds of sites from a single console – regardless of OEM vendor, network architecture, etc.
- leverage the scale of a team of experts rather than trying to recruit, hire, train and retain a full OT security team
Accelerated time to remediation
By aggregating data and leaning on a qualified, on-call team, organizations increases the speed to security maturity. Recruiting – or reassigning – and training OT security personnel is very time consuming. We have seen many organizations build a team of 3, 5 or 7 members only to start over in 18-24 months because of departures, performance, or lateral moves in the company. By tapping into a broader pool of talented OT security experts, this saves the time and effort of building and retaining an internal team.
In addition, the Verve platform allows rapid movement from assessing risks to remediating them. Because the remediation is integrated with the assessment capabilities, the time to maturity is much less.
Improved quality of overall OT security program
Each organization’s knowledge is often limited to their team and experiences. By tapping into Verve’s team, the organization can lean on learnings from dozens of other companies. For instance, topics such as “how do you manage identities in the OT environment?” or “have others seen negative impacts from patching these particular systems?” or “how do we most effectively mitigate this vulnerability if we can’t patch it?” all are aided through sharing of knowledge across organizations – a key component of Verve’s OT managed services.
1 – Two Big OT Security Concerns Related to People: Human Error and Staff Shortages