Subscribe to stay in the loop with the latest OT cyber security best practices.
Cyber attackers have industrial organizations squarely in their sights. Manufacturing and energy are now among the top three most targeted verticals, up from eighth and ninth, respectively, just two years ago. Ransomware has wreaked havoc on industries as diverse as fuel distribution pipelines, meatpacking, beer brewing, and paper packaging. Governments, insurers, customers, and boards of directors are responding to these attacks with greater emphasis and urgency on the security of operating technology, OT, the systems that control industrial processes.
In the spring and summer of 2021, the U.S. government issued several new regulatory initiatives — from the President’s Executive Order on supply chain security, to the TSA’s issuance of new regulatory requirements for the energy pipeline sector, to new rail and aerospace standards similar to the pipelines edict. Over the past two years, global governments have similarly instituted or reinforced a range of security requirements for OT — Chile’s adoption of the North American Electric Reliability Corporation Critical Infrastructure Protection (NERC CIP), various standards emerging in the Middle East, Singapore’s cybersecurity refocus placing greater emphasis on OT to name just a few.
The future of OT cyber security will be marked both by escalating threats as well as by increasingly stringent regulatory scrutiny. This paper won’t argue the pros or cons of such approaches. Its intent is to lay out likely future scenarios and examine how industrial organizations can prepare now to avoid being overwhelmed by the looming deluge of OT security compliance requirements.
For the past 15 years, Verve Industrial has been on the front lines with North American power companies addressing the compliance requirements of NERC CIP. We’ve seen organizations swamped with processes, complexity, and inefficiencies when faced with prescriptive cybersecurity demands. We’ve also seen many adapt successfully, finding efficient ways to secure their environments and achieve effective regulatory compliance.
AIR-RAID is Verve’s mnemonic for describing the challenges that face industrial operators in cybersecurity and the massive changes organizations will need to make in order to respond to modern cyber threats.
AIR-RAID stands for:
ATTACKERS: A rapidly increasing threat from “for-profit” attackers leveraging the financial incentives and business models inherent to modern ransomware.
IT: The need to apply IT-like security capabilities into the OT environment.
REGULATION: From TSA and DOE in the U.S. to DESC in Dubai and CED in Chile, authorities are increasingly focused on prescriptive requirements to halt potential disruptions.
RESOURCES: Already constrained OT resources do not have the capacity to manage manual or complex cybersecurity requirements.
ACCESS: With increasing remote access, there is a greater concern for direct connectivity to vulnerable devices.
INSURERS: Cyber insurance rates are increasing rapidly and insurers of industrial companies are now looking for the same assurances of security in OT as they demand in IT.
DIRECTORS: Business leaders recognize the financial threat ransomware poses to industrial organizations and are focusing on the attendant risks.
All of these factors create a perfect storm for those responsible for OT security. Addressing them requires significant changes from the status quo — either in industrial controls operations or in traditional IT security.
Of the above list, the shifting regulatory environment alone introduces some of the most disruptive challenges if not managed effectively.
Fifteen years ago, the U.S. Federal Energy Regulatory Commission (FERC) established a set of standards for the security of the Bulk Electric System (BES) — now known as the NERC CIP standards. The NERC standards had several “objective” components requiring regulated entities to meet certain objectives with no requirement as to how.
NERC also had several “procedural” standards that required establishing and following qualified plans. Many of these standards were prescriptive in nature, requiring covered entities to take specific actions within specific time periods, regardless of the outcome. These prescriptive standards created significant costs and complexities for many organizations.
Looking ahead to anticipated regulatory regimes over the next five to ten years, we expect a significant number to include such prescriptive standards, most of them subject to audit by third-party groups.
There’s much debate as to whether such standards offer the right balance of efficiency and effectiveness in driving security actions and improving maturity. Many argue that by forcing entities to take tactical actions, regulators don’t allow organizations to make common-sense trade-offs to achieve the most effective security for their own, unique infrastructures. The counterargument is that security is not objective; it’s difficult to measure success other than demonstrating the “null set” of “we haven’t been attacked yet.” As a result, proponents of prescriptive approaches argue that they represent the only way to create baseline measurability in the realm of the largely unmeasurable.
No matter which side of the argument one subscribes to, one thing is abundantly clear: robust, prescriptive security requirements will soon be a major component of global OT cybersecurity regulatory regimes. Governments and regulators are responding to real threats with potentially devastating impacts on human lives, economies, and, in extreme cases, even government stability.
Recent examples of the coming trend toward more prescriptive, auditable regulations include the U.S.’s recently released TSA pipeline security standards. According to the redacted version available online, security requirements include:
Other examples include Chile’s CEN (the government’s National Electricity Coordinator), which adopted NERC CIP standards, or the Middle East countries where regulators such as the Dubai Electronic Security Center (DESC) adopted more prescriptive OT cybersecurity requirements. What these global standards efforts have in common is their attempt to apply IT-like cybersecurity approaches to the OT realm. Many will argue this is not possible or practical, but the reality is the trend is heading in this direction. Industrial organizations need to find ways to apply IT-like security functions in a way that is safe and practical for OT but satisfies the requirements. This will require a significant shift in mindset along with increased investment among industrial organizations worldwide.
The future of OT cybersecurity regulation is clear — more prescriptive requirements and more auditing by regulatory bodies. And the regulatory changes are coming more quickly than ever. It took the North American electric power sector eight years to go from initial approval of NERC standards to robust audits under version 5 of the standard. But in today’s environment of unprecedented risk, new regulatory standards will be adopted with far greater urgency. This means less time for industrial organizations to prepare and evolve.
Most industrial organizations know well the challenges of achieving regulatory compliance. For years they’ve dealt with environmental, safety, occupation, food and drug, and other regulatory requirements. Security is the next in line. These organizations understand that prescriptive regulations of any type create escalating challenges to efficiency and effective response.
After some 15 years of managing power industry compliance in North America with NERC CIP, we’ve amassed valuable insights for those beginning to address OT cybersecurity risks on how to increase security and address growing regulatory requirements. Covered entities and their industry partners have developed new technologies and processes, and many have found ways to drive greater efficiency in a challenging environment.
Based on Verve’s work under the NERC CIP structure — as well as working with many clients outside the power industry — we’ve identified five key steps to successful, efficient OT cybersecurity and regulatory compliance.
For more than two decades, IT has conducted robust systems management — vulnerability assessment, configuration and patch management, user and account controls, log management, and more. Such systems-management functions are often missing in OT for a variety of reasons including lack of resources, complex legacy hardware, and software environments, multiple OEM systems, and distributed assets.
The coming wave of requirements demands robust OT systems management (OTSM) capabilities including identification of all assets; management of network connections; monitoring for missing patches; ensuring compliant configurations; and more. This requires leadership dedicated to managing these components. Unlike the basic “designated cybersecurity coordinator” included in the initial TSA directive, this key function goes beyond simple coordination; it rises to true leadership of the elements of cybersecurity management that the regulations demand.
OTSM elements are clearly seen in the new TSA guidelines, the DESC guidelines, the RIIO2 standards in the UK, and elsewhere. These are foundational components of a cybersecurity program. One highlight of the importance of these systems management components is the NICE Cyberseek database of open cybersecurity roles. More than three-quarters of all the tasks organizations are hiring for are systems management functions. Closing the OT cybersecurity talent gap is one of the first things that must be addressed in order to be successful.
The sooner an organization begins its security journey, the less painful the eventual regulatory burden will be. Cybersecurity is often referred to as “defense in depth.” Regardless of whether that phrase is a perfect summary for modern threats, there’s no question that success requires foundational elements and those elements take time to implement. An organization cannot skip steps and simply jump to maximum maturity in cybersecurity. The earlier it begins to plan its path — using NERC CIP and other frameworks as its guideposts — the more attainable full regulatory compliance becomes.
One challenge for organizations implementing NERC CIP is that success was not clearly articulated early on and evolved over time. As a result, many entities began by addressing the immediate need of the requirements in the earliest versions without considering where the current or future versions might take them.
This was driven, in part, by a sense of incredulity, a belief that “there’s no way will require us to do X,” or “It simply cannot be done.” In the end, many of those items in question ended up as had-and-fast requirements. As a result, for many entities, implementations of compliance controls were disconnected from one another. Organizations purchased tools to solve specific issues not realizing that future requirements might make that tool redundant or obsolete. They set up and trained for processes that had to be radically redesigned when new requirements emerged.
Today, the requirements roadmaps are a bit clearer. Standards such as NIST CSF, IEC 62443, CMMC, and NIST 800-52 all provide guidance on where industrial cybersecurity is expected to go. Successful organizations will leverage these frameworks and design an end state they want to achieve based on best practices and regulatory requirements. They will establish a clear security roadmap with foundational components that can be expanded to meet evolving needs.
If an entity knows it needs to conduct comprehensive systems management, for example, it should start with tools and processes that provide future capabilities in areas such as patching, vulnerability management, user and account management, software management, etc.
OT’s heterogeneous and distributed environments present a major roadblock to achieving security compliance. OT almost always consists of equipment produced by multiple OEM hardware and software vendors (GE, Emerson, ABB, Honeywell, Siemens to name a few). The list gets even longer for organizations dealing with power distribution, manufacturing, and the so-called Industrial Internet of Things (IIOT).
This vendor diversity-driven complexity is further complicated by the distributed physical and network environments of industrial operations. Many industrial systems operate over hundreds or thousands of miles — trains and pipelines for instance. Even those within a manufacturing plant, mine, or another site could have dozens or hundreds of different network structures connected to a central network interface.
Without consolidation of security and compliance information across vendor systems and geographic locations, the cost, and complexity of compliance skyrockets. As many utilities in the U.S. are already painfully aware, procedures for tracking compliance and monitoring security status often differ from vendor to vendor. At one generation facility, for example, there may be three or four different industrial control security “stacks” maintaining and reporting on security status. Each of those stacks has five to 10 different sub-components from various “white-label” providers. Add to that the dozens or hundreds of locations with no centralized visibility to security or compliance information and it’s clear that the complexity can quickly soar toward the unmanageable.
Organizations that seek to drive efficiency and effectiveness overcome this complexity by consolidating security and compliance technology into vendor-agnostic solutions. They centralize reporting into global databases that make compliance more efficient and security more effective. Optimized visibility provides detailed asset-level information and all software deployed, along with patch and configuration status, user and accounts details, and more. It’s critical for the long-term sustainability of the compliance programs that organizations centralize such information for monitoring and reporting. Without it, costs escalate quickly and compliance lags.
For compliance, monitoring is not enough. Organizations must take specific actions to maintain security status. This includes patch management, software management, configuration management, user and account management, etc. Many OT security approaches (outside of regulated environments) have relied on passive monitoring of network traffic. For compliance purposes, however, this is insufficient. Tools and technologies must enable actions.
OT practitioners understand that taking action on running control systems can cause more damage than a cyberattack might. Safe operations require that any action taken at an endpoint needs to consider the operating environment and current processes. These are best understood by local or subject matter experts on those processes.
The key to success is to automate actions without causing undue risk to the environment. Successfully compliant organizations often deploy platforms that allow key security actions to be designed centrally (for example, what patches are approved by the OEM, which ones are critical or security-related, what devices should be patched, and in what order). Those actions are then distributed to local operations. Moreover, the final execution of those actions, whether it be patch deployment or a user/account removal, is controlled by the operator closest to the process to ensure the action does not disrupt operations.
Verve calls this the Think Global: Act Local approach where the organization centralizes all compliance and security information and design of system management playbooks globally, then controls those automated actions locally to ensure safe OT operations. This is one of the major differentiators between OT and IT when it comes to security compliance. The security infrastructure needs to enable action but allow for control by local, trained operators.
Because Critical Infrastructure Protection compliance is mandatory and largely driven by self-reporting or through the audit cycle, a successful CIP program includes a constant drive to produce and maintain evidence of compliance. Each procedure should produce evidence of its successful performance. Such evidence should be sampled and reviewed periodically for completeness and correctness. That evidence should also be archived in easily retrievable manners so that compliance can be demonstrated quickly when needed.
Producing this evidence-based structure requires an integrated approach combining dedicated compliance personnel who design and gather evidence with input and cooperation from operations personnel who produce and supply the evidence. In large utilities, this structure is typically replicated across each business unit or functional organization.
One of the biggest challenges in compliance is ensuring trained personnel are consistently available. The key tasks involve systems management, many of which are similar to those in IT. But OTSM also requires additional knowledge that traditional ITSM practitioners usually don’t have. The inherent sensitivity of OT systems, interfacing with vendors for patch applicability and approval, and the unique network requirements in OT environments all play a role and each requires specific OT training for systems management practitioners.
Successful, CIP-compliant organizations chose one of two paths: develop a robust internal talent pipeline or outsource such responsibilities. In many cases, entities try to bring in or assign three or four personnel because that is what they believe they need to maintain security and compliance. What inevitably happens within a year or so is that one or two people leave or find a different role in the company, and one doesn’t work out. So, after 18 months, they have one person from the original group left and are forced to start over.
Training a team requires scale. Ten people seems to be the cut-off point below which it is difficult to maintain the recruiting, training, and development necessary to sustain ongoing performance. If more than 10 full-time employees are required, it is feasible for organizations to operate efficiently and effectively, but only if the operation is set up for success.
Successful “insourcers” follow all best practices for organizational design and development as they would in any other part of the organization. Failure happens when compliance is seen as “secondary” or “less critical” and doesn’t attract the appropriate level of leadership and talent. Compliance and security will become very important during an audit or if the entity has an incident, but at that point, it is too late to develop and train personnel. The key is to create that organizational model early on in the process.
If the organization cannot sustain a strong development pipeline, outsourcing significant pieces of the compliance or security regime is a viable option for many. This too requires internal leadership to ensure the partner has the right resources and skills to complete tasks effectively.
Industrial operators are facing an AIR-RAID of challenges to protecting their systems from cyberattacks. Governments are recognizing these threats and increasing their focus on cybersecurity requirements. Management teams need to get ahead of this coming tsunami of regulatory requirements by laying out a robust OT systems management roadmap that will allow their organization to define — and work toward — a desirable security end state in a comprehensive, integrated manner.
Verve’s work with organizations successfully managing compliance with NERC CIP and other regulatory regimes — along with our five key success factors —enables organizations to manage their OT cybersecurity and compliance efficiently and effectively.
NERC CIP standards ensure the safety of North America’s Bulk Electric System. They regulate, monitor, and manage the system. Learn more about NERC CIP and what it means for OT/ICS Security.
Learn MoreIn today’s large and complex industrial organizations, the right cyber security governance structure depends on the culture and existing model of the rest of the organization, as well as coordination and shared decision-rights across IT, security/risk management, operations, and finance. Download the “5 Principles for Designing a Successful Governance Model for OT Cyber Security” to discover the five guiding principles…
Learn MoreHow to take a true endpoint security risk management approach for successful cyber defense efforts. This approach provides an OT-specific way of conducting ITSM.
Learn More