In the spring and summer of 2021, the United States government issued several new regulatory initiatives – from the President’s Executive Order on supply chain security to the TSA’s issuance of new regulatory requirements for the energy pipeline sector to the rail and aerospace standards which will be similar to pipelines. These cyber security initiatives were in response to several very public ransomware attacks on critical infrastructure such as Colonial Pipeline, as well as lesser-known, government-tracked hacking attempts of other entities.

Also over the 2020-2021 period, global governments instituted or reinforced a range of security requirements for OT – from the adoption of NERC CIP by the Chilean regulatory entity or the various standards emerging in the Middle East, to Singapore’s refocusing of its overall cyber security standards to place greater emphasis on OT. The writing is on the wall that the future of OT cyber security is one of greater threats – and much greater regulatory scrutiny of protection and response.

This paper is not intended to argue the pros or cons of these approaches. There is plenty of public comment on the pros (and even more on) the cons of these approaches. Instead, this paper’s intent is to lay out the likely future scenario and how industrial organizations can make immediate progress while preparing their organizations for the coming deluge of industrial cyber security compliance requirements. This paper offers learnings on how to achieve a successful and efficient programmatic response to the current – and future – regulatory environment for ICS/OT cyber security.

These perspectives are based on practical experience. For the past 15 years, Verve Industrial has been on the front lines working with North American power companies to address the compliance requirements of NERC CIP. More recently, we have supported the response of multiple North American pipeline operators to the new regulatory requirements.

We have seen the challenges of addressing these more prescriptive cyber security requirements and understand how easy it is to become overwhelmed with the processes, complexity, and inefficiencies of this change. However, we remain confident and encouraged in managing cyber security regulations by the many organizations who successfully adapt and create efficient means to secure their environments and achieve effective compliance with regulatory requirements.

New regulatory challenges for industrial operations

Industrial organizations are in the crosshairs of cyber security attackers. Ransomware has caused outages across industries as divergent as pipelines, meatpacking, beer, and paper packaging,  but energy is now the third most targeted industry, up from the ninth in 2019. (Source: IBM X-force).  Governments, insurers, customers, and boards of directors are responding to these attacks with greater emphasis and urgency on the security of operating technology – or “OT” – the systems that control the industrial processes.

These factors create a perfect storm for those responsible for OT cybersecurity. To address them will require a significant change from the status quo – either in industrial controls operations or traditional IT security.

Of the above list, the shifting regulatory environment can cause significant challenges by itself if not managed effectively. Fifteen years ago, the United States Federal Energy Regulatory Commission established a set of standards for cybersecurity of the Bulk Electric System – now known as the NERC CIP standards. These standards had several “objective” components that required the regulated entity to achieve certain objectives with no requirement as to the “how”. They had several “procedural” standards that required the entity to establish and then follow qualified procedures and plans. However, many of the standards were “prescriptive” in nature, requiring the entity to take a specific set of actions within specific time periods, regardless of the measurable outcome. The prescriptive standards created significant costs and complexities for many entities.

As we look ahead at the coming regulatory regimes that will be implemented over the next five to ten years, we expect a significant number of them to require these “prescriptive” standards, auditable by third-party groups.

First, a note about “prescriptive” standards. There is debate as to whether such standards offer the right balance of efficiency and effectiveness in driving security actions. Many would argue that by forcing entities to take tactical actions, regulators do not allow intelligent entities to make trade-offs of where the most effective way of securing each one’s infrastructure. The challenge from the other side is that cyber security is not “objective” in that it is very hard to measure success other than the “null set” of “we haven’t been attacked”. As a result, proponents of prescriptive approaches argue that it is the only way to create baseline measurability in a world of the “un-measurable”.

This paper will not argue as to the benefits of one approach or the other. However, it does argue that this will be a major component of the global OT cyber security regulatory regimes. Governments and regulators are responding to real threats that cause dramatic impact on economies and even government stability in some geographies. As a result, the requirements will likely be more intense and prescriptive than they might be for lower impact risks.

Most industrial organizations know very well the challenges of achieving regulatory requirements. For years they have dealt with environmental, safety, occupation, food and drug, and other regulatory requirements. Cyber security is next in that line. They know that prescriptive regulations of any category create greater challenges in driving efficiency and effectiveness of response.

Industrial cyber security regulations will be more prescriptive and auditable in the years to come. Recent examples of this coming trend include the United States’ TSA pipeline cyber security standards which were recently released. According to the redacted version available online, security requirements include:

• Implementing network segmentation with a series of specific requirements of the way that segmentation should exist for instance prohibiting OT protocols from traversing the IT systems unless through an encrypted point-to-point tunnel
• Set Anti-virus scans across IT and OT on a weekly basis
• Implement patches (or have a documented reason why they have not been implemented) in a specific timeframe (similar to the debated CIP-007 mentioned above
• And many others

Other examples are in Chile where CEN (the government’s National Electricity Coordinator) has adopted the NERC CIP standards or the Middle East countries where regulators such as the DESC in Dubai have adopted more prescriptive OT cyber security requirements.

Essentially these standards are trying to apply IT-like cyber security into the OT realm. Many will argue that this is not possible or practical. But the reality is that the trend is headed in this direction. Industrial organizations will need to find ways to apply these IT-like security functions in a way that is safe and practical for OT but satisfies the requirements.

The future of OT cyber security regulation is clear – more prescriptive requirements and more auditing by regulatory bodies.

This will require a significant shift in mindsets, investments, and efforts among industrial organizations around the world. It took the North American electric power sector eight years from the first approval of NERC standards to robust audits under version 5 of the standard…and another five years to today. Because the risks are even greater, we expect these new regulatory standards to be adopted with even greater urgency than NERC CIP was. This means less time to prepare and evolve than was the case in North America.

The NERC CIP standard as one guidepost to future industrial cyber security

The NERC CIP standards are the mandatory security standards that apply to entities that own or manage facilities that are part of the U.S. and Canadian electric power grid. They were initially approved by the Federal Energy Regulatory Commission back in 2008. Their wide-ranging requirements drive a significant amount of investment by the regulated utilities and have helped create a foundation of cybersecurity awareness in the electric utility sector in North America.  But it is their foundation as a model for an emerging set of Operating Technology cybersecurity regulations around the world that should make studying them required reading for industrial operators worldwide.

The first version of the CIP standards was released in 2006 and approved by the Federal Energy Regulatory Commission in 2008. That core body of standards went through what are generally considered to be five versions before revision numbering was abandoned for the body in favor of tracking versions of individual standards. Versions 3 and 5 represented significant steps forward for the industry. With the change to per-standard revision monitoring, incremental changes such as the addition of a supply chain security standard and consideration for better support for virtualization have been possible.

As of this writing, NERC CIP standards include the following categories:

The standards encompass the same breadth of topics, generally, as other cyber security frameworks such as the NIST CSF or CIS Top 18 Controls, but they are more prescriptive than those frameworks and are enforceable on those entities that are subject to them, including the application of potentially large fines in cases of non-compliance.

What is the TSA pipeline security directive?

DHS and CISA released the Pipeline Security Directive. “The cybersecurity landscape is constantly evolving and we must adapt to address new and emerging threats,” said Secretary of Homeland Security Alejandro N. Mayorkas.  “The recent ransomware attack on a major petroleum pipeline demonstrates that the cybersecurity of pipeline systems is critical to our homeland security. DHS will continue to work closely with our private sector partners to support their operations and increase the resilience of our nation’s critical infrastructure.”

The pipeline security directive was a very quick reaction that reinforced the suggestions that TSA already had provided to pipeline operators around regular internal assessments and added requirements around naming a responsible individual and reporting incidents. The enhancements in July 2021 takes a different tone – Instead of simply reporting and assessment requirements, TSA is following a model that we see becoming the norm: specific requirements of protections and remediating actions.

Almost 15 years ago, the United States introduced the NERC CIP regulatory regime for the bulk electric system. NERC CIP is a very regimented approach with a specific set of controls that can be mapped to other control models such as NIST 800-53, CIS Top 20 (now 18), etc. It is a prescriptive and auditable standard. Prescriptive means it requires utilities to take certain actions, track certain data, and maintain specific standards. Auditable in that NERC regularly audits the compliance with the prescribed controls and can penalize (fine) entities that fail to achieve consistent compliance.

The new TSA pipeline security directive is certainly prescriptive by requiring a set of security controls across an operator’s infrastructure. It is unclear at this point whether these controls will become auditable as well. But given the initial indications, it is likely this will come down the road.

3 components of the TSA Directive:

• Report confirmed any potential cybersecurity incidents to the DHS Cybersecurity and Infrastructure Security Agency (CISA)

This requires that pipeline companies build an incident response capability, which is included in the recommended cyber security elements of the original DHS may 2018 security release. This order adds the requirement to share any cyber incidents with CISA

• Designate a Cybersecurity Coordinator to be available 24X7
• Review current cybersecurity practices and identify any gaps as well as related remediation measures and report those to TSA within 30 days

This final directive relates back to March 2018 (updated in April 2021) Pipeline Security Guidelines — which were only recommendations. This directive implies they will become mandatory.  This is likely the most significant part of the order as it begins a regime of more compliance requirements. These recommendations are a relatively comprehensive list of security controls and will likely require significant effort for many pipeline operators to achieve.

Perhaps most importantly, the directive makes clear that this is the first step in what is likely to be a more extensive set of requirements over the coming months.

These guidelines will form the basis of any review for pipeline operators. So, the first question is: What are the cyber security controls included in the current TSA pipeline recommendations?

TSA constructs its recommendations into the same categories as the NIST Cybersecurity Framework of Identify-Protect-Detect-Respond-Recover. TSA then narrowed the traditional NIST components to a more targeted set of controls that are relevant for converged cyber-physical systems such as pipelines. We won’t try to speculate here and now as to how this list may expand in any future regulatory orders. The current list of controls will already be a challenge for many pipelines to achieve efficiently and effectively.

The list of controls is included below and include both procedural and technical requirements. They do not distinguish between IT and OT systems, but the implication is that the guidelines should apply to both, with any necessary adjustments for the OT environment.

What can you learn from successful OT cyber security compliance practitioners?

The good news for those beginning to address OT cyber security risks such as North American pipeline operators is that after almost 15 years of managing compliance with NERC CIP, there are great learnings from the North American power industry on how to increase cyber security and address these growing regulatory prescriptions. Furthermore, in the past twelve months, Verve has applied similar practical approaches for pipelines and other organizations under the new TSA compliance requirements.

Based on this experience, we recommend 5 key steps to successful, efficient OT cyber security security and regulatory compliance.

1) Design your program with the eventual goal in mind

Compliance can become like “whack-a-mole”. One of the challenges with compliance is that the goalposts evolve over time. NERC CIP’s history sets a precedent of rules evolving over time. As a result, many entities began addressing the immediate need of the requirements of early versions, without taking into account where Version 5 or future versions might take them. Often it was compliance for compliance’s sake, without a context of an overall effective and efficient cybersecurity program. As a result, the implementation of compliance controls were disconnected. Tools were bought to solve a single component, not realizing that future requirements might make the tool redundant. Processes were set up and trained, only to be re-designed and re-trained when new requirements emerged.

We strongly recommend a programmatic approach from day one. Today’s requirements for inventory and patching are likely to lead to tomorrow’s requirements for configuration hardening, user & account control, etc. It’s a bit like the old children’s book “If You Give a Mouse a Cookie”…next thing you know he’ll be asking for milk, a napkin, a nap, and so on.

A good starting point for a programmatic approach is to select a framework such as NIST CSF, IEC 62443, or others that provide a roadmap including the likely regulatory requirements.  Successful organizations leverage these frameworks and design an end state they want to achieve based on best cyber practices as well as likely regulatory requirements. They establish a clear roadmap of actions with foundational components that expand to meet the evolving needs of the requirements over time.

These programmatic efforts start with foundational elements – namely an accurate, actionable, detailed inventory of all hardware, software, users, accounts, network connections, ports, and services, etc. This inventory is not an end in itself; it is the foundation for other components of the program. So as you build it, you need to consider how and what is going its use will evolve down the road. Visibility is not sufficient, although it is necessary.

Just as one example, the TSA guidelines are very specific with their patching requirements, so any asset inventory should be able to answer the following questions related just to patching:

• Does it provide accurate, timely and certain patch data that will satisfy audit and compliance requirements?
• Does it directly tie to the patch deployment and enable accurate tracking of new patch applications?
• Does it allow us to monitor vendor-approved vs. not approved patches?
• Can it track the time from patch release to review to application or mitigation?
• Does it track mitigation actions on an asset-by-asset basis in case we cannot patch?

This is just in patching, but the point would apply to the rest of the program. Ensuring a view of where you want to go with the program allows you to start on the right foot and not waste money and human resources deploying things that are not sustainable in the long run.

> See the video on building a roadmap here.

2) Employ global vendor-agnostic security & compliance

One of the biggest challenges in achieving OT security compliance is the OT heterogeneous and distributed environment. OT almost always consists of equipment produced by multiple different OEM vendors. But the hardware and software manufacturer lists are endless as you cross into power distribution, manufacturing, and now IIOT. Vendor diversity and complexity are further complicated by the distributed physical and network environments of many industrial operations. Many industrial systems operate over hundreds or thousands of miles – trains and pipelines for instance. Even those that are within a manufacturing plant or mine or other “site”, may have dozens or hundreds of different network structures that may or may not be connected back to a central network interface.

Without the consolidation of security and compliance information across vendor systems and across geographic locations, costs, and complexity of compliance skyrocket. Just imagine, as many utilities in the United States do, that tracking compliance and monitoring security status is done differently for each OEM vendor. At one generation facility, for instance, there may be three or four different industrial control security “stacks” maintaining and reporting on security status. Each of those stacks has 5-10 different sub-components provided by different “white-labeled” providers.  Now, add that to dozens or hundreds of locations with no centralized visibility to security or compliance information.  The complexity soars.

Organizations that seek to drive efficiency and effectiveness overcome this complexity by consolidating security and compliance technology into vendor-agnostic solutions. They centralize reporting into global databases that make both compliance efficient and security more effective. This visibility needs to provide detailed asset-level information including 100% of all software deployed, patch status, full configuration status, users and accounts including local users, etc. In many cases, this information does not exist at all or is contained in spreadsheets at each site. It is critical to the long-term sustainability of the compliance program that the organization centralize this information for monitoring and reporting. Without it, the costs escalate quickly and the compliance lags.

> Read more on vendor-agnostic OT cybersecurity.

3) Enable efficient local actions

For compliance, monitoring is not enough. Organizations must take action to maintain security status. This includes patch management, software management, configuration management, user and account management, etc. Many OT security approaches (outside of regulated environments) have relied on passive monitoring of network traffic to date. Unfortunately for compliance, this is not sufficient. The tools and technologies must enable actions.

OT practitioners, however, realize that taking action on running control systems can cause more damage than a cyber-attack might. Safe operations requires that any action taken at an endpoint needs to consider the operating environment and current processes. These are best understood by local or subject matter experts on those processes.

The key to success is automating actions without causing undue risk to the operating environment.  Successful compliance organizations have deployed platforms where the key security actions can be designed centrally – e.g., what patches are approved by the OEM, which ones are critical or security related, what devices should be patched, and in what order. Then those are distributed to the local operations. But, importantly, the final execution of those actions whether it be a patch deployment or a user/account removal, etc. is controlled by the operator closest to the process to ensure the action does not disrupt operations.

Verve calls this “Think Global: Act Local”.  The notion of “TG:AL” is that the entity centralizes the compliance and security information as well as the design of system management playbooks globally (per item #3 above), and controls those automated actions locally to ensure safe OT operations. This is one of the big changes compared to IT security necessary for safe OT cybersecurity compliance. The cybersecurity infrastructure needs to enable the action, but also the control by local, trained operators.

The “TG:AL” process is especially critical for TSA pipeline regulatory compliance. These devices are spread across hundreds or thousands of miles. To support the TSA patching requirements – discovery, review, and application or mitigation within a set time period – requires the ability to accurately capture all software – OS, applications, firmware, etc. – accurately, to then centrally discover and review available patches, centrally develop mitigation and deployment plans, but then deploy those patches efficiently through automation BUT MOST IMPORTANTLY with local control to ensure operational reliability of these sensitive systems.

> Understand the Think Global: Act Local architecture.

4) Start early by assigning dedicated leadership for OT Systems Management

For 20+ years, IT has conducted robust systems management – vulnerability assessment, patch management, configuration management, user & account control, log management, etc. However, in OT these “systems management” functions are often missing for a variety of reasons – lack of resources, complex legacy hardware and software environments, multiple OEM systems, distributed assets, etc.  All these compliance components require OT systems management – the ability to identify all of your assets, manage network connections, monitor missing patches, ensure configurations remain in compliance with security standards, etc.  And to do this requires leadership dedicated to managing these components. This is different from the “designated cybersecurity coordinator” that the TSA’s initial directive required. This function goes beyond coordinating to truly leading the elements of cybersecurity management that the regulations require.

OTSM elements can be seen clearly in the new TSA guidelines, the DESC guidelines, and the RIIO2 standards in the UK and elsewhere. These are foundational components of a cybersecurity program. One of the best highlights of the importance of these systems management components is the NICE Cyberseek database of open cybersecurity roles. As can be seen below, over ¾ of all the tasks that organizations are hiring for are “systems management” functions. Unfortunately, these functions are often ignored in OT. This will need to be one of the first things to change to be successful.

The earlier an organization begins its journey, the less painful the eventual regulatory burden is. Cyber security is often referred to as “defense in depth”. Whether that phrase is a perfect summary for the modern threats, there is no question that success requires foundational elements, and those foundational elements take time. An organization cannot just jump to maturity “5”. The earlier it begins to draw its path – using NERC CIP and other frameworks as its guideposts – the more feasible achieving future regulatory compliance will be.

> Read the OT Systems Management whitepaper.

5) Bring technology and people together to drive efficiency and consider a blended internal and external resource model to build talent

The number one challenge of OT cybersecurity leaders is a lack of available resources.

Successful programs define an efficient process early in the journey recognizing this challenge. They use a combination of simplifying technology, external expert resources, and internal dedicated team members to achieve the true objective of not only compliance but security as well.

Technology: The key success factor is choosing a technology approach that aggregates the critical compliance data into a single interface. One of the greatest time requirements in compliance is the recording, maintaining, and aggregation of data for compliance purposes. The technology should provide that data directly or easily integrate with other sources to provide a database that is easy to search and analyze.

Talent: We all know software alone isn’t enough. Producing this evidence-based structure requires an integrated approach combining technology with dedicated personnel who design and gather necessary internal and external information such as patch releases, take appropriate security actions, and maintain evidence of those actions.

One of the biggest challenges in compliance is ensuring trained personnel are available on a consistent basis. OTSM requires additional knowledge that traditional ITSM practitioners might not have: sensitivity of OT systems, interfacing with vendors for patch applicability and approval network requirements in OT environments, etc.  All of these require specific OT training for these systems management practitioners.

Successful organizations combine two paths: develop a robust internal talent pipeline and outsource key scalable functions.  In many cases, entities try to bring in or assign three or four personnel because that is what they believe they will need to maintain security and compliance. What inevitably happens within a year or so is that one or two of those people leave or find a different role in the company and 1 turns out not to work out. So after 18 months, they have one person from the original group left, and they are starting over. Training a team requires scale. Ten people seems to be about the cut-off point below which it is difficult to maintain the recruiting, training, and development necessary to sustain ongoing performance.  If more than 10 FTE’s are required, it is feasible for organizations to operate efficiently and effectively, but only if the operation is set up for success.

But even then, there are certain functions that may benefit from the scale of an outsourced partner. One of those is patch review. Vendors release patches for all systems. Conducting that review once on a particular vendor’s website is more efficient than doing it once for each entity. Therefore, there are likely economies of scale to bringing in a third party for patch discovery and review.

In any event, it is key to follow the best practice organizational design and development practices as in any other part of the organization. Failure mode is when compliance or security is seen as “secondary work” or “less critical” work and it does not attract the right level of leadership and talent. Compliance and security will become VERY important during an audit or if the entity has an incident, but it is too late to develop and train at that point. The key is to create that organizational model early on in the process.

If the organization cannot sustain a strong development pipeline, others have had success in outsourcing significant components of the compliance or security regime. As expected, this too requires internal leadership in ensuring the partner has the right resources, skills, etc. to complete the tasks effectively. However, this has been a successful path for many.

Improve pipeline cybersecurity

Verve has worked with pipeline operational technology for over a quarter-century. We have developed solutions, combining our unique security management software platform along with expert Verve design-for-defense solutions. Verve provides a comprehensive solution to support our clients leveraging our almost 30 years of operations controls experience. Our team provides assessments of the requirements to determine the gaps present as well as develop the appropriate roadmap to close these gaps.

We leverage the Verve Security Center which gathers a comprehensive “360-degree” risk score that includes the elements of the TSA guidelines.  Therefore, the assessment enables a single view and reporting of status and gaps. Most importantly, the Verve platform enables operators to immediately pivot from that assessment to remediating actions, instead of a long gap between assessing and remediating. Verve enables a “closed-loop” approach to demonstrate maturity improvement within 30 days.

The Verve Security Center platform brings together these security elements into a single platform to drive management efficiency. If an organization has invested in prior tools, Verve integrates with dozens of tools to provide a single pane of glass for analysis and reporting.

This all starts with the foundation of a robust inventory, but from there Verve enables all of the other security requirements.