5 Principles for Designing a Successful Governance Model for OT Cyber Security

What is the right governance model for OT cyber security?

Governance: Who has authority? Who is accountable? These are the two most important questions in reducing cyber risk to operations.

There are “big G” Governance questions such as:

• Who should set the overall OT cyber security agenda?
• What metrics should be achieved?
• Who should have the authority to make the ultimate risk tradeoffs?
• Who is accountable if a cyber security incident occurs?

There are also “small g” governance questions such as:

• Who will decide whether to patch a specific device or create a mitigation plan?
• What tools will a business use to address cyber risks?
• Should a specific device be replaced if its firmware is out of date, or can it wait until the next upgrade cycle?

More than talent, tools, or tactics, governance is the most fundamental decision to get right in order to achieve success in defending critical infrastructure.

We often hear debates about the involvement and authority of IT and OT departments in governing security. Should the CISO, Head of Operations, or CIO lead the charge? Who should control the security decisions on OT assets within a plant or SCADA environment?

If the CISO is held accountable, shouldn’t they also be the ones to make cyber security decisions? If the CISO holds authority and accountability, shouldn’t they also hold the budget and resources?

In today’s large and complex industrial organizations, three themes emerge:

1. There is no “one-size-fits-all” answer: The right governance structure depends on the culture and existing model of the rest of the organization
2. There is no “single point of authority and accountability” for all decisions: The right governance involves coordination and shared decision-rights across IT, security and risk management, operations, and finance.
3. The biggest stumbling block is gaining true C-suite alignment to the risk and the practical remediation strategies necessary.

Although it would be nice to have a standard construct where accountability and authority are vested in one person or organizational function, this is nearly impossible given the realities of managing operations, assets and processes.

If the right answer is critical to OT cyber security success, yet it is so varied, how do you design the right approach for your unique organizational needs?

5 key principles to establishing the right governance model for OT cyber security

Secure C-suite alignment

Achieving the right governance model requires clear alignment from the C-suite to determine the risks to operations, the risk appetite of senior leadership and board of directors, a rough cost estimate to achieve different levels of security maturity, and how the senior team will make decisions in each area.

The natural leader for this exercise is the CISO. While a CISO wears many hats, leading a coordinated effort across the C-suite is crucial for success in security governance. This does not necessarily mean the CISO is granted authority to make all decisions. Rather, the CISO plays the role of an influencer when seeking alignment in decision-making, taking into consideration the expectation of balancing resources across the business.

Although specific governance models often focus on where authority and accountability reside, many RACI charts quickly become mundane exercises without a shared understanding of objectives and priorities from leadership.

C-suite alignment ensures budgets, metrics, and resources are based on agreed-upon objectives. If you find yourself midway through the OT cyber security journey, the best option is to reset and establish an agreement on key objectives to encourage future progress.

But the question is “how” to gain this alignment. In most industrial organizations, each leader has different underlying objectives:

• CISO to ensure security and respond to requests from the board of directors and regulators about the current security status
• COO/VP Operations to ensure uptime for production at the lowest cost and highest quality
• CFO to ensure profitability and ability to forecast results

How do you align these objectives to an industrial cybersecurity program? In many cases, alignment falls short. An external party completes a general assessment of the OT environment and identifies dozens of risks and the potential for operational or life-safety issues from potential cyber threats. Senior team leaders are appropriately nervous and agree “something needs to be done,” so a team is assigned to “solve the problem.”

The practical realities of what’s required is not debated or defined. True security requires managing OT assets, updating software and firmware, taking additional outages to deploy new firewalls and network segmentation, and resources to continuously remediate and maintain the security. In other words, it’s the “hard stuff” that makes decisions difficult.

The only way to gain proper alignment is to build a robust business case, including a detailed technology-enabled assessment to identify detailed gaps – as well as the detailed plan to overcome the gaps. This business case defines threats and the cost to remediate those threats in terms of headcount, technology and potential outages to ensure new protections are included.

It includes the requirements to push back on OEM vendors who plant operators rely on. It includes the trade-offs of speed and comprehensiveness to ensure the security program keeps pace with active threats, rather than traditional industrial controls systems upgrade cycles.

Alignment is messy. It requires debate and robust disagreement. It also requires facts and emotions. But without this process, the ultimate objective will not be achieved. There will be false starts, partial solutions, frustration, and ultimately significant unplanned operational disruptions – or worse – safe events.

Be in style [case study]

The second key factor is to ensure the alignment and overall process fits the “style” of the way the organization makes decisions and executes them. “Style” is one of the seven S’s represented in McKinsey & Co.’s well-known model for organizational success and represents how an organization makes decisions – it is hierarchical, collaborative, innovative, first-mover, or late adopter?

A very successful OT cyber security governance execution comes from a utility holding company, built on a “style’ of individual business unit independence and ownership of their own results.

The company’s incumbent governance model used the distributed business unit P&L ownership model, as seen by industrial organizations such as Emerson Electric, Illinois Tool Works, and Danaher.

The distributed business unit model intends to identify clear accountabilities around the “what,” such as targets and objectives, while allowing management of each business unit full authority as to the “how,” the strategies, and tactics used to deliver results.

The senior team at this utility holding company established a very clear, top-down directive, prescribing the cyber security objectives and standards each business unit was expected to achieve, down to the specific maturity levels of each sub-control, according to CSC Top 20 Control standards.

The CISO was heavily involved in shaping the processes and desired outcomes, but the “how” was left to the discretion of each business unit. While the business units were given authority to make decisions, they must fall within the specific set of objectives and metrics.

They needed to make decisions such as selecting the appropriate tools to deploy, balancing compensation controls, documenting specific approaches to achieving least-privilege settings, and determining action plans for incident response.

As with any approach, there are weaknesses. A few that come to mind are duplicating efforts, the inefficient use of underlying tools, missed opportunity to apply the corporate best-in-class approach to each business unit, the need for additional cyber security expertise where talent is limited, and the focus is limited to a set of standards, rather than reducing threats or time to remediation.

While these limitations may be glaring to some, keep in mind this organization did not have a culture of centralized experts or top-down directives of shared tools or infrastructure. To create such a model would require opposition to the primary mode of operation. Had the CISO tried to push in this direction, it would most likely end in failure because it was not in the organization’s DNA.

No governance model is perfect. Successfully OT cyber security leaders take time to understand the overall governance culture of their organization and build a model that works with the current flow, rather than trying to force-fit a theoretically “better” governance model. At that point, the CISO will address gaps in the approach to ensure limitations do not become hindrances.

This organization followed the alignment approach indicated in step one. They debated vigorously the required steps and gained agreement on the trade-offs needed to secure the environment. This produced a notion of a distributed approach to the “how” with a clear mandate on the “what”.

After achieving rapid progress, they began the process of realigning integrated and comprehensive solutions to drive lower cost and consistency. The learnings from the initial phase allow the team to centralize around a shared fact base of what worked and what didn’t to renew the alignment on phase 2 of progress for greater maturity and efficiency moving forward.

Different organizations may take alternative approaches given their organization “style”. Successful CISOs or industrial cyber security leaders first understand the unique approach their organization uses and develop alignment and implementation approaches to fit.

Determine holistic cyber spend

One of the most challenging aspects of governance is aligning budgets with accountability. Part of the business case includes establishing expected budgets to achieve a target maturity level. This sounds simple enough: Add up the technology requirements, additional headcount, potential downtime from the installation of new security processes, tools, etc. and use this as a long-term projection.

But the reality is different with many participants included in the overall cyber security journey. In many organizations, cyber security spend is distributed across the company.

Distributed OT or ICS cyber security spend across an organization may look something like this:

• Plants have responsibility for the budgets of their OT systems including updates, patches, and ongoing management. They may already be investing in technology or services that add or detract from security.
• There is often a budget to spend with OEMs and vendors on various software and services that are partially related to cyber security.
• Corporate IT manages budgets of network gear and segmentation
• CISO oversees spend on security-specific initiatives, such as anti-malware or monitoring logs for threat detection
• HR holds the budget for training and awareness development
• Facilities management is responsible for building systems, which are critical to operations

In this type of distributed environment, capturing total cyber security spend and prioritizing future budget for new protective or detective measures is difficult. But there are different ways to adapt to this situation.

Some companies create a shadow accounting system, aggregating spend from various business units into a holistic cyber security budget. Others ask business units to achieve established objectives while managing overall budgets in line with typical year-over-year increases, making trade-offs for spending on cyber security vs. other items. Still, other companies manage security compliance plant-by-plant to ensure budgets take cyber security into account as one key element to measure.

In many cases, security spending drives efficiencies in other “non-security” spending areas like network operations or software and systems management which require troubleshooting and onsite visits or OEM/vendor support. Cybersecurity investments can drive a reduction in cost because of its ability to analyze data and respond with centralized resources.

Whether your organization uses one of these models or another alternative, it is important to gain visibility into total cyber security spend in order to align budget authority with security accountability for effective risk management.

Adopt scorecards and KPIs

Successful OT organizations run on metrics, targets, detailed procedures, and tactical results that are monitored on an hourly, daily, and weekly basis. Cyber security objectives are often too subtle or aspirational: reduce vulnerabilities, identify potential malware, identify attackers, improve incident response by x%, etc.

The best OT cyber security approaches work with the flow of operations management to transform subtle objectives into tactical targets and metrics that can be displayed on simple red, yellow, and green charts. Let’s look at an example of an industrial organization that used this operational approach effectively. After adopting the NIST Cyber Security Framework, they implemented a set of measures to be tracked on a weekly, monthly and quarterly basis.

Each control area had a set of targets and metrics, such as the number of critical patches not deployed, number of machines without a backup in the last week, number of false positive alerts, time spent by operational personnel responding to false alarms, etc.

The corporate SOC analyzing threat data was treated like an upstream supplier of material. They were held to standards for threat detection quality and timeliness. The data was shared regularly between operations and the SOC to ensure accountability to one another. When items were not “in the green,” remediation plans were put in place, as they would for a product quality metric.

Operations are accustomed to managing a balanced scorecard of KPIs beyond product volume and cost. In addition to operational metrics, they manage occupational safety, environmental quality and product quality. Including cyber security as an additional element to the balanced scorecard, organizations align accountability with the authority to assign resources and take action.

Get tactical

The NIST CSF contains five core areas and 98 specific subcategories. CIS 18 has over 170 sub-controls. It is not practical for a high-level governance model to succeed across the entirety of these sub-elements.

Just as operations do, the OT cyber security team should build detailed procedures identifying accountable parties and their levels of authority for specific deliverables. Governance tends to break down at the micro-level. For instance, in the Identify component of NIST CSF, who oversees the asset database with required information? The IT department may take ownership, but an OT team could argue that running IT tools on OT networks are not safe or appropriate.

In some organizations, the information gathered from plant-level assets may be excessive to what corporate requires from a cyber security management point of view. In other organizations, there is an ongoing debate whether to patch a critical device immediately, leave it until an outage occurs, or leave it semi-permanently until the device is upgraded.

In critical operations, where a wrong, or even a correct, but delayed decision leads to lost production, injury, or even death, detailed and assigned decision-rights are crucial. Successful operators take time to thoroughly document the decision rights, as well as details such as who will take necessary actions in maintenance and quality.

Verve’s approach called “Think Global: Act Local” was designed to address these tactical challenges. “Think Global” refers to aggregating data across all vendor equipment across all sites (plants, mills, mines, buildings, substations, etc.) into a single database for analysis and planning. This allows an organization to reduce the cost of analysis up to 70% vs. conducting the analysis at each site or with each individual vendor. This requires an OT systems management platform to combine security and operational data into a common, secure database.

“Act Local” refers to remediating and response actions necessary to secure the system… the “last mile” as it were. So the centralized team analyzes threats, patches, configuration risks, etc. However, in OT, we do not want a team hundreds or thousands of miles from operations to take action. Therefore, the right governance ensures actions are controlled “locally.” This term does not mean “on-site” in every case, but refers to the local knowledge of controls experts in that process.

Importantly, these local actions can be automated and designed centrally, but distributed for local personnel to review, approve, schedule, and deploy appropriately. This “Think Global: Act Local” tactical approach drives dramatically improved efficiency while ensuring the reliability of the critical process controls.

These five principles should serve as a guide to designing an OT cyber security governance model that works with an organization’s current methods of running operations.