Subscribe to stay in the loop with the latest OT cyber security best practices.
For the past 25+ years, Verve Industrial has designed and secured complex, multi-OEM vendor environments for industrial, manufacturing, and critical infrastructure organizations. This experience allowed us to work alongside dozens of CISOs and hundreds of senior IT security leaders to bring the same level of security to OT systems as they have in IT. While industrial organizations struggle with differences in technology, there are even more differences in the mindset and priorities of OT leaders.
The following guide was built from our experience of how a CISO – or the CIO or other senior leader assigned to drive cybersecurity improvement – of an industrial organization can make rapid, consistent progress.
Operating Technology (OT) is in the crosshairs of global cyber security attackers. The need to address OT cyber risks has never been greater. New threats are emerging every day. According to IBM, manufacturing, and energy are now the second and third most targeted industries, respectively, increasing in only a year from eighth and ninth.
Why? As Willie Sutton said when asked why he robbed banks, “because that’s where the money is”. Operating technology is critical to keeping industrial operations running. Downtime is expensive. As a result, ransomware gangs – whether private or government-supported – have discovered the financial opportunity from targeting industrial organizations, large and small.
In the past two years, ransomware has moved from the sixth biggest risk in ICS security to the first. Now over 50% of ICS security leaders say ransomware is their top risk, based on the most recent SANS survey. These ransomware attacks cause significant financial and operational impacts. In the past 12 months alone, multiple companies had 5-50 days of manufacturing downtime due to ransomware. Over the past several years, Merck, Maersk, Mondelez, and others have suffered hundreds of millions of dollars in losses from ransomware attacks.
Industrial organizations are now fighting a war, whether they know it or not. We have coined an acronym for this battle – AIR-RAID:
Taken together, these seven drivers are dramatically shifting and accelerating the requirements for OT security. Gone are the days when just monitoring the perimeter firewalls for anomalous network traffic was enough. “Visibility” is just the beginning.
With the current assumptions in place, CISOs (or the person responsible for OT cyber security) face the following challenges:
So where does a CISO begin to address these challenges? Follow our guide to learn how cyber security leaders can manage risk and improve the overall OT security program with these best practices:
CISOs have protected IT systems for over a decade. You likely have dozens of tools at your disposal to address cybersecurity, so do you really need a specific OT cybersecurity program? Our conclusion after 25 years of OT cyber experience is “Yes, you do.” There is much you can leverage from IT security, but the unique characteristics of the OT systems, processes, and resources require a different, OT-specific approach.
For many CISOs, CIOs, or IT security leads, OT is a bit of a black box. OT systems have historically been managed by a completely independent group of people, often not centralized, but managed on a plant-by-plant/mill-by-mill/substation-by-substation model. They have been the domain of process controls experts and OEM vendors. They rely on mechanical and electrical engineers rather than a computer or IT engineers. They control physical processes, not information processing. As a result, IT and their security counterparts often find their traditional approaches break down as they cross over into OT.
For years, IT teams have secured their networks and endpoints using a range of defense-in-depth models. They act to manage users & accounts, software vulnerabilities and patches, configuration risks, network connections, least privilege access, etc. They leverage endpoint detection and response to rapidly take remediation and response actions. They carefully manage network devices and connections to ensure rules are enforced and reset in case of inappropriate change. Fundamental to these defenses is the active management of systems such as networking devices, servers, cloud applications or laptops.
In most OT environments, however, this type of “systems management” often does not exist based on assumptions about the systems within the OT networks:
Important: This is not to say that OT systems can be managed the same way as an office worker’s laptop. There are many unique characteristics of these operational assets combined with the processes they control that require an OT-specific to systems management:
Many organizations try to apply their overall cyber security program to the OT systems which leads to pushback from the controls systems experts and, perhaps, negative operational impact from systems that are tripped or shut down because of inappropriate scanning or patching or network changes.
Because of these challenges, many have turned to passive tools which offer a promise of visibility and detection with a minimal presence on the network/assets. But these solutions lack key elements: a.) They lack deep visibility of each asset such as all the users and accounts, password settings, unused software, accurate application patches, etc. b.) They cannot remediate potential risks other than possibly integrating with a firewall ruleset. The user is often left with a limited view and a set of alerts that need much further analysis and manual tasks for resolution.
However, if industrial organizations are to truly improve their operational cyber security, they need to question these four assumptions to begin actively managing the security of these systems.
Delivering on the required defense and reporting driven by the AIR-RAID requires CISOs and IT teams to question the assumptions highlighted above and the current approach to managing these OT systems. They need to bring the same core principles of IT security (endpoint management, taking remediation and response actions, defense-in-depth) rather than relying on network monitoring and “Purdue models”. They also must work closely with OT to do this without disrupting operational resilience and uptime.
Almost all industrial companies have some level of cyber security underway. But often the question is where to focus first to improve the security of the OT systems. Options usually include network protection such as segmentation and separation, endpoint protection, network anomaly detection, asset visibility and inventory for improved vulnerability management, security event monitoring, and analysis.
But before an organization starts with tools, technologies, or standards, the most important objective is to build the right team. In many IT organizations, the answer is clear: Security requires networking, endpoint, cloud, regulatory, and other IT partners. In OT security, however, getting the “who” right is critical and often more complex.
Depending on the organization, the “who” may include the head of process control technology, the SVP/EVP/VP of operations or manufacturing or supply chain, influential plant managers, quality, or similar regulatory personnel. This is on top of the more “typical” groups involved in IT security.
Successful CISOs create a steering committee of IT and OT personnel in addition to the operations leaders who understand the technical challenges of the systems. When key personnel is not included early in the process to identify bottlenecks or technical challenges, the entire cyber security program stalls.
This joint team is imperative for organizations to gain buy-in for the necessary technical changes and required support personnel to achieve success. Together, this group forms the right process for deciding aspirations, technical feasibility, etc. It is critical for the CISO or security leader to bring these groups together early on to align on objectives, expectations, and decision processes.
While there are many ways to organize the teams managing a cyber security program, it is ultimately determined by the way the organization is structured more broadly. Is the culture top-down with a drive for operational consistency, even if that takes longer to align different parts of the organization? Is the culture driven by set targets with business units to determine how to hit those targets? Is there a close working relationship between IT and OT today? These questions inform how best to organize your approach.
Several design elements differentiate successful from unsuccessful leadership.
Regardless of structure, key elements for optimizing OT security include the following steps:
One of the biggest differentiators between success and failure in OT security programs is clarity of purpose, expectations and what success looks like. This challenge also exists in IT security given the constantly evolving threat landscape. In OT, however, the issue is exacerbated by the system management realities.
Each organization has its own set of priorities – uptime, protection of critical intellectual property, safety, etc. These are the ultimate objectives of any security program. But the question is how to define a set of measurable cyber security goals to achieve the ultimate objectives. The good news is that there is now a range of standards that mirror the concepts in IT security standards. This list includes NIST CSF and several industrial controls components of 800-52., IEC 62443, CIS Top 20 (18), etc.
According to the 2021 SANS ICS Security Survey, NIST CSF is the most used framework, but as seen below, companies use a range of different frameworks.
Establishing a clear maturity objective against one or a combination of these frameworks and guidelines is key to driving results. One of Verve’s most successful deployments used the CIS Top 20 controls to apply the same framework across IT and OT. They worked as a joint team to define which controls to adjust for the OT environment and defined technical feasibility exceptions as necessary. This approach allowed for a much more integrated cybersecurity program that the CISO, C-suite, and board of directors could manage more easily.
The key is to balance the analysis-paralysis that may arise when customizing targets to the specific risk factors in your organization while making rapid progress against these rising threats. Certainly, no organization wants to waste money applying challenging controls on assets that are less critical to operations or safety.
Any good cybersecurity program will prioritize the remediation based on potential impact. By the same token, we have seen many organizations spend months trying to define these different enterprise risk frameworks, delaying the critical work at hand.
Based on this experience, we strongly recommend doing these activities in parallel. There are many objectives and standards that will not raise an argument – e.g., gathering a robust asset inventory or ensuring firewalls are properly configured – but there are others that could drag on in defining the right approach. Our suggestion is to set a milestone of two months or so to define those risk ratings to ensure there is a timeline to get answers and move forward.
Frameworks and target maturities must be matched with targets for efficiency: comprehensiveness of attack surface view, response times, dwell times, and total cost of ownership of the program. Being intentional about what “good” looks like for these metrics is also critical.
Perhaps the most challenging part is budgeting for OT cyber security. Each organization should decide what is realistic and relative to its risk. Driving efficiency in most industrial organizations has historically meant minimal investment in the operating controls systems. Many controls have been in operation for years or decades which is different than the three or four upgrade cycles in IT.
Similarly, there are not usually dedicated resources for managing the “IT” devices which are now prevalent in OT. It typically falls to process engineers who are already overloaded with tasks. In short, security is a “new spend” in an industry looking to reduce current spend. To define the budgets appropriately, a well-defined business case is critical. This budgeting exercise should explain the potential financial and operational risks and the likelihood of those events based on the risks of the environment.
Regardless of budget, the total cost of ownership (TCO) is a key target to agree on. Too often, initiatives are kicked off and tools are purchased with little insight into the ongoing cost of the labor to manage those effectively. One example is the notion of separate security tools for each OEM system, managed at a plant level. This creates huge administrative costs and limits the ability to scale internal resources to manage the security across the whole OT enterprise. Clear objectives should be set of what is expected for long-term management costs, including the labor components.
As in most cyber security efforts, beginning with an assessment allows the organization to develop a comprehensive roadmap of gaps and remediation initiatives. The most effective way to do this in distributed, global organizations is to leverage a technology-enabled approach. The traditional approach requires surveys and review of documentation on assets, networks, procedures, etc. In almost every case, this leads to an ineffective assessment. In many (most) organizations, asset inventories are inaccurate or non-existent, network diagrams are out-of-date or non-existent, and the survey respondents do not know the answers to the key security questions.
A technology-enabled approach solves these challenges. It deploys OT-safe software into the operational environment to gather endpoint and network information, rather than relying on document review of inaccurate information. This technology provides deeper visibility and allows the team to make appropriate prioritization of different risks and remediation actions. But then, most importantly for making progress, the software enables you to immediately remediate rather than wait another six or nine months before the actual security adjustments can begin.
The foundation for the technology-enabled assessment is a robust asset inventory with “360-degree” visibility on hardware, software, network connections, users and accounts, vulnerabilities, etc. To make network protection effective, you must know what you are protecting and how it needs to communicate. To make proper risk management decisions, you need clarity of the comprehensive 360-degree risk because, in OT, not all assets can be patched or upgraded.
Alternative “mitigating” controls may be required and prioritization is key. Security event monitoring requires knowledge of the assets to monitor, their operations and asset criticality, etc.
This 360-degree view provides comprehensive visibility into the risks and how they interact. For instance, two devices may have similar vulnerability or patch status, but one has application whitelisting locked-down, a robust backup, hardened configuration settings, and sits behind a well-configured firewall, where another does not. Or one operates critical operational processes whereas the other does not. Even more so than in IT, these relative priorities are critical in OT given the challenges of taking rapid remediating actions.
A 360-degree asset inventory is an initial step in almost any cyber security standard. The good news is that many of the standards familiar in IT security can be applied to OT with the right adjustments. You can apply many of the same controls and gain consistency across IT and OT such as CIS Top 20 or NIST CSF.
The risk view naturally leads to a prioritized roadmap of initiatives to close those gaps. Unfortunately, many organizations jump to each individual initiative without considering the necessary sequencing involved. Success for many of these initiatives requires prior foundational elements in place. There is an open debate on whether asset inventory and visibility, for instance, should be the initial element of an OT cyber program when IT-OT segmentation can generate much greater security than just knowing what your assets and risks are.
This is hard to argue with until you realize that conducting proper segmentation requires the information that a robust asset inventory provides. Deploying a firewall sounds simple enough, but often the communications paths, assets to protect, applications required are not well documented and current staff was not in seat when the system was initially deployed. Gathering required data is nearly impossible without a deep asset and network inventory that informs the rules and movement of certain devices behind the firewall.
This is just one example where a lack of thought-out sequence leads to the program bogging down. For instance, a push to deploy application whitelisting sounds like a great idea, but the key devices may not support the whitelisting agent, or they may not route back to a central location to manage the lists, etc.
The same goes for rapid deployment of backups or other tools as well. This becomes an even bigger issue when the initiative includes monitoring network traffic and conducting deep packet inspection. In a pilot, this looks very easy because the traffic is narrowed to see what is coming through an individual switch. But in reality, gathering the depth needed requires a significant investment in spans, taps, cabling, collectors, etc. to see the packets to detect real threats.
A roadmap should break down these initiatives with specific timing and expected security maturity achieved within each step. One example is below.
Most industrial organizations have an existing IT security organization that manages its IT cyber security program and, rightfully, want to leverage that for OT as well. In fact, there are many components of those IT programs that are relevant and leverageable in OT but defining where the differences are and how to manage these unique OT factors is important.
Recall the real differences between IT and OT listed earlier in this guide (e.g., sensitive embedded devices, lack of resources, etc.) and the questionable assumptions placed on OT security (e.g., network detection is the only effective approach, OEMs control security on their systems, etc.)… The program management must recognize and address the realities of the OT environment but question the ingrained assumptions to achieve efficiency and security effectiveness.
The program will likely move through three phases. There is usually a one-time significant remediation effort, but the program must also consider the maintenance component.
By applying OT Systems Management using a “Think Global: Act Local” approach, cyber security achieves the same type of security principles in OT as they do in IT without creating operational risks to the process.
The single largest challenge in OT security is the resource constraint of skilled personnel that understands cybersecurity as well as OT. Any security management must start by addressing this challenge.
Over the past 20 years, IT developed silos of security functions and toolsets to address the many requirements of security and systems management as these requirements grew organically over time. This approach, “Think Global: Act Local” promotes specialization in vulnerability management or user access management or network management. While no one argues this is a perfect situation in IT, it has been workable due to the scale organizations gain across very similar, connected IT endpoints and the ability to generally apply each security function relatively independently of one another. For instance, if a device needs to be patched, patch it. If a threat is discovered, take response actions. If the anti-virus is not up to date, immediately update the signature, etc.
In OT, however, the specialization assumption fails. First, each asset needs to be managed based on its operational context, OEM vendor approvals, local process knowledge, etc. As a result, knowledgeable OT personnel must be involved before centralized remediation or response actions are taken. Similarly, the potential remediating actions need to be considered holistically across various defenses, allowing the most effective, efficient, and operationally least disruptive response to be selected. Patching, configuration management, network protections, antivirus and whitelisting status, user and account management all need to be balanced into the risk of an asset to determine the best remediation tactic to take.
Finally, as mentioned above, OT cyber knowledgeable resources are very limited, therefore the organization needs to scale the analysis across globally distributed sites, hundreds of different device types, and sub-segmented networks.
A “Think Global” approach consolidates the risk data across all OEM systems and all sites into a centralized database. First, this drives the necessary efficiency in security analysis that is impossible if the data is only available at the local site or for a specific OEM. Second, it enables the right risk response and mitigating controls trade-offs. So we are, this does not mean one centralized vulnerability database and a separate user and account management database, and a separate AV/whitelisting status database. The ability to “Think Globally” requires a non-silo’d view to make the appropriate trade-offs. This pushes against the assumption of leveraging IT vulnerability management resources.
The “Act Local” component means that the OT security remediation and response actions need to happen “locally” – i.e., within the purview of the OT personnel that know the process best. Any OT security management approach must enable OT personnel to scale their involvement in the final approval and execution of these actions to ensure both buy-in to the security program and the operational resilience of the process.
The second half of this approach, “Act Local” is critical to protect the reliability of OT systems. All actions (patching, configuration hardening, user & account management, OT network, etc.) must have “local” oversight and control to ensure resilience. By “local”, we mean someone who is knowledgeable about the process. It may not be “local” in the physical sense if the system is a distribution grid or pipeline, but the representative has control over the operational processes.
Every process control engineer has either personal experience or a story about how IT disrupted operations by running a scan or patching a system or making a network change without appropriate operational review. The “Act Local” approach ensures remediation actions are under the control of an operationally knowledgeable person.
An example is probably in order: The “Think Global” component has visibility into required patches across the environment. This exists not by active scanning, but through OT-safe collection means (see Verve’s OT endpoint management article). The centralized platform aggregates approved patch lists from each major ICS OEM vendor so the centralized team can quickly develop remediation automation actions for distribution to local sites.
“Think Global: Act Local” empowers CISOs to adopt an OT Systems Management approach, mirroring the same techniques that IT conducts on IT systems (and actually represent over 75% of all IT security tasks). This includes functions such as patching, vulnerability management, configuration management, user and access management, etc.
However, one significant difference is these actions are applied in a manner safe and effective for OT systems. For instance: gather real-time vulnerability information without scanning, conduct patch management in coordination with the process controls engineers who understand the appropriate timing and implications of applying those patches, apply compensating controls such as firewall protection or application whitelisting when patches cannot be deployed, use endpoint management tools to manage users & accounts that are not tied into an enterprise active directory server.
This comprehensive set of management actions ensures protection and hardening of these devices in advance, as well as the detection of anomalies of ongoing attacks. They also align IT and OT security into consistent practice areas for monitoring and tracking.
This begins with continuous 360-degree risk management and visibility. As stated earlier, this means a real-time, complete view of network and endpoint risks. Asset inventory is the foundation of any comprehensive security program, but not all inventories are created equal. To provide a comprehensive view of risk, the inventory needs to include all the assets, whether they are OS-based (Windows/Linux/Unix) HMI’s or servers, network devices, PLCs, relays, RTUs, or other types of embedded gear. A surface view based on what is communicating through core switches at the top of your segmented network is insufficient.
The inventory needs to include the make, model, OS and all application software, users, and accounts whether in use or dormant, all network firewall and switch rules, password and configuration settings, status from key protection and recovery tools such as application whitelisting or backups, etc. Finally, an inventory needs to provide the asset’s OT context or criticality to safe operations. This type of 360-degree inventory allows for security practitioners to accurately assess and prioritize appropriate trade-offs and risk remediation activities.
What sets a “360-degree” asset inventory apart from other asset inventories? OEM agnostic, a 360-degree view requires the consolidation of all data at an enterprise-level rather than isolated at each plant. It captures data directly from endpoints rather than network traffic. It is acquired in real-time, so analysts are informed of new risks as they arise.
It then continues with the need for ongoing OT-safe remediation and detection and response.
Security is much more than detection. A CISO wouldn’t accept that the only security possible for IT was network protections and network monitoring. Patching, vulnerability remediation, user and account management, configuration management, anti-malware, endpoint protection, etc. are all necessary components of a mature security posture. Similarly, when a threat (or possible threat) is detected, the organization needs the ability to quickly respond by taking action. There is a reason there is an “R” in EDR and XDR. To protect critical systems, you need to respond. Today, if these activities are done at all in OT, they rely on many manual process steps, individual tools which cannot be managed centrally, do not integrate key data to provide a single risk view, etc.
Over the past dozen years, Verve has proven that organizations can act to protect their OT endpoints in an OT-safe manner. Active OT security management identifies the most effective, efficient, and operationally practical means of protecting each endpoint. For instance, a patch may be needed, but not approved by the OEM vendor as it may cause operational disruptions.
OT security management would prioritize alternative mitigating controls (e.g., harden configuration settings, ensure application whitelisting is locked down, ensure least privilege and limitation of administrative and local accounts, etc.). OT security management applies those security controls within the control of the OT operator.
The assumption that OEM vendor-agnostic endpoint protection is not feasible or efficient in OT needs to be challenged. The assumption that network packet monitoring is the only real option for OT security needs to be challenged. The assumption that an organization cannot apply the core elements of systems management on OT devices is wrong. It can be done. And it can be done without disrupting operations.
CISOs are in the hot seat as threats to industrial operations continue to rise. We believe a robust OT cyber security program implements the necessary structure for security leaders to make – and demonstrate – quantifiable progress in their cyber security posture.
For more information on managing risk in your OT/ICS environment, please contact us.
To develop robust OT cyber security roadmaps and foundations, organizations with OT systems (everything from manufacturing process controls to building control systems to security access systems) should embrace the concept of OTSM (OT Systems Management), paralleling their ITSM practices, but within the unique environments of operating systems. Achieving a mature level of OTSM is critical to improve overall ROI from…
Learn MoreThis webinar focuses on how to raise awareness of the cyber risk in your industrial environment. Many people see the threats facing their controls systems environments, but do not know how to best raise awareness and mobilize the organization to act. In this webinar, we share experiences in how to create a shared awareness of risks and urgency with IT…
Learn MoreWhile there has been significant emphasis on the pursuit of asset inventory as a starting point for many OT cybersecurity programs, asset inventory is the first and most important step in your cybersecurity journey. But many industrial organizations fail to recognize it is only one component of security maturity and end up falling short of their end goal. Hear from…
Learn MoreJoin Verve’s experts as they share key insights from 2024 and practical strategies to secure your OT environments in 2025. Don’t miss this exclusive webinar—reserve your spot now!