4 OT/ICS Security Patching Lessons Learned from a Decade of Experience

Over the past decade, Verve Industrial Protection worked with clients across a range of OT/ICS environments – from water to power to oil & gas to manufacturing – to assist them with patching software and managing vulnerabilities. With our extensive experience in patch management, we developed a range of learnings that we leverage in our work, and that others in industrial environments can benefit from when it comes to patching OT systems.

First, Art Manion from CERT/CC presented a framework at the S4x19 conference to discuss what to patch and when that is referred to “Never, Next, Now”.  We would generally agree with the positions presented in that framework as well as the subsequent articles by Dale Peterson on this approach.

A simple structure of different classes of assets for patching makes sense. As one of the commenters noted, patches are often only applied annually during outages which limits the effectiveness relative to traditional IT patching. This would be Dale’s translation of “Never” to “Annual” (during outages).

This article, however, focuses on what to do when you decide to patch. What are our learnings that provide a framework of “how”?  Questions such as: Which patches should you apply on devices that are in the “Now” or “Next” categories?  How do you apply patches safely?  Which software patches are “security-related” and how do you capture all of those? What should you do with patches that are not reviewed by OEM vendors?

Verve provides a comprehensive ICS/OT endpoint management and security platform that includes automated vulnerability and patch identification, along with security services such as vulnerability assessment, patch review, and patch deployment. In conducting this work, we address these hard challenges of “how” we effectively, efficiently, and safely deploy the right patches to address vulnerabilities in our clients’ environments.

Patch Management Learnings

Ensure you have a comprehensive list of security-related patches by using OT-specific patching tools to gather complete software, vulnerability, and security patch information.

This sounds simple. IT security professionals will tell you to just scan it with a vulnerability assessment tool. Or, if you have an OEM-vendor relationship, use their “approved” patch list. Or, check the vendor’s website for patches labeled “security”. Or manually or automatically identify CVEs for the software/firmware on your devices and access the patches that address those CVEs.

OT patch management is far from simple.

• Vulnerability scanning tools negatively impact operations. It is key to use an OT-specific vulnerability assessment platform to ensure deep visibility without risking the sensitive OT assets.
• In many cases, operators do not have a robust asset or software inventory. Some rely on manual spreadsheets, others on passive inventory tools which only see what’s on the wire. Deep visibility to every piece of software on these systems, the accurate patch status on each, and tying those to vulnerabilities is important. (Note: this is why we built Verve with an agent-agentless approach to gather detailed software and firmware inventories rather than relying on traffic or manual collection. This is paired with real-time detection and updates to keep inventories accurate.)
• OEM-approved software patch lists often exclude patches for “critical” or “high” rated vulnerabilities. Without a comprehensive view of the vulnerabilities present and “relevant, if not approved” patches, operators are blind to their risk
• OEM-approved patch lists are NOT always inclusive of all third-party software deployed on OT devices. Yes, we know, you’re not supposed to load this software on these types of systems, but based on Verve’s vulnerability assessments, we know this is quite often not the case. Adobe, Office, Java, and a range of more surprising software are often present on OT systems. If these applications are not required, remove them. But if you are not able to remove them, ensure you are independently checking these other applications that the OEM vendor may never test.
• Often, OEM vendors’ labeling convention for their patches as “security” excludes patches that may have security-related content within them. Make sure you review not just those called “security” but all patch notes to capture all patches that have security-related content relevant to the environment.
• Without automated vulnerability management tools, a manual review of the CVE and other databases may lead to missed vulnerabilities if you’re just searching for specific versions. CVEs are not always structured in a standard format. It is critical that your automated tool or manual process has a deep view of the “details” in the CVE database and have wide logical search criteria to ensure you capture the full number of relevant vulnerabilities and therefore patches for your environment.
• Finally, OT/ICS embedded devices are constructed from a range of underlying software packages. The Ripple20 and Urgent11 vulnerabilities were major industry milestones as they identified critical vulnerabilities in underlying software components. Unless the vendor has a robust SBOM and active/aggressive vulnerability program, the asset that sits in your asset database may not appear in the vulnerability databases, even though the underlying components are vulnerable. Ensure that your vulnerability management tool can identify potential “unseen” vulnerabilities in the SBOM.

Review security patch relevance to your specific systems, status, and configuration.

A simple vulnerability assessment suggests the need for a patch, even though your particular system may not require that patch for a variety of reasons. The patch may not be “relevant” for that machine as other patches supersede it. Or, the configuration of the device may make the patch irrelevant because the device is not set up to run certain services or capabilities. For instance, many networking device vulnerabilities would imply the need for a patch, but further reading would find that the patch is only necessary if the device has certain enabled configurations.  Several key recommendations flow from this:

• Confirm your vulnerability and patch tools see detailed patch status, not just an OS version. This is why Verve leverages deep insight into all patches deployed and which ones are relevant, not just by looking at OS version.
• Ensure your vulnerability tool considers configuration settings and other compensating controls to reduce the need and urgency for a particular patch.
• Look to mitigate the “Never” or “Next” vulnerabilities with alternative mechanisms that negate the need for a patch if certain configuration settings changed or services disabled.

Ensure safe and secure install path and process.

Four issues must be considered in the patch management process:

• First, in many cases, ICS devices have not been patched in many years. New patches may require the operator to “walk” the patch path from earlier patches up to the most recent one. Deploying the most recent patch can cause operational errors and be difficult to reverse. Plan the patch walk carefully for each asset prior to beginning. An example would be a customer of ours with an OEM system that was never patched and had to walk through 10 years of patches, batch at a time.
• Second, make certain you have updated and tested backups. Prior to conducting any patching activity, the operator should ensure that the device has a robust and recent backup. Verve provides regular updates on backup status so you know before you begin whether you have a quality backup available in case you need to restore in case of patch issues. We have seen OEM-approved patches destroy an OPC server used for collecting setpoint information and controlling the process and the backups were invaluable in getting the plant up and running again.
• Third, confirm HASH values of patches prior to deployment. The Dragonfly attack several years ago highlighted the risk of insecure patching. In the North American power industry, NERC now requires this as a step in all patching. This should, however, be common practice in all ICS patching to ensure the secure delivery of patch files.
• Fourth, test and confirm. No ICS patching should be “spray and pray”. Take time to test on a set of devices. Test operations after deployment. Let those devices soak for some period of time. Then move on to other, similar devices. It is not enough to just make sure the device reboots effectively. Proper patch deployment includes testing of the devices and services after the fact to ensure that all key operations processes are in working order.

When patching ICS/OT HMIs, workstations, or servers, ensure “downstream” effects are identified.

Control systems patching challenges do not stop at the need to reboot. In many cases, changes to software at one part of the process have implications on other parts of the process. Worse, a “Now” or “Next” device patch could force a “Never” device patch.  So, you need to know the implications of applying such patches to a control system.

This can become complex and require deep knowledge of the OEM patching process. For instance, an update to the OEM’s engineering software may force updates to controller firmware or I/O cards, making it impossible for the standard software to work again unless these devices are updated.

Controller firmware updates may not have been in the original plan or may require an outage, but now loading a single OEM-approved patch results in the loss of all engineering capability until the firmware updates are completed on the controller and IO cards.

Verve’s 25+ years of OT security patching experience

Verve has worked across OEM vendor environments for over 25 years. We have a history of how these systems work together. We learned from these experiences that OEM release notes do not always include these integrated effects.

In some cases, the only way to understand it is to actually review the code/scripts of what the patch is doing.  Some will rely on “approved” patches as the solution. Our experience is “approved” does not necessarily work through all of the other elements and necessary considerations in many specific environments. A detailed review and testing are required to have a patch deployment progress safely and as planned.

Patching is a critical component of ICS cyber security. The “Now, Next, Never/annual” is a good framework of when and what assets to patch. However, identifying specific patches to deploy and how to deploy them (or mitigate them with compensating controls) separates successful and operationally safe patching programs from those that are less successful at mitigating vulnerabilities.

Verve’s 25+ years of vendor-agnostic OT systems management services along with the only endpoint management platform built for ICS enables us to help our clients deliver cross-vendor patching success, without risking critical processes.

Learn more about Verve’s patch management solution for operational technology.