Vulnerability Remediation and Prioritization

This is the fourth post in our five-part series on Vulnerability Management (VM) in OT. So far, we have defined:

• An overview of what a robust vulnerability management program looks like, from planning to protection
• The value of a real-time, automated OT inventory in designing and supporting an OT vulnerability management program
• The challenges of vulnerability scanning tools in cataloging and reporting on asset-specific risks

This blog’s focus is on the remediation of risks. This is where the task of vulnerability management requires action. Software patching in OT environments is not as straightforward as deploying and rebooting the system.

Operating technology is challenged when a new patch or update comes out on a multitude of fronts. (To recap our specific discussion on how the recent Remote Desktop update went for two very different customers, read our blog here.)

Vulnerability management challenges in OT environments:

• Varying classes of assets = unique patching techniques
  • When assessing how many software patches are required, you may be overwhelmed by the large number your system reports back. But not all assets are created equal. Buried in those numbers are the most critical assets and contributing factors to help you prioritize which assets need immediate patching. Some of these factors include identifying your assets by location, OS type, criticality to operations, its configuration (or lack thereof) of security hardening parameters, and understanding if the system is redundant. Can we apply compensating controls? Does it have a recent restore image? These items are critical in triaging the order and urgency of patching, thus the importance of establishing a robust, real-time inventory asset management solution.

• Large number of ICS assets in scope
  • The amount of IP-enabled assets on a plant floor or along a transmission line vastly outnumber the staff tasked with supporting them. The range of asset OS-types, varying software and OEM warranty requirements means large scale automation of patching is uncommon. Solutions providing global visibility, but last mile OT oversight, allow for scaling scarce resources to support multiple sites.

• Alternative actions are sometimes required instead of applying a patch
  • There are many reasons why a patch cannot be deployed immediately, or completely, throughout an OT environment. Whether it is a permanent deferral due to a lack of vendor support or a temporary one while you roll out your test plans, other controls (like disabling individual ports/services/accounts, etc.) provide a necessary and valuable compensating control.

OT patching programs are not always easy or straightforward. OT patching does not look like anything you see in IT environments, where the latest patches and updated cycles are automated and enforced on a standard set of assets.

Recommendations for executing an OT-based vulnerability management remediation and patching program

An agent-based, automated asset inventory that allows you to append tribal knowledge, third-party security information (like backup or whitelisting status) and profiles the asset beyond just missing patches (like system hardening parameters) will provide the single biggest insight and value to triaging a host of patches across a patchwork of systems.

It is the fastest and most efficient way to cut through an insurmountable raw list of assets that need software patching.

A vulnerability management dashboard that showed over 35,000 total patches (low to critical) was filtered to show critical assets (operations deemed these assets to be critical to safe operations) with a critical risk that failed their recent backup and do not have whitelisting in lockdown mode.

The dashboard eliminated 34,934 risks to focus attention on the highest priorities:

Centralized oversight and support, combined with last mile OT oversight is recommended. We refer to this ideas as the Think Global, Act Local approach.

Visibility into multiple sites allows for a small, centralized team to research, test and help automate the deployment of patches. Centralized reporting filters views to target assets by region, criticality, owner, site, OS type, etc.

The agent-based approach allows for the deployment and installation of the patches. But even more importantly, the central patch deployment can be tuned to load files, not install at certain times of day.

Throttling bandwidth to that asset or segment, the patch cycle disables auto reboot of the command in the form of an offer. This means a local OT representative must log into the specific console and accept the command to install the patch. If he/she does not do so, the patch will not be installed. This is central visibility and specialty, coupled with boots on the ground, OT oversight. This is the best possible example of an IT/OT convergence in action.

What about compensating controls? An agent-based approach enumerates endpoint security settings like disabling the guest account from initiating remote access protocols or listing, then disabling, known bad ports or services if they are not needed. An agent tunes any parameter on the endpoint, providing the ability to filter at-risk assets, target specific compensating controls, and automate the execution of applying those compensating controls.

In essence, an agent-based technology, coupled with a central reporting capability, allows for the most effective, yet OT-safe approach to the remediation portion of a vulnerability management program.