Can't Apply A Software Patch? Try These 5 Alternatives

Patch management is not always a solution, but it can be a means to an end to proactively reduce risks and vulnerability exposure where possible. However, in the industrial control system (ICS) and Operational Technology (OT) domains, what can you do if you cannot patch?

Your options will greatly depend on whether the system you intend to remediate has embedded vulnerabilities or a Windows/userspace application.

For these systems, patching will not absolve them of any true risk – plenty of insecure by design, and zero-days exist in these products.  However, an organization may choose to:

• Acknowledge the exception in a risk register if updates are not possible. Otherwise, where possible, upgrade devices to their latest firmware (if possible and relevant) during an appropriately scheduled outage
• Migrate to secure versions of protocols where possible (e.g., SSH vs. Telnet)
• Disable unused services or insecure services on the devices (e.g., HTTP is not needed)
• Ensure physical/run protection switch is turned off (and monitored/checked periodically)
• Protect the devices from unfettered network access by way of network segmentation, and ICS-specific Deep-Packet Inspection (DPI)
• Frequently poll and monitor devices for changes to firmware and project revisions, and or configurations/user accesses
• Protect devices from physical access and programming
• Create logging and detection use-cases on conditions that can be used to respond to unauthorized changes
• Develop processes to handle forensics, and replacement using spare devices (if available)

And while there might be secure deployment guidelines provided by the vendor or even followed to a “T” – they are likely insufficient.

Why deployment guidelines are not entirely sufficient for OT embedded systems:

1. Legitimate programming interfaces are a great platform to attack vulnerable devices due to their privileged position. These applications easily push an empty project file out to a device or make an unauthorized change with minimal effort given that the target device’s projects are often located on them too! They can issue enough commands across a network that results in a simulated denial of service attack… OOOPS!
2. Vendor software often has a variety of vulnerabilities either in their native code or in their dependencies. This excludes the Operating System (OS), but this software may have large issues that are easily exploitable (queue list of ICS-related CVEs) due to missing software patterns or gaps in input validation for example.
3. Programming/vendor software is on commodity platforms that are easily exploited. Even if the vendor software is vulnerable, it might not be easily accessible. But malware may traverse from one system to another over legacy workgroup communication protocols, and easily compromise systems (e.g., Eternal Blue or a variety of Ransomware attacks).
4. Management platforms are often transient (aka laptops), so they roam across networks while performing a variety of tasks with a complete “kitchen sink of software” – some of which are authorized, others which are not. In addition to the available software portfolio, these systems are not strictly monitored, poorly hardened, or left ignored due to a pervasive myth that security improvements on these systems will prevent technicians from making necessary interventions in times of need. Generally, this is not the case.
5. Human Machine Interfaces (HMI) systems or even those transient Windows machines might be a necessary participant when monitoring the safety and reliability of the process. In those cases, it might not be necessary to attack the “vulnerable devices”, but rather just ransomware the Windows machines that are effectively their caretakers, and the same goal of disruption is accomplished.
6. From experience, users and applications often execute actions with an over-provisioned account with weak passwords. For example, they are often running as Administrator, or with near Administrator-like levels of permissions, with a password that is easily guessable/default.  No need to hack a device when the management systems are wide-open.
7. Remote access, even for systems on-site, are often controlled or used through remote desktop functionality. Even if it is a thin-client, or easily accessible in person, remote support or administrative interfaces are often present, and poorly secured.  If another adjacent system is compromised, and credentials are discovered/defaults, then an attacker can easily move around within an organization.
8. Air-gaps are not perfect, and neither are diodes. When looking at the cyber security trifecta, the harder/more inconvenient security becomes, the more users become creative and bypass it.  Users often install USB wireless adapters and connect to corporate site Wi-Fi or their cellular hotspot. Or they use removable media to move software around if systems allow for the automatic mounting of external software.

You may have noticed I’ve avoided the question “what should you do if can’t manage embedded systems”, but the reasoning for this is that to protect them, you must protect the systems around them whilst adding a layered defense.

It is, unfortunately, the consequence and a factor of the realities of embedded industrial automation and control system (IACS) devices. It’s usually not feasible to “rip and replace them”, enough insecure functionality is present by design such that an upgrade would be irrelevant or require a complete redesign and deployment of all technology known today. That is not to say it is not possible to secure what you have, or that all embedded devices are insecure, but embedded vulnerabilities and exploiting them is often multi-part and nuanced. 

For example, a long-lived PLC with known vulnerabilities should not be deployed directly on the Internet, they should be sheltered, and often there is a Windows system (or two) communicating with them.  What would you recommend?

Well, fortunately, the solution is multipart and layered (see above example diagram with callouts). Get ready auto enthusiasts – it’s like owning a Front-Wheel Drive (FWD) car and driving through a rough winter storm.  The reality is you might have consciously purchased a vehicle that is widely acknowledged for having design flaws that are not able to be completely mitigated even with the best snow tires available, and you might wind up high-centered on a snowbank needing a tow. What would you do?

• Buy insurance to help offset the costs of replacement or repair, but not completely absolve us from liability or replacement costs (hint – it’s like cyber insurance)
• Keep our cars maintained (ideally) by frequently replacing consumables and ensuring they are optimal. Brakes are meant to break, and tires are meant to provide traction – ensure they have an adequate grip on asphalt.
• Drive appropriately for the conditions, at a decreased speed, or keep it to designated or cleared routes (e.g., roads frequented by city transit)
• Keep the car in the garage or parked if the conditions are poor (cannot crash if it stays in the stall)
• Selectively drive or opt to not drive at all/sell the vehicle/forfeit your license

Notice that “sell the vehicle” or “do not drive” are not the most feasible or reasonable solutions?  Replacing insecure systems when patching is not possible is a similar concept when you cannot patch a system due to:

• it’s 24-7 operation
• patches aren’t available
• when the product is End-of-Life (EoL) and works just fine!

5 alternative options when patching is not possible:

1. Protect access to the devices through network segmentation/isolation and secure the devices that protect these systems. If you cannot protect the embedded systems and need to rely on a “Maginot Line”. It should be as secure as possible, and that also means ensuring those assets are inventoried, hardened, and updated.  Network routers, protocol gateways, point-to-point RF devices, managed switches – all are a form of endpoints, and some are more commodity than they appear. Whatever you do, prevent direct accesses, or at least ensure multiple hurdles are present from exterior VPN/remote users.
2. Secure the Windows systems that often speak to vulnerable devices directly or through your “Maginot defenses.” OT Systems Management (OTSM) often means ensuring they are patched, have hardened configurations, policy enforcement, user-access controls, utilize firewalls and other technologies such as Anti-Virus, and application whitelisting.  It also means hardening the systems your users take in and out of the plant or use remotely from home.
3. Backup and virtualize the management systems if you cannot upgrade/harden the systems controlling/interacting with the embedded systems. If a secure “golden image” can be built, and it can be restored periodically to reduce “rot”, it might be a viable solution to keep a working system working as expected. This is especially true when many commodity malware target commodity IT (Windows) systems are left unpatched or migrated.
4. Frequently monitor for anomalous conditions, accesses, and changes. This is not merely about “passively” sniffing packets to look for write conditions (although it could be part of your strategy), but rather ingesting the event message sources from your networking infrastructure, using NetFlow, and commodity systems (including application logs).  It also means applying alerts on detailed asset inventory attributes to determine a change in logic or configuration has occurred. To do that, you need a frequent active strategy to inventory your assets and their data points, defined/tested alert use-cases to detect unauthorized access/changes/rogue systems, and the right processes/procedures to manage an event/incident.
5. Be prepared, backed up, trained to respond, and ready to act. It sounds pedantic, but many organizations lack sufficient documentation for cyber-related activities on the OT/IACS side of the business, but furthermore, adequate training/handling of events is often insufficient. Could your site overseers describe all cyber work that has occurred in the last 24 hours? Could they triage an incident? Are there adequate physical spares onsite for common controllers? Is there last known logic handy? Are there individuals trained to go from point A to Z?  Probably not, but this also means that if you cannot protect yourself adequately, and an incident is known to have occurred or is ongoing, you will move to containment and response.  The response typically is to isolate systems, rebuild from backups, and get the process back running ASAP – so all the pieces required to recover need to be recent, readily available, and working/tested.

After all, it is a strategic race, and a game focused on maximizing profit while balancing productivity and prioritizing safety.  It’s not a war of attrition, and it’s not “Command and Conquer”, but securing an embedded system with known (or assumed) vulnerabilities takes a grander approach to decision making.  And one of the pre-requisites is having sufficient detailed information for informed decisions, and the others are controlling your environment to control your cyber risk to a level the organization is comfortable with.