5 OT Vulnerability Management Challenges (and How to Overcome Them)
Common challenges to vulnerability management in OT cyber security and ways to overcome them to create safer industrial and operational environments.
Learn MoreSubscribe to stay in the loop with the latest OT cyber security best practices.
Before exploring hardware-related vulnerabilities that represent a set of specific cyber security risks, and potential vectors for threat actors to exploit, I want to introduce you to some basics in embedded systems engineering, their complications, the concept of errata, cyber vulnerabilities, and a few examples of attacks leveraging hardware such as Row Hammer.
In the Personal Computing (PC) world, most people recognize there is at a minimum a Central Processing Unit (CPU), Random Access Memory (RAM), a hard drive in some form, whether a traditional rotating design or a Solid State Disk (SSD), and a Network Interface Card (NIC), which provides Wireless, Ethernet, Bluetooth, LTE etc.
Those familiar with mobile devices such as cellphones or tablets, may recognize these devices have similar components, but some of their components use different technology than that of the PC (x86 or x86-64) architectures. Here it could be lesser amounts of RAM, different CPU vendors and architectures (e.g., ARM), storage, and even other features such as components that enable other technology such as Near Field Communications (NFC).
All the above share functional similarities and could even be the same components, but effectively, they are all tied together in a system of systems as illustrated in the next section.
In continuation, let’s look at an Internet of Things (IoT) embedded device – a smart thermostat:
As illustrated, there are several components making up an electronic device, but this is drilled down into another layer of products and components. These are often modular in design (e.g., an Integrated Chip (IC), or even as a System On a Chip (SOC)) and pre-packaged to quicken development time, isolate functionality, reduce compliance efforts and system complexity on the surface (e.g., these chips may be “black box” and simply “speak” via known and documented interfaces by their Original Equipment Manufacturers (OEMs)).
As computers are networked and talk TCP/IP, and a plethora of network protocols depending on the environment, application or task on hand, hardware does the same. Hardware communicates over several different mechanisms on a Printed Circuit Board (PCB) via specialized connectors, and carefully designed traces that connect the components based on various requirements such as latency, Signal-to-Noise Ratios (SNR), bandwidth, power consumption, synchronized and asynchronous protocols.
There are entire schools of thought, degrees, and studies related to the professional and art of designing electronic circuits or hardware, and signal analysis as well, but they key point to understand is that “hardware is quite complicated” and any number of security “bypasses” can exist given the number of integrated components or potential points often used for testing and manufacturing.
Unfortunately, those same complexities, test points, and components provide malicious parties (although not necessarily) opportunities to compromise the system and bypass technologies that enable the system’s cyber security. There is a ton of research to support this whether cryptographic bypasses or JTAG exploitation. Worse yet, many of those components in IoT widgets are exploitable because they are systems with their own resources (especially the 802.11/Bluetooth IC/SOC)!
With the media and many security organizations sensationalizing recent researcher findings such as “all intel processors can leak your secrets in X years,” you might be losing sleep with thoughts keeping you up at night such as: Is my computer vulnerable? What about Row Hammer: How will my data be secure in the cloud?
Your front door’s traditional tumbler-style deadbolt is also vulnerable to a capable locksmith or person with lock-picking skills. However, it’s also vulnerable to a pry bar, and/or a SWAT team door ram. And I suspect, there are a lot of valuables in your house too!
But the good news is hardware exploits are dependent on many factors related to time, resources, expertise, and specific revisions and designs of a component.
While it is usually difficult to accomplish, here are a few things to note when exploiting a perceived or theoretical vulnerability:
A key aspect that is often ignored is specific chip version, revision, and batch. Allow me to explain with an example:
Below is a screenshot for an errata document published by ARM for their reference Cortex-A7:
As we can see, it is classified as CAT B (which has its own implications under ARM’s nomenclature/classification language), and this document states errata ID 844169 applies to all revisions of the hardware vs. specific revisions.
The concept of hardware-specific vulnerabilities, flaws, or errata is not new. If we include these potential vulnerabilities in vulnerability databases for CVSS scoring, it would be a lot bigger and useful. Many of these vulnerabilities are far scarier and more prevalent than some of the ones seen in the media (e.g., Spectre, Row Hammer, etc.).
It takes a lot of work to exploit a hardware vulnerability, and the presence of a vulnerability does not mean exploitability, but this is only realized under specific conditions or by the implementation and/or configuration of the component. As an example, I’ve seen embedded Physical Access Layer (PHY) ICs receive and transmit network packets from the wire for forwarding to an Operating System (OS) over a Serial Peripheral Interface (SPI) bus, starve a hardware watchdog on a CPU, resulting in a non-avoidable reboot.
This type of bug is not present by looking at the stand-alone components, but rather end-to-end when examining the complete solution (therefore comprehensive Quality Assurance (QA) and compliance testing are required). The impacts of such a bug could have disastrous consequences where continual monitoring of an industrial process is required.
The answer to this question is that it depends. Hardware vulnerabilities and their related likelihood of being realized by a threat are dependent on multiple factors:
On the latter point, it is a question of risk transference and assurance, but at the end of the day, I do have to possess some degree or implicit trust otherwise the entire system would/will collapse. Hardware vulnerabilities should also be among those same trust but verify discussions with vendors and various independent certification processes and organizations.
You could go to your nearest vendor or solution’s provider and purchase the best new shiny solution that does not have known hardware vulnerabilities, but how likely is that? Not very in most situations. Row Hammer (and related e.g. drammer) attacks have not been publicly reported and are generally seen as non-practical, except under limited conditions.
However, Fortinet did find malware samples exploiting Meltdown and Spectre vulnerabilities, which affected all sorts of CPUs from ARM, Intel and even AMD by leveraging a technique intended to garner performance by improving memory caching via speculation. In this case, software patches and configuration are a possible workaround.
More recently, there have been other vulnerabilities in Intel CPUs that can bypass a variety of security measures buried into their silicon and your choice of mitigations is update firmware if possible, and your Intel product is not end-of-life (EoL), or live with the risk that an attacker could theoretically circumvent other controls while compromising your system. Not very helpful, right?
Realistically, not much can be done as a one-time fix (unlike taking your car to the dealership for a recall), so try to get a good night’s sleep to manage these vulnerabilities with a clear head and objective vulnerability management program component.
If you, the asset owner, are notified or aware of potential vulnerabilities in hardware that you may possess, then here are some strategies to implement to reduce your risks:
Cyber security is never deterministic, and hardware, once built, is even harder to fix when compared to software, especially in environments with extended deployment lifecycles (e.g., such as that in OT or critical infrastructure). But there is hope.
Manage vulnerabilities with continued investment, just as you would constantly update and maintain your vaccinations, or your vehicles tires. Hardware vulnerabilities are not the end of the world, and everything needs context in order to be properly assessed.
Common challenges to vulnerability management in OT cyber security and ways to overcome them to create safer industrial and operational environments.
Learn MoreVerve addresses OT vulnerability management challenges with software that gathers a full range of risk information and provides automated scoring and immediate remediation of risk from the same platform.
Learn MoreProtect control systems with 'Think Global, Act Local' for efficient and safe OT Vulnerability Management in 4 key steps.
Learn More