Embedded OT Vulnerabilities: An Asset Owner Perspective

Embedded systems are often thought of as something that resembles a circuit board, as Programable Logic Controllers (PLCs), Internet of Things (IoT), or even the widgetry buried inside of enclosed product. 

Regardless of the buzzword, in the most traditional sense, it is the electronics that provide some sort of defined functionality inside of a product. It could also be multiple pieces of “embedded” electronics tied together to make an embedded system.

The real challenge for asset owners, consumers, and those even from the PC/Server/IT crowds is comprised of several questions:

• What is in these systems and what makes them different than commodity Windows/IOS/Linux systems?
• What is the software inside of them? What is firmware? How does that all go together?
• How are these vulnerabilities different from those on a Windows box for example?
• How come embedded vulnerabilities are harder to remediate?
• What can you do about them?

This is a huge topic, worthy of multiple degrees and decades of study, but for the purposes of this blog, we will begin with an initial explanation of the above questions, and provide a longer (but still brief) discourse on the topic in this paper.

What is an embedded OT system and what makes them different from IT (OS-based) devices?

In short, an embedded system has an operating system (OS), a CPU, storage, memory, and often several features that are present such as those on a laptop (for example).  And as the days march by, and architectures rise or decline, the lines between the two become blurry.  Regardless, in Operational Technology (OT), embedded systems are:

• Purpose-driven systems designed to operate 24/7/365 for extended lifetimes.
• Non-generic and unable to execute a wide ecosystem of applications. They are not multi-purpose in comparison to the variety of workloads seen on your typical workstation or server.
• Closed off/proprietary, inter-actable over a few select channels, and often their updates come only from their manufacturer (e.g., the OEM).

You may say on the last point that these systems run an OS like Linux, and that’s not closed source, right? Yes, Linux is Open Source, but the vendor often has customizations (you could request the source code though) and does not have to provide you the right nor the mechanism to interact with the system or load your own software.

Therefore, Tivoism highlights one fundamental difference between current IT systems.  On the other hand, even if a proprietary OS is used (such as VxWorks), it doesn’t necessarily improve security in any way shape or form.

There is often a variety of testing, product motivations, intellectual property (IP), and most importantly, highly specialized hardware/software that cannot be easily updated.  And the latter, is another fundamental difference that again explains (besides environmental/operational constraints) how they cannot be trivially updated in the same way ubiquitous devices often can be administrated.

From a general perspective, the tighter the link between software and hardware in purpose, function, speciality, and product – the more embedded and different it is from a traditional enterprise device even if they share similarities (e.g., a IT router vs. an OT din rail router/switch). 

What is firmware, what is contained in it, and how does it relate to embedded systems?

Think of firmware like a file folder, or better yet, something closer to a DVD ISO image.  An ISO image is a standard that effectively wraps up files, configurations, binaries, and even operating systems in a format that makes it possible to be installed onto media for a variety of uses (e.g., booting a Linux live OS, or backing up files).

Following that analogy, embedded firmware can be broken up into multiple sections, contain a hodgepodge array of components, be a complete image that has all the bits and bytes necessary to make a product a product, or even be as slim as a “hotfix”.

In the following example, we see that embedded firmware might have multiple components such as a Kernel, a Bootloader, binaries/applications, and even files or a complete Filesystem (FS).  These initiate a system, start the operating system, and run all of our applications. Without them, you would not have a functioning device, but merely a hollow shell of a product.

So a firmware (or firmware update) from the end user perspective can be end-to-end and contain all or a few of the above components, or a select few in four scenarios:

• The complete software necessary to start, and ready a device for end-consumer usage
• A complete update to the system
• A partial update to the system
• A user-configuration change on the device

Technically, it doesn’t matter about the nature of the word firmware, but rather to be aware that multiple types of firmware might exist inside of a firmware or “update”, and so it is best to keep track of them where possible; this includes configurations.

What vulnerabilities are found in embedded systems, and how do they differ from traditional IT vulnerabilities?

Given these types of systems do not often run on commodity hardware or architectures seen in usual enterprise environments, and they are closed, the differences in vulnerabilities is more about what can a specific vulnerability “do” when exploited (e.g., the impact), the veil of false security when embedded in a product casing, and the nature of the flaw and remediation.

Therefore, the differences between commodity systems and embedded OT devices are seen better by observation in the following Venn-diagram:

There is overlap for certain types of devices (not depicted), but the “openness” of the platform and the standardized hardware plays a key role in the wholesale ability for enterprise/IT to easily identify and remediate identified vulnerabilities.

Without explaining the types of vulnerabilities specifically or what a vulnerability is, it is important to understand vulnerabilities differ from data-focused ones because in OT embedded systems, their stability, integrity, reliability, and continued safe operations are of paramount importance.

For example, some security primitives such as the need for completely encrypted and tunneled industrial network protocols are not necessarily desirable (or required), but ensuring secure updates or the stability of a device such that it cannot be subject to memory management flaws when parsing network packets is of far more importance; especially those packets are used in a critical application and the device can be crashed.

From experience, a good approach for examining vulnerability disclosures (or CVEs) or even “insecure by design” risks is to look at them this way:

• Examine the nature of the device, the function it provides, and how it contributes to the operation/revenue of the business.
• What are the Health, Safety, and Environmental (HSE) consequences if a device is exploited successfully?
• What are the controls in front of the affected device?  Is there a more secure option that could be used?
• What is the organization doing to provide comprehensive security for vulnerable systems? From all perspectives across People-Process-Technology (PPT)?
• And what about the stuff not being reported such as insecure logic programming practices, physical access, or vulnerabilities hidden within the façade of the product or the ICS protocols themselves?

If you can answer those questions, then you might have a better chance in assessing a vulnerability vs. relying solely on a CVSS score (which is incomplete and without context) and be able to prioritize and engage in remediation if possible.

Why are embedded vulnerabilities harder to remediate, and what can you do about them?

Besides the highlighted closed nature of embedded products, their customized function, or less-than-frequent updates through closed OEM channels, the future is not as bleak as it once was; especially if we consider the number of abandoned/end-of-life and insecure by design products in the OT/ICS universe.

The reason for that positivity, is that over my career, and interactions with devices or their development, the commoditization of hardware and its ability mainline into ecosystems such as Linux has reduced the need for completely proprietary, and hopefully this will:

• Improve the matching of identified vulnerabilities to products (e.g., SBOMs)
• De-fragment the product space by making updates easier and more generic
• Improve security in products overall due to cheaper vendor costs (less custom code, equals more automation and product maintenance overhead, which will increase feasibility for OEMs)
• Offer more secure solutions and functionality out of the box, and allow the vendors to focus on with they are good at: OT applications

Even if the above have or will improve OT cyber security and robustness of embedded products, vendors are the bottleneck in providing updates. The asset owner needs to apply updates, and we still should always assume that embedded solutions in OT are subject to a number of environmental challenges such as:

• Will not patch, or will delay patching
• Specific windows of downtime, or limited resources
• Testing and change controls/processes
• Abandoned/EoL products
• Forever days and insecure by design protocols or functionality

Still, with respect to securing the device, a multi-layered approach is recommended:

• Assume all devices are insecure at some level
• Inventory your devices at a sufficient level of detail, and securely manage their configurations & logic
• Take precautions such that there are multiple layers of controls to reduce exploitation
• Prevent access to devices to only authorized systems, users, and connections
• Configure devices as securely as possible with available security features
• Monitor and frequently inventory these systems for changes
• Request that vendors have security programs, and certifications (e.g., ISA/62443)
• Always have a process to assess and handle cyber security appropriately for embedded AND OT/ICS systems

Of course, this is only a small array of items an asset owner can apply to help remediate cyber risk to tolerable levels, and securing these devices is both feasible and worthy. For more information – please download the complete white paper on Embedded Vulnerabilities, and register for our upcoming webinar on November 5th where we will cover this topic in more detail.