Securing Embedded Systems Takes a Focused Approach

Embedded systems play a significant role in all of our lives, showing up in everything from the simplest home routers to the most complex critical infrastructure, IoT, and industrial controls. Securing these systems, however, remains challenging not just because of their proximity to hardware and design, but also due to their operational environments.

Today, most embedded hardware includes dozens of physical components and packs software complexity similar to that of early PCs.  In the quest to secure them, we need to ask:

• What is an embedded system in the context of Industrial Control Systems (ICS) and the Internet of Things (IoT)?
• How do today’s embedded devices differ from those of the past?
• What software runs on embedded systems?
• Why can’t everything be solved with a silver bullet?
• What risks do modern embedded systems pose to their owners?
• What can defenders do to mitigate that risk?

Defining embedded ICS and IoT/IIoT systems

The proliferation of small computational devices such as smartphones along with the commoditization of network infrastructure has blurred the definition of embedded systems. When we talk about such systems in the realm of ICS and IoT, it’s important to distinguish them from commodity systems. Think of them like this:

In OT/ICS as we know these as devices like Programmable Logic Controllers (PLCs), Distributed Control Systems (DCSs), Remote Terminal Units (RTUs), Variable Speed Drives (VFDs), flow meters and valves, relays, TCP-to-serial gateways, embedded Human Machine Interfaces (HMIs).

On the consumer IoT front, or even among the new PLC 2.0 type devices that comprise the so-called  Industrial Internet of Things (IIoT), they might also include integrated mesh-based sensors, smart lighting and locks, smart refrigeration, and so on.

While some of these devices might run Linux, use Bash, and run Python, many others have specialized hardware that leverages microcontrollers, FPGAs, and real-time operating systems (RTOS) to ensure predictability and deterministic behavior without the need for a full-blown OS.

Today’s embedded devices are different

In the symbiotic relationship between the hardware and software that comprises most embedded systems, it’s the software that largely dictates when and how the device works.

In the good ol’ days, computational resources were scarce; the performance of embedded devices depended on proprietary software/hardware and low-level programming languages.  Programmers and engineers knew where every byte was going and there was little extraneous code.  While occasional flaws — those found in the C programming language, for example — were present, such problems were practically non-existent when compared to embedded environments of today.

Embedded systems tightly linked to hardware still exist, but many modern devices now run common software, application frameworks, modern protocols, webservers, and Linux.  Newer platforms boast better available resources (e.g., 1GB vs 32MB of RAM), wireless connectivity via certified System-on-Chip (SoC) modules, and copious software such as FOSS, Python, and more. For good and for ill, this evolution has changed the way newer devices are developed and the features they offer.

What software runs on these systems?

In the embedded realm, software gets wrapped into all-encompassing packages dubbed firmware.  Firmware updates often pack a variety of constituents including OS and drivers, filesystems, apps libraries, interpreted scripts/languages, services (FTP, SSH, HTTP/HTTPS, MQTT, etc.), logging daemons, and network stacks.

As such, an embedded system is the same as any other computer. There are bound to be flaws in the software and in the system design itself. Defenders must protect embedded systems from unfettered access, update where possible, harden if capable, and monitor continuously.  The more abstract — or kludged together —software gets the more likely vulnerabilities within it become.

Why quick fixes for embedded systems are not enough

Imagine for a moment flawless code.  Picture the most technologically complex system operating without issue.  Conjure a single, silver-bullet solution that will save humankind from itself. Hard to imagine, right?

Thanks to the way devices are designed, engineered, developed, maintained, and sold, embedded systems, like any other enterprise computing product, will be flawed.  While there have been major improvements in code analysis, fundamental software design problems continue to slip through into production. Most programmers remain woefully inept at making good security decisions in the development stage and profit-motivated vendors have little appetite to address that shortcoming.

Among the embedded security challenges not addressed by any single solution:

• Patches and updates don’t cover third-party code. Keeping up to date with mainline code repositories is expensive and time-consuming, so most product vendors eschew it. They maintain their own code and apps when bugs arise, but leave fundamental flaws unfixed.
• Design issues make many fundamental problems immune to updating. Insecure-by-design choices spanning eras of product development and limitations on the ability of updates to affect device integrity conspire to weaken follow-up security efforts. For example, if an update mechanism on a vulnerable device allows insecure, trivial code uploads, then the updated firmware can’t ever be considered secure.
• Signed and encrypted updates won’t eliminate supply chain problems.  Code signing has value within a security design, but as we witnessed with SolarWinds, even well-defended development infrastructure can be compromised.  Practices such as code reviews may have caught the compromise, but it is likely the attack would have still gone unnoticed.
• If it’s insecure by default, it’s insecure. Almost universally, the software is left to run in its default configuration. If VSFTP or Zebra is in a product, it still has default credentials included. Weak, insecure defaults make things easy for the customer. How many of them go out of their way to harden a product when they deploy it?  Not many.
• Secure update strategies are not standard and may have flaws. Similar to cryptography, if the update process is not robust and well designed, then the device security may be compromised even if the firmware to be updated is from an authorized and authentic source.
• Multi-factor authentication isn’t a savior. If there are gaps in implementation or flaws in the libraries that enable MFA, its defensive value is greatly reduced. And if an attacker gains physical access to a device, all bets are off no matter the authentication controls.
• Public-key infrastructure (PKI) and cryptography are still risky. Flaws in implementation, improper use of the algorithm, leaked and re-used secrets, all add up to persistent risk.  One common trend is to transmit all network traffic through a TLS tunnel or to use HTTPS, which is good practice. But be aware that mitigations focused on access controls have their own challenges.  User credentials are constantly being targeted by attackers and weak passwords will remain a source of risk.
• Software Bills of Material (SBoMs) are of limited value to users.  Unless these types of lists of software components are validated for the accuracy or implemented into a capable asset management system, they won’t provide much real insight for users.  SBoMs remains a good idea, but for now, most users will be unable to fully leverage them.
• Logging and alerts are confusing and not always actionable.  Most embedded devices lack adequate glossaries to describe the frequency, format, and importance of log events and alerts.  Even if systems did have such supporting documentation, end-users would have to onboard them into their overall security program, which remains a rarity.

As systems developed with minimal rigor become over-provisioned, vulnerabilities flourish and risks proliferate. Today, an attacker rarely needs to bring their own hacking tools to an embedded device. They simply take advantage of the systems’ inherent flaws, live off the land, and persist in the environment as long as they please.

What risks do embedded systems pose to owners?

Risks for systems with network connectivity that use large amounts of software of varying quality are prevalent across ICS/OT, IoT, and even commodity products. As long as they are managed appropriately, however, asset owners can likely live with them. It’s a lot like driving a car – you can be careful, but accidents still happen, and much of the onus for risk reduction is on the driver.

Some risks to consider when it comes to embedded systems:

Most devices feature a hodgepodge of source code.  Inside that code are known vulnerabilities that frequently go unreported and ultimately require mediation and management by the asset owner.

Embedded systems can contain out-of-date software and unsupported components.  Compensating controls are needed to improve security with or without updates and patches.

Modern embedded devices are dangerously easy to deploy.  Contractors, third parties, even your own employees can install rogue devices that can communicate to outbound systems via wireless/RF technologies such as 5G or LTE.  Comprehensive asset management and physical site security program is a must.

Compromised cloud-based management destroys all other defenses. An attacker with unauthorized access to a centralized cloud management system has direct access to the plant floor at scale.  Management platforms should be highly secure, and devices isolated and monitored for anomalies.

If the device is on the internet or accessed even by a port forward, it is exposed.  It’s best to consider allowing only connections from specific IP addresses and limiting access if possible.

If a device is compromised, it may be used to affect other systems. A malicious intruder with sufficient time, resources, or skill can get a foothold on one vulnerable device and use it as a launchpad for more widespread — and devastating — attacks.

Devices still have physical access risks regardless of age.  Many devices, even new ones, lack sufficient physical tampering protections.  Access to devices should be limited to authorized individuals where possible.

Humans are still a risk. People will continue to misconfigure devices, make mistakes, use weak passwords, and more. In addition to security awareness training, periodic internal audits should be performed to see if devices are configured or deployed insecurely. And when it comes to embedded systems, employees should be trained to download software only from trusted sites and how to validate it against known good sources.

Getting embedded systems risk in check

If you’re now panicking at the scope of embedded systems insecurity, take heart.  Not all devices are easily exploitable or they are exploitable only under certain conditions largely affected by how you deploy and configure them. 

One key to addressing the challenge is to get ahead of the embedded security problem before it gets a foothold in the organization. Owners must insist on robust security during procurement, design of solutions, and throughout cybersecurity factory acceptance and site testing. This way, OEMs and vendors will learn they cannot continue unchallenged.  Trust, but always verify.

As a community, we should not let poorly secured products gain traction in the market. We must demand security as a necessary feature.  Software engineers and developers take note – even if you are a cog in the machine, we are all affected; especially when embedded devices become integral to the systems responsible for our lights, our water, our health, our daily lives.

Take a deeper dive into embedded security by downloading our on-demand webinar or reading the white paper on embedded systems.

In the meantime, if we force the market to deliver drastically improved solutions, we can avoid the security apocalypse plaguing ICS environments.  And before the naysayers state that all of these demands will make products more expensive, a well-designed, robust product is often no more expensive or time-consuming to develop. The price tag is always far less than the cost of getting breached.