Weathering the Deluge of OT Vulnerabilities: A Pragmatic Approach

Tip of the iceberg

If you’ve been looking at ICS CERT and following cybersecurity news lately, you’ve no doubt noticed a spike in scary-sounding news.  When infosec conferences such as Blackhat or various media firms get fixated on certain vulnerabilities, it often starts to feel like the world is one hack away from absolute chaos.

But is it really?

Beyond the debate over the CVSS scoring mechanism applied to products in industrial and critical infrastructure environments, here are the facts:

• The number of vulnerable devices and software in OT/ICS continues to grow.
• Asset owner vulnerability “swamps” are expanding, especially for embedded devices.
• IoT devices and associated network connectivity are raising security risk levels for most organizations.
• Compensating controls have become an increasingly necessary component for securing insecure architectures and by-designs.

Embedded systems in OT have many vulnerabilities

What else is missing?

Over the last few years, we’ve heard numerous discussions about various stack-related vulnerabilities in embedded systems and how there are hundreds of thousands —perhaps even millions — of devices affected. The truth is, however, how and how many devices are affected and the real impact of those flaws is a nuanced discussion.  Vulnerability and risk discussions require an adept touch in understanding their roots and intricacies; especially in OT/ICS devices. To wit:

• A vulnerability may exist in a codebase (e.g., within Linux kernel version X, but the module is not compiled in or configured for use).
• A vulnerability may exist in a package or component, but the way that it is used makes it non-exploitable assuming an intact system (e.g., libzip may not properly parse files, but the software merely creates and outputs a package with no chance of being exploited).
• A vulnerability may exist and affects a specific device, but because of a user configuration setting, it is rendered unexploitable.
• A vulnerability may exist and affects a specific device, but the device is protected and monitored by way of compensating controls.

If we examine any number of news articles or advisories for URG11, Schneider Electric Modicon UMAS vulnerability, GE UR relay vulnerabilities, or even our recent discovery for the Bachmann M1 series of PLCs, we can assume that all devices and software have flaws.

Using that assumption, we can begin to look at vulnerabilities differently, prioritize which classes or types we should focus on. We also need to recognize that vulnerabilities come in two categories: User-space application-related (E.g., RS Logix or project file); and Embedded-related (E.g., Modbus implementation or RTOS)

IoT is following the trend — but don’t panic if you are conscious of risk

Setting aside for the moment those products moving to cloud-connected devices with direct or required Internet access, and those increasingly encumbered by vendor lock-in, we find the majority of vulnerabilities reported on CISA/ICS CERT last year were related to applications hosted on endpoints. From this we can assume:

• Nearly any data file, database or project file leaks information or secrets and will continue to do so.
• Most such software can be virtualized and frequently restored or refreshed.
• Systems are likely networked but can also be isolated or restricted.
• Such systems typically reside on commodity host OSes that can be configured and secured with a bit of effort.

We can discuss the merits of OT application patching ad nauseum, but application-related vulnerabilities and risks are much easier to manage than those in embedded systems, even when their insecure, vulnerable-by-design protocols are run over the network a la Modbus, OPC Classic, or EIP/CIP.  As for vulnerabilities related to the secrets in files, we understand they “shouldn’t be there,” but we concede most asset owners have passwords written on SOPs, use weak passwords, or leave files unprotected against modification on fileshares and shared engineering workstations.

Should we push vendors to make better software? Yes. But we also need to do a better job of securing our environments, cleaning up PLC project artifacts and configurations, hardening endpoints, and more.  Both the MDT Autosave and Schneider Electric EcoStructure vulnerabilities demonstrate sloppy handiwork by the vendors, product risk/quality owners, and developing parties involved. We should expect this and always validate for security.

Unless the software or affected device is directly on the Internet or exposed to myriad threats, the sky won’t fall today due to an application flaw. The culprit is more likely to be a weakness in the underlying OS, VPN terminators, or edge firewalls or be the product of weak passwords or unmanaged remote access software like VNC/TeamViewer as we saw in the incidents at Colonial Pipeline and the Oldsmar (Fla.) water treatment facility.

All an attacker needs to do is get in, and leverage legitimate functionality — coupled with poor organizational security fundamentals — against the asset owner.

The more challenging vulnerabilities are those that live in the embedded devices that drive most physical processes in OT/ICS networks.  These devices are rarely patched for security, have numerous flaws by design, and are rarely configured to be more secure even if they have features to make them so.  Faced with this bleak reality, one might ask:

• When did society condone such shoddy workmanship, with programmers merely programming for the happy path?
• When did engineers stop planning to take worst-case scenarios into account, blindly relying on security, resiliency and safety while declining to engineer out the risk?
• When did owners stop working to ensure their assets remain sufficiently protected and supported?

We see it over and over again: Facilities capable of large, “forklift” upgrades and major implementations of new equipment, won’t spend a cent securing or improving infrastructure like the controls network or endpoints.  Security and maintenance should not cost more when developing products, and I should know. I helped bring several embedded products to market in the ICS space.  And for asset owners, this is the result of paper-mâché asset ownership without taking a pause to correct security rot or addressing security paradigm change results.

What can be said about vulnerabilities such as our recent discovery on authentication weaknesses in the Bachmann M1 controllers that affects every device in existence?  Well, the fix doesn’t fix much, and here’s why:

• Despite having the option to do so, essentially none of the affected devices out there are running in the secure mode necessary to prevent this vulnerability from being exploited.
• By default, the vendor/OEM programming software uses insecure modes to configure and access the devices; it’s unlikely anyone deploying these assets would ever use the security functionality unless explicitly instructed to do so.
• The hash algorithm used in the original was insecure. The newer algo, while improved cryptographically, does not use a salt and stores hashes in an easily accessed location in almost all modes – or by way of physical access/modification of the flashcard.
• By default, the password hash can be accessed via an FTP-accessible location without login credentials by default. The system also allows a malicious user to add their own credentials.
• If an attacker can access passwords, it’s likely they can also access the SSH or Telnet shells and change the context of running applications on the device via built-in VxWorks functionality without being discovered in most cases.
• Because these devices are deployed next to other devices they can degrade the security of the neighboring assets or affect their operation (e.g., in a windmill where other DCS controllers and devices communicate as one system).

The advisory is replete with CERT shenanigans and no small amount of legalese, vendor recommendations, workarounds, and suggestions for base practices that hint at any number of issues that might have been reported as separate CVEs themselves.  Sure, MD5 is broken and its replacement with SHA512 does not address the core issues. The vulnerability was an improperly secure secret/configuration store, and insecure default deployment options that both allowed further exploitation or was enabled by an overall insecure design.

The OEM snuck in another issue fix – upgrading of an insecure component to resolve the OpenSSL issues.  Unfortunately, the vendor recommended the use of TLS in the advisory yet they updated OpenSSL silently (unless you read the update notes on the vendor’s portal).  I would have suggested CISA post this as a CWE/CVE in the actual advisory or ensure that NVD would have a CPE update.  But choose your battles as they say, and I am grateful the vendor took steps to work with the CERT, setup vulnerability reporting processes and so on. That’s progress!

Conclusion

In summary, don’t necessarily be afraid of “whats” in the news regarding vulnerabilities. Some are scarier than others, but the majority need context.  To gain knowledge on this topic, read my Ultimate Guide to Reading ICS Cyber Security Advisories Like a Pro. Beyond that, as a community, we need to:

• Put pressure on vendors to create more secure products and have our systems integrators and service providers deploy systems in a secure, hardened manner by default.
• Incentivize developers and engineers to bring back pragmatic engineering that will handle anomalous conditions gracefully and to write better code.
• Improve vulnerability reporting and scoring systems.
• Continue to report on vulnerabilities but ensure the accuracy — and the realities — are clear to all reader types and not obscured with arcane language or overly broad fear-mongering claims.

If you are an asset owner, before you panic, use a platform like Verve to assess your risk continuously in order to get:

• Detailed asset management to know what you have and how it’s configured.
• Technology-based vulnerability and risk management to augment your resources through single pane reporting.
• Improved focus on actions that reduce risk within your environments versus chasing every APT-styled threat vulnerability claim.

Comments, questions, thoughts are always welcome.  Expect more information to be shared about what we observe on embedded system security, and what you can do about it.