Respond to Growing OT Vulnerabilities with Endpoint Systems Management

A recent cybersecurity advisory alert details the top 15 Common Vulnerabilities and Exposures (CVEs) routinely exploited by malicious cyber actors in 2021, as well as other CVEs frequently exploited. In addition, there have been multiple other releases about topics such as the Industroyer2 threat, emerging risks due to the Russian-Ukraine conflict, the newly discovered malware, Incontroller, which is an advanced type of malware targeted specifically at ICS devices.

The increase in threats to OT environments has pushed the National Security Agency (NSA) and Cybersecurity and Infrastructure Security Agency (CISA) to issue warnings about cyber-actors’ willingness to conduct malicious cyber-activity against critical infrastructure by exploiting internet-accessible OT assets.

U.S., Australian, Canadian, New Zealand, and UK cybersecurity authorities assess, in 2021, malicious cyber actors aggressively targeted newly disclosed critical software vulnerabilities against broad target sets, including public and private sector organizations worldwide. To a lesser extent, malicious cyber actors continued to exploit publicly known, dated software vulnerabilities across a broad spectrum of targets.

While there has been a string of alerts and buzz around naming the various threats, the recommendations remain the same: Manage your OT systems through core security fundamentals.  The recommendations all play the same fundamental tune around CISA’s ICS Best Practices.

They center around areas such as:

• Maintain an ICS asset inventory of all hardware and software
• Update software using a risk-based assessment approach to determine which assets should participate in the patch management program
• Implement allow/white listing on HMIs and workstations
• Isolate ICS/SCADA systems from corporate and internet networks using perimeter controls
• Disable unused ports and services on devices
• Enforce multi-factor authentication for remote access
• Change all passwords on regular basis and monitor password status
• Maintain known-good backups
• Protect systems with strong anti-virus and other endpoint detection capabilities
• Implement log collection and retection
• Leverage OT monitoring solutions to alert on malicious behaviors

In summary, these are all about what we call “OT Systems Management”. This phrase encompasses these fundamental elements of OT security – from asset inventory, to endpoint management of vulnerabilities, patches, configurations, etc., to managed network segmentation as well as controlled access, and eventually monitoring and recovery.

While these alerts are absolutely valuable in raising awareness of the community, they can cause confusion if not read thoroughly and understood for the recommendations being made. We often receive calls about the latest alert because an organization is chasing down a specific threat or particular threat actor or malware that has recently been seen in an ICS system somewhere.  To be clear, it is critical that we as an industry know about these emerging threats, and CISA has contributed greatly to the awareness of these threats.

However, it is key that organizations read down to the bottom of these releases to that section that includes recommended mitigations or actions.  That section is what really matters – what can you do as an organization to address these threats.  And in that, the message is consistent – OT systems management. The consistent application of fundamental security controls.

While it would be great if every OT operator had advanced beyond these fundamentals, the truth is that most are still working on these core elements. It is there where we believe we must focus as an industry.

As one example, many industrial organizations do not actively manage their OT endpoints. In many cases, they do not have accurate inventories of what those endpoints are. If the inventory exists, there often is a lack of actively managing those devices whether that be patching, hardening configurations, updating passwords, updating firmware, etc.

Some may have OEM vendors that come on some basis to apply OS patches and application patches for that particular OEM application set. But all too often, we who come in after that patching process, find in looking at the output of the Verve Endpoint Management platform that those patches left many critical vulnerabilities either because those patches did not include other application software on that device or OS patches that were not approved but solve critical vulnerabilities.

In our 2020 ICS Advisory Report, we found a 47% increase in CVE’s in ICS-CERT advisories between 2019 and 2020. In our 2021 summary report about to be released, we found another 59% increase in the number of ICS vulnerabilities, but most organizations do not have a comprehensive, vendor-agnostic patch management program.

OT systems management includes the development of an ICS-specific patch management effort.

We are not naïve, however. We have been designing and securing ICS systems for almost 30 years. Patching OT is hard.

The key challenges with patch management:

• Tracking what patches are relevant for a specific device
• Knowing if the patch is approved by the vendor – as well as end-of-life software or systems no longer supported by the vendor
• Challenges of required re-boots to apply patches in ongoing process environments
• Operational risks from patches that may disrupt operations if not tested appropriately
• Devices requiring firmware updates that may have knock-on effects on other parts of the system require an overall system upgrade to accomplish, etc.
• Lack of staff/resources to manage the process

It’s no surprise organizations are always a step (or two or three) behind on software patching and spend valuable time in the weeds manually tracking and managing the patching program.

Patching is only one part of this overall OT systems management effort. See the whitepaper on patch management here.

But just that one example demonstrates the challenges in achieving those fundamental mitigations that CISA lays out.

OT Systems Management requires the “operationalization” of security. The great news for ICS practitioners is that if we can “operationalize” security, operators understand how to execute. Controls engineers and production personnel live every day by improving the operations of their plants. They have metrics, targets, specific quality improvement plans, 6-sigma or other lean principles, balanced scorecards, etc. The world’s base of industrial operators continually improves productivity each and every year through operations improvement programs.

If instead of considering cyber security as a place only for people with advanced cyber expertise, and started to operationalize it into a series of fundamental tasks which can improve each and every day, we can start to apply lean and other principles to improve performance.

But if we let the latest headline and new threat name distract us from those fundamentals at the bottom of those CISA alerts, we can lose the thread of the overall mission.

Each of these alerts should be seen not as “new news”, but as a reminder of what CISA and others have been saying for years – execute on fundamental OT security practices. If we do that we will be addressing old and new security risks at the same time.