The Verve Research Team

Verve’s mission is to help industrial clients ensure the security and reliability of their most critical assets: their industrial control systems. Verve Industrial brings over 25 years of ICS/OT experience or what is possible to bridge the IT OT challenges of securing these environments.

We were born out of an industrial services and engineering company (formerly known as Rkneal) that got its start in 1994 as an OT systems integrator over twenty-five years ago.  Today, the legacy lives on in the 1,000+ automation and control system projects we completed as we transitioned into the Verve you know today. Verve Industrial now serves industries across utilities (such as power, oil & gas, water), manufacturing, healthcare, chemicals, oil & gas, mining, and building controls.

We are a partner to our clients in their security and reliability journey. That journey is not solved with a single project or the one-time sale of a tool.  We walk alongside our clients in helping them increase the maturity of their systems and processes over time.

This summarization of disclosures aims to help our customers see through the fear-uncertainty-doubt (FUD) and balance cyber-risk orientated decisions.  Ron Brash, Verve’s Director of Cyber Security Insights (known for his S4 Detection ICS challenges and embedded development at Tofino Security) leads and fields wider discussions for asset owner benefit – evident for you herein we hope.

Introduction

2020 was a tough year all around. Let’s dive into the numbers of ICS vulnerabilities to get a state of the ICS risk today and what it may be tomorrow. 

2020 was a tumultuous year for most of us in cyber security and outside of it.  It started with a mysterious biological virus emerging globally and resulted in changing the way modern business is conducted, placing many of us in lockdowns/quarantines. It demonstrated and accelerated the increasing need for secure process automation and remote work.  The latter observation was already well underway for many organizations and individuals, but the crisis certainly raised its relevance.

ICS environments were not immune to the growing connectivity and, in fact, we saw organizations significantly accelerate their Industry 4.1/”connected plant” initiatives during the pandemic to manage remote staff.

To provide some context to the growing risk landscape for ICS, Verve analyzed the trends over the past several years with a deeper dive into 2019 and 2020. Although the trend is towards more and more advisories and CVEs, the reality is that this shows an increasing maturity of our industry. The publication of vulnerabilities enables better sharing of data and accelerated risk remediation.

To get behind the headline numbers of growth, risk, etc., Verve analyzed public data sources and reviewed our own vulnerability analysis data from the past couple of years to draw practical insights for operators and security personnel:

• Verve analyzed using various CVE datastores from four top IT vendors present in OT as an example of commodity (IT/OT convergent) vulnerability disclosures,
• Examined the 248 ICS-CERT advisories for 2020 and extracted the key insights,
• Contrasted all 2020 advisories against the 192 from the previous year (2019),
• Assessed the potential implications of those advisories,
• Developed a list of practical recommendations for ICS security staff,
• Put forth a perspective of what the future may hold over the next couple of years as the iceberg begins to take shape above the waterline,
• And hopefully saved you a lot of data analysis to give you an inkling of where 2020 struggled with some emerging problems and raise some questions of what the community can do better in 2021.

Vulnerabilities are not the whole security story but provide an initial window into risk on a broader basis.

Executive Summary

In 2020, ICS-CERT issued 248 cyber security advisories for public consumption on the CISA’s ICS-CERT portal.  Verve analyzed all these advisories, regardless of whether they came from large or small vendors to ensure accuracy even for geographies where the primary vendors are lesser-known.  We compared them to 2019 releases. This report summarizes the conclusions, implications on remediation strategies, as well as a perspective on what 2021 might hold.

------------------------------
ICS-CERT released 248 ICS-related advisories spanning 67 vendors/OEMs, 
710 CVEs containing references to different products, and a matrix of affected versions.
------------------------------

ICS-CERT advisories increased by ~30% over 2019 with the number of CVEs growing by almost 50%, and the average CVSS score of these CVE’s increased to over 8.0 out of 10.  These advisories were equally split between OEM application software and embedded device vulnerabilities.

------------------------------
Embedded devices are outnumbered with traditional IT Operational Systems and have had 
nearly identical advisory release ratios as software or applications 
for those OS devices in both years.
------------------------------

These vulnerabilities create risks as networks are increasingly connected. Almost two-thirds of them (63%) are exploitable remotely with little skill required. This is up from 56% in the 2019 total. And the total number of remotely exploitable/low skill vulnerabilities increased by 66% in 2020 vs. 2019. This should raise the hair on the back of the necks of OT operators everywhere, especially in a world of greater remote connectivity to critical infrastructure during COVID and in the future.

These are not just small, minor risks, either. The trend is towards higher risk scores with advisories scored as 8 through 10 increased at roughly twice the rate of those ranked 7 or less. So, not only are there more risks, but the severity is growing.  The largest type (~35%) relates to authentication or validation errors which can enable unapproved actors to make changes. And the second (30%) relate to Buffer Overflows. For the security researcher, this term is well-known, but to the operator the implications of this are enormous.  These types of vulnerabilities can be used to:

• Consume/exhaust a device’s or software’s available resources and require a restart.
• Create unsafe conditions where a device is unable to respond deterministically.
• Generate a HALT or a STOP condition due to unexpected communications or commands.
• Potentially compromise the system or application via remote code exploit (RCE).

As in 2019, Siemens constituted the largest number of advisories. In 2020, 31% of alerts were related to Siemens wherein 2019 the percentage was 23%. This is not to say that Siemens devices are “less secure”. In ICS, we need to learn that published vulnerabilities may, in fact, indicate a more mature vulnerability management program rather than more risky software. One telling fact is that almost three-fourths (73%) of vulnerabilities were reported by third parties. While it’s great that more researchers are diving into the complex world of ICS, this also means the adversaries are also diving into these devices and finding vulnerabilities that are not being disclosed. The bar for OEMs is quickly raising to begin more aggressive efforts to discover, report, and disclose vulnerabilities.

One of the most significant underlying trends is the increasing threat that emerges as we evaluate the software bills of materials (SBOM) on these ICS embedded devices. In 2020, 36 out of the 248 advisories were related to these supply chain risks. But we believe this dramatically understates the risks to these hidden vulnerabilities. Through our independent research, we regularly discover embedded devices leveraging insecure software stack elements. Our view is that this will be a major cause of increasing vulnerability disclosures in years to come.

In addition to these ICS-CERT advisories specifically focused on ICS devices and software, almost all OT environments contain a wide variety of “IT” equipment such as servers, workstations, networking gear, etc. While this report is not intended as a deep dive on these more traditional IT vulnerabilities, it is critical to note that the two live together and vulnerability teams need to integrate the two components into a single view of the risk of their OT environments.

The implications of this increase in vulnerability disclosures as well as the likely underlying hidden risk in the software bills of material are numerous and are cause for immediate action.  We often hear that patching is just not feasible in OT. The reality is true in some cases, but in many cases, there are feasible means to patch effectively to address the most severe risks. But short of updating all of the firmware on embedded devices, as long as an operator has deep visibility of these vulnerabilities, they can take a range of compensating controls to address the risks at least partially. We provide a robust list at the end of the document. But in short, apply robust patch management on the HMI/Server layer combined with improved configuration management and user/account management of the OS and embedded devices and strong network protection can provide a range of benefits that can deliver greater security until the upgrade cycle can deliver the updates necessary.

We hope this 2020 review serves to raise the focus and energy on protecting our most critical infrastructure.

Methodology and Data

After nearly a decade of in-the-field analysis, sometimes the best-laid approach is the simplest and most effective.  The approach for understanding the data in this approach was straight forward:

• We collected all the ICS CERT advisory results and CVEs.
• We analyzed the results and reviewed for discrepancies or gaps for the 2019 and 2020 periods. Reviewed the disclosures for indirect links based on industry knowledge.  This included:
  • Understanding if supply chain/software bill of materials was affected.
  • Nature of the disclosure based on provided data.
  • Examining causes noted in advisories.
  • Superficially verifying the accuracy of noted NVD entries.
• Focused exclusively on ICS vulnerabilities excluding the nine marine and medical device vulnerabilities.
• Reviewed IT vulnerabilities as they are present in almost all ICS environments and need to be included in any analysis of OT risk. This included evaluating Linux vulnerabilities as many ICS products are built on these vulnerable components.
• Distilled all findings and aggregated them together for final analysis.

For each ICS-CERT advisory, we analyzed them for severity, exploit vectors, link to product names and software versions, what the relevant risk entailed, etc. they were recorded, visited, and their information archived.  We checked to see if CVEs were missing/reserved, validated scores to determine if they were marked correctly and did the CPE strings reflect initial expectations (e.g., did the vendor name match, or was the product name correct?). The information was cross-referenced with reasonable scrutiny to discern common links & identified transgressions (e.g., X product is using Linux or Y license manager because it said so).

Analysis and Findings

Information Technology cyber security vulnerability disclosures in 2020

Like it or not, Information Technology (IT) is highly prominent in industrial environments, and it has been present there for decades. IT OT convergence is a “hot topic” today, but, in fact, it has been a reality for over 20 years. Then came the ubiquitous nature of devices, and networking and here we are. Windows, Linux (in devices), Webservers on devices using open-source components, and even IT networking is creeping into ICS environments at a tremendous rate; this is not saying ICS/IACS boundaries will go away, but the delineation is rather blurred and jagged.

Starting with the big names in the room, it appears that vendor product security teams had a busy year in 2020, but not particularly more so when compared to years past:

• 1257~ Microsoft vulnerabilities (in various components)
• 160~ Linux kernel vulnerabilities (these are key as they form the foundations of many ICS product components)
• 190~ Cisco vulnerabilities (key as many OT environments rely on IT networking equipment or OEM devices that have rebranded Cisco gear)
• 0 WindRiver VxWorks vulnerabilities (this is surprising given the # of devices/vendors using this OS & the fact past disclosures often spur further research)

We focused on these names because they form key elements of the OT environment. In no way is this a comprehensive review of all IT-related vulnerabilities present in OT environments. But as you can see, these risks overwhelm the number of ICS advisories and need to be addressed within the OT vulnerability management program.

------------------------------ 
2020 vulnerability disclosures are roughly in line with expectations - an increase 
from Microsoft, a slight decline of CVEs in Cisco and Linux totals 
(albeit some classification issues)
------------------------------ 

The number of Microsoft vulnerabilities increased significantly in 2020 by almost 100%. And as can be seen in the below chart, the number of disclosed vulnerabilities has increased dramatically since 2014. This has come with leveraging more open-source software as well as with a greater push by Microsoft.

An additional review for more noticeable vulnerabilities for Microsoft specifically highlights the fact that that many of these Microsoft vulnerabilities do not involve the version of the OS, itself. They are components of the overall Microsoft software stack. The reason this is important is that to identify the presence of these vulnerabilities and to remediating them appropriately, any evaluation of those assets must include deep software assessment, rather than just identifying the OS version which might be available in the header of a packet.  True vulnerability assessment requires a 100% software view on each asset.  Too often we see organizations rely on simply the OS to determine patch status. But in fact, this is often misleading.

Linux vulnerabilities declined in 2020, but this should not provide a false sense of security.

• The Linux and/or Opensource ecosystem is very inconsistent and does not include per distros, custom compiled applications, and the software that may be used.
• Microsoft includes several of their other products vs. Linux, which is just the mainline Kernel (not specific distributions or software running on them); Microsoft’s is more accurate, but also scrambled across components, versions of Windows, etc.

Beyond the above observations, a further point to add is that a variety of issues (bugs) are resolved in either OS, but the Linux ecosystem seems to tend to solve issues quicker and go without disclosure reporting.

Cisco is another interesting vendor where they too had a drastic upshift earlier in the decade, but 2020, resulted in a decline.  There may be multiple reasons for the increases between 2012 and 2018 due to:

• The vendor’s expansion into userland applications/software security solutions or acquiring of other products/vendors.
• Investments in product security, and now they are “coasting”, but during 2020 there was a decrease over previous years.

Analysis of ICS vulnerabilities based on CISA advisories

Managing ICS vulnerabilities is a tough business for operators and their cyber security peers. Lack of robust hardware, software, and firmware inventory, challenges with risky IT vulnerability scanning, lack of vendors’ contributing to public vulnerability databases, lack of knowledgeable ICS resources all make this a challenging process. The future is frightening as well. As vulnerabilities grow and attackers discover more weaknesses the burden on OT staff and their cybersecurity peers will grow significantly. Without robust, ICS-specific endpoint security management tools the burden will grow.

This is becoming a huge risk and dramatic burden on already overtasked OT personnel in the years to come. The growth rate in ICS vulnerabilities is accelerating rapidly with a growth of 65% over 2019.  Furthermore, the number of devices impacted is growing rapidly and CVSS scores are increasing too.  On average, 62% of embedded vulnerabilities affect multiple products, and 99% affected multiple versions.  This is on top of the ~50% increase in 2019 over 2018. We are getting to a point where these vulnerabilities cannot be ignored – or managed manually through OT teams.

Importantly, these below numbers do not account for the large number of vendors that normally do not make public disclosures but instead provide vulnerability information privately to their customers. If these were to be included the total numbers may double. They are not even present in one database which can be leveraged by vulnerability teams without significant manual processes. This growth of risk combined with a lack of automation is a recipe for security disaster.

From the 257~ original ICS-CERT advisories, marine, medical, and a specific tractor-trailer advisory were excluded, and 248 advisories remained.  Of the 248 advisories, they had a collective average score via the tagged CVSS of 8.0 (HIGH), and there were more than a single vulnerability assigned for each advisory.

In addition to the above summary statistics:

• 122 were for “software” (49%), 117 for “embedded” products (47%), seven were for both “software and embedded” (3%), and two accounted for “other” (~0%).
• 63% were exploitable remotely with low skill, compared to only 56% in 2019.
• 145 affected multiple products (58%), and 246 multiple versions (99%).
• 36 affected the supply chain in some way (14.5%).

Discovery & Reporting

• 52 were self/company reported (21%), 183 by researchers or organizations external to the company (74%), seven by the government (3%), and four were unmentioned/unknown (1.6%).
• Siemens had two instances where a Chinese organization reported vulnerabilities (one was a government agency, and the other was a researcher group). This might not be terrifying, but it is interesting that a country with open challenges with another nation-state may be SIGNALLING its expertise or capabilities through ICS-CERT (CTI people may have missed this)
• Roughly 30% had issues when comparing advisory details. There are mismatches or errors with the names of vendors or subsidiaries, products being relabeled, and data refinement was required in some cases.

CVSS Rating Challenges

• ICSA-20-042-13 had a CVSS score of 2.4 but suggested 6.x and 4.x as the actual CVE scores assigned to it. This is a huge discrepancy, especially when the CWE is dangerous malicious file upload… on a common poorly secured TCP/IP serial gateway.
• Advisories relating to cryptography and improper handling of secrets need assistance in normalizing. How is one a 10, and another 4 or a 7?  Of course, context is needed on the usage and exposure.

Supply Chain/ SBOM Risks

• 36 out of 248 advisories were related to supply chain components. This is an underlying threat to legacy devices with hidden vulnerabilities. The reality is that these underlying components – vulnerabilities relating to CodeMeter, or InstallShield – are present in many devices and the impact cannot even begin to be measured in the number of downloads, asset owners affected, or any useful metric.
• Projects hosted on Github or FOSS repositories have issues in reporting (if they are disclosed at all). Plus, who is the vendor?  It is source code and can be changed with a keystroke, and 99% of the time, the CPE is never noted properly to affect products – even though they are seen often.  e.g., SharpLib, Proftpd, SNMPD, Busybox, Linux, etc.

Vendor Disclosures

• Advantech notably had 8 disclosures, all were reported by researchers, average CVSS score 9.2/CRITICAL, and 32 CVEs of the total 710. It is used for network management, and the flaws reported are quite comprehensive.
• Siemens scored the most CVEs (54) attached to an advisory (ICSA-19-351-02) out of all of them.

It is also important to note that a higher disclosure rate by vendors might be a sign of security program/development lifecycle improvement, and not the opposite – more reporting means more active & investment in security or a change in internal culture, which hopefully will result in improved products overall.  It may also signal that the vendors not reporting issues, are not practicing sufficient security, and should be forced to do so by the asset owners (who have a decisive voice).

In addition, from a data perspective this chart has several caveats that a reader needs to be aware of:

• Many vendors are not reporting vulnerabilities or “sharing” 1:1 with affected customers in “sensitive” situations.
• Researchers have not had access to systems that are not commonplace, or yet to be retired, and found their way onto eBay.
• Advisories may still be under embargo despite being reported within the 2020 calendar year.
• Many products are End-of-Life (EOL), are without a patch, insecure by design (also known as forever days), use insecure defaults, and have a blanket legalese statement discussing their “remediation” (if you can call it that).

Of course, patching all vulnerabilities is not a panacea, nor is the reporting of issues, but many ICS assets have updates that are not cited as security-related and improve the stability of various products due to the elimination of “errata” or “anomalies”; which can be useful for ensuring SRP over the asset’s lifetime.

Top Vectors & Issues

Each of the ICS-CERT advisories had difficulty/vector/exploit and problem/issue fields.  Low skill level to exploit, exploitable remotely, and stack-related programming issues are still prevalent.  We saw an increase in the number of low skill/remotely exploitable vulnerabilities. Over 80% of all vulnerabilities have a low skill level required to exploit.

If an attacker can get access, most vulnerabilities are exploitable with the relatively low skill required.

Of the 248 vulnerabilities 86 unique vulnerability/issue values were found after sorting and counting.  The top 20 issues and their frequency are as follows. In summary, they include:

• Authentication/Validation: 35%
• Buffer Overflow: 30%
• Sensitive Information Risk: 25%
• Path Traversal/Cross-site scripting: 15%
• Improper Restriction of Ops within Bounds of Memory Buffer: 10%
• SQL Injection: 7%

The above vulnerability causes highlight the significant risks to operations that these vulnerabilities present. These are not “minor” issues. In critical infrastructure, these processes must operate at high reliability and unapproved changes can cause significant damage or downtime.

Over one-third of alerts involved the ability to get around authentication and validation requirements offering the potential for attackers to make changes to critical processes. Another 30% involved the possibility for buffer overflow where the impact could be to overflow devices with information thereby causing operational disruption. So two-thirds of all vulnerabilities involved those that could cause significant damage to the environments if unfettered access by a malicious actor was able to sufficiently breach security controls.

It is important to note:

• Most of those can be found via tools, and part of a rigorous SDLC/automated testing program.
• They can be removed via proper pointer arithmetic and solid input sanitization.
• Removal of web/HTTP would help (really).
• SQL injection should be a thing of the past.
• Proper storage of credentials or usage of cryptography would alleviate many woes.

In the 2019 causes, there may be lesser occurrence counts due to overall numbers of CVEs (~300 less), and advisories (~50 less).  However, there are similarities to the 2019 advisories – programming errors, stack overflows, lack of sanitization, and improper use of security features/cryptographic techniques.

Supply Chain Cyber Security Gaps will Result in a Paradigm Shift

One of the most critical, but potentially overlooked areas of advisories are those related to supply chain. Beyond the newly appointed U.S. DoD CMMC framework for designated industries, NERC CIP-13 or BOMs for FCC certification by top tier vendors, there are a lot of misunderstandings and a lack of understanding of the security of packaged software products (including their embedded variants).

Regardless of not knowing what X organization may possess or even at what configuration it may be running in, there were several high-profile supply chain-related announcements in 2020 (e.g., Ripple20, Treck/IP, URG11, Amnesia:33, etc.).  However, when you review the advisories, you find almost none of them are related to these “hot topic’ vulnerabilities. Many vendors argued that these were overstated for ICS devices. Some released private advisories on their websites for customers regarding potential devices. But finding these vulnerabilities within the ICS inventory of an organization is often very difficult without specialized tools.

In both 2019 and 2020, roughly the same amount of supply chain/component related advisories were issued. There are several issues with the reporting system, but the decrease in 2020, and the increase of products on the market hints at a hidden issue lurking below the surface; this is not limited to embedded systems, but software that has statically compiled dependencies, uses third-party libraries (FOSS or purchased/COTS), and more.

For example, many ICS devices (particularly those that provide networking) are often running very “old” components and are lacking (at a minimum), vulnerability disclosures citing CWE-1104: Use of Unmaintained Third-Party Components or CWE-398: Indicator of Poor Code Quality. This would indicate that new strategies are needed for managing components used within products, and the growing problem when OEMs are developing products using open source (FOSS) & are unable to handle the component’s astronomical update frequencies (that never are reflected into their actual product upgrade cycles).

Case Study of Embedded Vulnerabilities

To illustrate the challenge when reporting on components is illustrated below.  This device is not by any means a cherry-picked lone example, thousands, potentially hundreds of thousands of devices are in a similar state with few vulnerabilities associated with them, but they have many components that likely are never upgraded, or have other issues.

The device above demonstrates other undocumented risks.  It is a product still deployed at countless sites:

• An industrial radio’s Telnet session
• Leaking a Linux kernel with hundreds of vulnerabilities
• A Busybox shell with countless “features”
• Demonstrated insecure by design functions

------------------------------
Countless device firmware dependencies are not tagged as an advisory on ICS-CERT, 
nor is the vendor (or many other vendors for that matter providing core OS updates).
------------------------------

To provide some context, the below chart is the number of Linux vulnerabilities in 2020. These components are present in thousands of pieces of software. Many ICS products would contain some of these components. But finding them requires a deep software bill of materials.

Many ICS devices have a very low product refresh rate, patching is difficult, products are abandoned by the vendor (e.g., end of life), teams have retired or shifted to other products, companies are acquired or sold, and embedded updates and changes can be complicated due to the proximity to the OT processes.

What then are the consequences or the results of a growing supply chain cyber security and interest area?  It might just become a paradigm shift, and it might require some adjustment for security teams & product developers.  Instinctively, with the embedded stack vulnerabilities released in 2019/2020 for URG11, AMNESISA:33, and Treck/IP – it would have been assumed there would have been a very large portion of affected devices & numerous advisories, but this was not the case.

------------------------------
"Branded" embedded stack vulnerabilities resulted in few advisories on ICS-CERT, and
the claims of millions of devices affected is questionable. (E.g., if 33 CVEs
are in the "vanilla" stack, why do only some apply? Are there differences in 
scores and applicability?)
------------------------------

With relation to supply chain issues: Verve predicts that over 2021 that many asset owners and researchers will begin to assess more device firmware for inherently insecure components and the results might be as follows:

• Explosion of ICS-related CVEs at the device level, which will require an increased emphasis on detailed inventorying, monitoring, and segmentation.
• Supply chain consequences for vendors, integrators, and asset owners will become a large focus for any security-focused organization moving forward (especially under NERC CIP), but also will demonstrate how insecure assets likely are under the hood.
• Increased need to interact with, manage configurations, and patch/update Internet of Things (IoT) devices.
• Decoupling of vendor applications from the core/base operating system by way of fewer proprietary operating systems or components will continue to increase.
• ICS/device/application security needs to improve – most vulnerabilities are due to poor software engineering, and many of the obvious or easily exploitable vulnerabilities would have been found during a thorough Cyber Security Factory Acceptance Test and/ Site Test (CFAT vs. CSAT)
• Many devices ship with poor security settings and integrators/asset owners deploy them in a state where improved security features are not used even though they are present or with default credentials.
• New legislation regarding software security, integrity, and supply chain management required for critical infrastructure or even IoT for consumers.
• Organizations that actively disclose AND remediate vulnerabilities (vs. posting a disclosure without a concrete solution) may be more advanced in their security practices than those who do not.
• Alternatively, organizations that take a proactive stance on disclosing security issues (even without issues) may also be more advanced than those that have few security postings overall (often due to business/legal decisions for right or wrong).

Remediation

The above summary should worry all of us involved in ICS (or OT) security. The number of published vulnerabilities is increasing and they are becoming more severe. In a way, this is good news. Vendors are being more transparent; researchers are producing more insights, and our joint institutions are sharing like never. But the news is a wake-up call to everyone trying to operate the critical control systems that operate so much of our economy. We need to do better to do identify the vulnerabilities in our environments and eventually remediating those risks.

------------------------------
Operators face a set of challenges in dealing with the
data provided by ICS-CERT and other vendors.
------------------------------

First, they need to inventory all of their assets including 100% of the software on those devices as well as the firmware present on the embedded devices. This needs to be combined with ICS-CERT/CVE data as well as vendor-specific releases that may not be done publicly.

• Continually maintain an accurate inventory of deployed assets and consider embedded systems as part of your vulnerability, risk, and OTSM programs. These systems may have different operational constraints, however, several techniques are available to safely identify, inventory, and manage non-commodity devices.  Verve recommends including them as part of a holistic solution that raises the visibility of these devices such that they are illuminated for actionable risk management reduction & remediation.
• Track vendor security portals through automation & mailing list subscriptions. Of course, it should be obvious, and it might take a little bit of investment, but security updates in OT are far less frequent than those in IT, and this should be less strenuous than you might imagine, especially if partners offer services to support risk and vulnerability management.
• Map vulnerabilities from the various databases to the inventoried assets to prioritize remediation actions based on risk and device criticality.

Second, develop a comprehensive risk assessment of each asset that includes the vulnerability as well as the compensating controls that are currently enabled on the asset.

• Asset, vulnerability, configuration, and risk management are intrinsically linked. They require accurate and timely information, and so all applicable assets should be included not just for posterity, but for others to be informed of.  This has a second benefit by helping in other secondary tasks but can aid in recovery or active threat prevention and applies to software and embedded/infrastructure assets as well.
• Understand user/access control on each device. Vulnerabilities are only one element of risk in ICS. One of the most frequent risks we see in our assessments is insecure user/account management. In many cases, dormant accounts, admin users, etc. exist on key HMIs, domain controllers, etc. Cleaning up these risks is key, regardless of vulnerability management.

Third, develop remediation strategies relevant for each asset given its criticality, risk score, and feasibility of patching/updating.

• Patch or update affected devices or software that has an update available WHEN/HOWEVER that is applicable, and/or re-configure systems/devices/applications such that they do not use vulnerable protocols or features where feasible. This can potentially lower the risk exposure of a device, improve device stability, prevent accidental encounters (e.g., situations that are discovered by accident without a malicious entity) with these vulnerabilities, and deny a network-based attack vector from potentially affecting your environment. This is at the discretion of the asset owner, and a mature risk/vulnerability management process is required to assess whether a firmware update is applicable or required (if it even exists at all)
• Use security features that are native to an Operating System (OS) to limit honest users from performing risky activities and prevent common malware from being executed.
• Protect systems, network infrastructure, endpoints, and points of ingress that are commonly compromised, have a higher risk exposure and possess a statistical likelihood of being targeted. A majority (likely most attacks) originate from commodity IT systems or have a significant effect on the success of an attacker, and as such, should employ a variety of recommended cyber security controls and best practices.  This includes protection against commodity malware threats, security updates & patches, application whitelisting, secure configuration/hardening, firewalls, monitoring, and other technologies. Embedded devices often provide configurable options (whether in their OS, or by additional logic) that can improve security, reduce accidental exposures, report logs/device health, and have their passwords changed from defaults.
• Ensure default passwords are changed to something of reasonable hardiness and ensure insecure legacy interfaces are disabled at deployment IF secure alternatives exist. The chances of an organization migrating to a more secure “modern” alternative is very low in a running environment, but also CVEs or alerts are less likely to ever be released for a known security issue IF a secure option is documented/provided; therefore, vulnerabilities will NEVER/RARELY ever be noted by single focus vulnerability management solutions that rely solely on scanning.
• Deploy protective mechanisms on the endpoint or at the network to provide security until patching can occur. Application whitelisting is a very practical solution for OS-based devices in OT environments, especially where patching is not immediately feasible. Prevent direct access to vulnerable devices by using network access controls, segmentation, and limiting network communications to only authorized systems and/or specific communication protocols; in particular, isolate them from business networks & functions.

Fourth, monitor networks and endpoints for potential behaviors that may indicate exploitation of known vulnerabilities.

• Sanitize network protocols for irregularities or non-standard usage by a capable network security device may also reduce the risk or reduce unintended behavior by a device using a stack such as this one.
• Monitor ARP tables, DHCP requests, DNS lookups, IP connections, and other network-related infrastructure for conditions that would enable an organization to identify malicious activity or an anomaly. All logs should be forwarded, monitored, and investigated as part of a continuous process to manage alarms in a way to detect events, and differentiate those from accidental, steady-state, and malicious activity.
• Use an OT SIEM to receive asset and application logs where possible and monitor for deviances or fluctuations in the environment. Most environments are steady-state (rarely change or are very predictable) and so identifying anomalies should be infrequent if alarming is sufficiently tuned (just as you would on an HMI or SIS).
• Perform regular polling of changes on a system for unauthorized software/firmware revisions, configuration changes, or process logic. This is even more true when devices have functionality that allows for localized clearing of logs or when connections are not always online to “upload” the latest copy of logic to a managing station.

Fifth, ensure appropriate governance and procedures to both reduce risk and ensure a rapid incident response.

• Adequate governance, policy, and procedures to sufficiently handle risk, vulnerabilities, and incidents. It is one thing to receive an alarm, but another to be able to act on it in a procedural and consistent manner.
• Regularly perform “fire drills” for cyber security in ICS environments. This includes performing forensics on systems where possible, but also the end-to-end testing of a process designed for disaster or event handling, reporting, and recovery.

Next Steps: Should There be Panic or Alarm?

2020 was another year of increasing alarm bells for ICS security. As stated above, additional vulnerabilities are not “bad” per se. They likely indicate a maturing industry. The growth rates, however, should be a wake-up call that the time to act is now. Greater connectivity is coming – not to mention all the current remote access that already exists as evidenced in the recent attack on the water facility in Florida.  As this connectivity increases, the access to these vulnerabilities increases. No longer is it enough just to argue that “air gaps” protect these endpoints.

OT security management (OTSM) is a necessary requirement of all ICS environments. The increase in vulnerabilities highlights the tip of an iceberg, and that iceberg is heading our way. The current approaches to OT security relying on perimeter network defense and monitoring, will not be enough. CISOs, boards, regulators, and other stakeholders will require that ICS apply the same endpoint management and security as traditional IT environments. As this happens, the burdens on already overtaxed OT teams will explode.

The time is now to begin to get ahead of this and develop a plan to identify and manage these vulnerabilities – and those to come – in a more holistic way than today’s primarily manual efforts.

There is much good news as well. The fact that more researchers and companies are sharing vulnerabilities means maturity is increasing. Although one might argue whether some have been over-hyped, the more “expansive” disclosures such as Amnesia 33 or Urgent11 have caused an increase in discussions about the risks of these embedded OT devices.  We also see an uptick in overall recognition of the need for some visibility into OT.

However, the challenge will continue to grow over the coming years. We hope the industry meets the challenge with an increase in attention and focus at the same rate as the growth in known vulnerabilities.