The State of OT Cyber Security
Verve Industrial Protection assessed the history of OT cyber attacks and determined where they are likely headed based on twenty-five years of industrial cyber security experience in ICS environments.Learn More
Subscribe to stay in the loop with the latest OT cyber security best practices.
Verve’s mission is to help industrial clients ensure the security and reliability of their most critical assets: their industrial control systems. Verve Industrial brings over 25 years of ICS/OT experience or what is possible to bridge the IT OT challenges of securing these environments.
We were born out of an industrial services and engineering company (formerly known as Rkneal) that got its start in 1994 as an OT systems integrator over twenty-five years ago. Today, the legacy lives on in the 1,000+ automation and control system projects we completed as we transitioned into the Verve you know today. Verve Industrial now serves industries across utilities (such as power, oil & gas, water), manufacturing, healthcare, chemicals, oil & gas, mining, and building controls.
We are a partner to our clients in their security and reliability journey. That journey is not solved with a single project or the one-time sale of a tool. We walk alongside our clients in helping them increase the maturity of their systems and processes over time.
This summarization of disclosures aims to help our customers see through the fear-uncertainty-doubt (FUD) and balance cyber-risk orientated decisions. Ron Brash, Verve’s Director of Cyber Security Insights (known for his S4 Detection ICS challenges and embedded development at Tofino Security) leads and fields wider discussions for asset owner benefit – evident for you herein we hope.
2020 was a tough year all around. Let’s dive into the numbers of ICS vulnerabilities to get a state of the ICS risk today and what it may be tomorrow.
2020 was a tumultuous year for most of us in cyber security and outside of it. It started with a mysterious biological virus emerging globally and resulted in changing the way modern business is conducted, placing many of us in lockdowns/quarantines. It demonstrated and accelerated the increasing need for secure process automation and remote work. The latter observation was already well underway for many organizations and individuals, but the crisis certainly raised its relevance.
ICS environments were not immune to the growing connectivity and, in fact, we saw organizations significantly accelerate their Industry 4.1/”connected plant” initiatives during the pandemic to manage remote staff.
To provide some context to the growing risk landscape for ICS, Verve analyzed the trends over the past several years with a deeper dive into 2019 and 2020. Although the trend is towards more and more advisories and CVEs, the reality is that this shows an increasing maturity of our industry. The publication of vulnerabilities enables better sharing of data and accelerated risk remediation.
To get behind the headline numbers of growth, risk, etc., Verve analyzed public data sources and reviewed our own vulnerability analysis data from the past couple of years to draw practical insights for operators and security personnel:
Vulnerabilities are not the whole security story but provide an initial window into risk on a broader basis.
In 2020, ICS-CERT issued 248 cyber security advisories for public consumption on the CISA’s ICS-CERT portal. Verve analyzed all these advisories, regardless of whether they came from large or small vendors to ensure accuracy even for geographies where the primary vendors are lesser-known. We compared them to 2019 releases. This report summarizes the conclusions, implications on remediation strategies, as well as a perspective on what 2021 might hold.
------------------------------ ICS-CERT released 248 ICS-related advisories spanning 67 vendors/OEMs, 710 CVEs containing references to different products, and a matrix of affected versions. ------------------------------
ICS-CERT advisories increased by ~30% over 2019 with the number of CVEs growing by almost 50%, and the average CVSS score of these CVE’s increased to over 8.0 out of 10. These advisories were equally split between OEM application software and embedded device vulnerabilities.
------------------------------ Embedded devices are outnumbered with traditional IT Operational Systems and have had nearly identical advisory release ratios as software or applications for those OS devices in both years. ------------------------------
These vulnerabilities create risks as networks are increasingly connected. Almost two-thirds of them (63%) are exploitable remotely with little skill required. This is up from 56% in the 2019 total. And the total number of remotely exploitable/low skill vulnerabilities increased by 66% in 2020 vs. 2019. This should raise the hair on the back of the necks of OT operators everywhere, especially in a world of greater remote connectivity to critical infrastructure during COVID and in the future.
These are not just small, minor risks, either. The trend is towards higher risk scores with advisories scored as 8 through 10 increased at roughly twice the rate of those ranked 7 or less. So, not only are there more risks, but the severity is growing. The largest type (~35%) relates to authentication or validation errors which can enable unapproved actors to make changes. And the second (30%) relate to Buffer Overflows. For the security researcher, this term is well-known, but to the operator the implications of this are enormous. These types of vulnerabilities can be used to:
As in 2019, Siemens constituted the largest number of advisories. In 2020, 31% of alerts were related to Siemens wherein 2019 the percentage was 23%. This is not to say that Siemens devices are “less secure”. In ICS, we need to learn that published vulnerabilities may, in fact, indicate a more mature vulnerability management program rather than more risky software. One telling fact is that almost three-fourths (73%) of vulnerabilities were reported by third parties. While it’s great that more researchers are diving into the complex world of ICS, this also means the adversaries are also diving into these devices and finding vulnerabilities that are not being disclosed. The bar for OEMs is quickly raising to begin more aggressive efforts to discover, report, and disclose vulnerabilities.
One of the most significant underlying trends is the increasing threat that emerges as we evaluate the software bills of materials (SBOM) on these ICS embedded devices. In 2020, 36 out of the 248 advisories were related to these supply chain risks. But we believe this dramatically understates the risks to these hidden vulnerabilities. Through our independent research, we regularly discover embedded devices leveraging insecure software stack elements. Our view is that this will be a major cause of increasing vulnerability disclosures in years to come.
In addition to these ICS-CERT advisories specifically focused on ICS devices and software, almost all OT environments contain a wide variety of “IT” equipment such as servers, workstations, networking gear, etc. While this report is not intended as a deep dive on these more traditional IT vulnerabilities, it is critical to note that the two live together and vulnerability teams need to integrate the two components into a single view of the risk of their OT environments.
The implications of this increase in vulnerability disclosures as well as the likely underlying hidden risk in the software bills of material are numerous and are cause for immediate action. We often hear that patching is just not feasible in OT. The reality is true in some cases, but in many cases, there are feasible means to patch effectively to address the most severe risks. But short of updating all of the firmware on embedded devices, as long as an operator has deep visibility of these vulnerabilities, they can take a range of compensating controls to address the risks at least partially. We provide a robust list at the end of the document. But in short, apply robust patch management on the HMI/Server layer combined with improved configuration management and user/account management of the OS and embedded devices and strong network protection can provide a range of benefits that can deliver greater security until the upgrade cycle can deliver the updates necessary.
We hope this 2020 review serves to raise the focus and energy on protecting our most critical infrastructure.
After nearly a decade of in-the-field analysis, sometimes the best-laid approach is the simplest and most effective. The approach for understanding the data in this approach was straight forward:
For each ICS-CERT advisory, we analyzed them for severity, exploit vectors, link to product names and software versions, what the relevant risk entailed, etc. they were recorded, visited, and their information archived. We checked to see if CVEs were missing/reserved, validated scores to determine if they were marked correctly and did the CPE strings reflect initial expectations (e.g., did the vendor name match, or was the product name correct?). The information was cross-referenced with reasonable scrutiny to discern common links & identified transgressions (e.g., X product is using Linux or Y license manager because it said so).
Like it or not, Information Technology (IT) is highly prominent in industrial environments, and it has been present there for decades. IT OT convergence is a “hot topic” today, but, in fact, it has been a reality for over 20 years. Then came the ubiquitous nature of devices, and networking and here we are. Windows, Linux (in devices), Webservers on devices using open-source components, and even IT networking is creeping into ICS environments at a tremendous rate; this is not saying ICS/IACS boundaries will go away, but the delineation is rather blurred and jagged.
Starting with the big names in the room, it appears that vendor product security teams had a busy year in 2020, but not particularly more so when compared to years past:
We focused on these names because they form key elements of the OT environment. In no way is this a comprehensive review of all IT-related vulnerabilities present in OT environments. But as you can see, these risks overwhelm the number of ICS advisories and need to be addressed within the OT vulnerability management program.
------------------------------ 2020 vulnerability disclosures are roughly in line with expectations - an increase from Microsoft, a slight decline of CVEs in Cisco and Linux totals (albeit some classification issues) ------------------------------
The number of Microsoft vulnerabilities increased significantly in 2020 by almost 100%. And as can be seen in the below chart, the number of disclosed vulnerabilities has increased dramatically since 2014. This has come with leveraging more open-source software as well as with a greater push by Microsoft.
An additional review for more noticeable vulnerabilities for Microsoft specifically highlights the fact that that many of these Microsoft vulnerabilities do not involve the version of the OS, itself. They are components of the overall Microsoft software stack. The reason this is important is that to identify the presence of these vulnerabilities and to remediating them appropriately, any evaluation of those assets must include deep software assessment, rather than just identifying the OS version which might be available in the header of a packet. True vulnerability assessment requires a 100% software view on each asset. Too often we see organizations rely on simply the OS to determine patch status. But in fact, this is often misleading.
Linux vulnerabilities declined in 2020, but this should not provide a false sense of security.
Beyond the above observations, a further point to add is that a variety of issues (bugs) are resolved in either OS, but the Linux ecosystem seems to tend to solve issues quicker and go without disclosure reporting.
Cisco is another interesting vendor where they too had a drastic upshift earlier in the decade, but 2020, resulted in a decline. There may be multiple reasons for the increases between 2012 and 2018 due to:
Managing ICS vulnerabilities is a tough business for operators and their cyber security peers. Lack of robust hardware, software, and firmware inventory, challenges with risky IT vulnerability scanning, lack of vendors’ contributing to public vulnerability databases, lack of knowledgeable ICS resources all make this a challenging process. The future is frightening as well. As vulnerabilities grow and attackers discover more weaknesses the burden on OT staff and their cybersecurity peers will grow significantly. Without robust, ICS-specific endpoint security management tools the burden will grow.
This is becoming a huge risk and dramatic burden on already overtasked OT personnel in the years to come. The growth rate in ICS vulnerabilities is accelerating rapidly with a growth of 65% over 2019. Furthermore, the number of devices impacted is growing rapidly and CVSS scores are increasing too. On average, 62% of embedded vulnerabilities affect multiple products, and 99% affected multiple versions. This is on top of the ~50% increase in 2019 over 2018. We are getting to a point where these vulnerabilities cannot be ignored – or managed manually through OT teams.
Importantly, these below numbers do not account for the large number of vendors that normally do not make public disclosures but instead provide vulnerability information privately to their customers. If these were to be included the total numbers may double. They are not even present in one database which can be leveraged by vulnerability teams without significant manual processes. This growth of risk combined with a lack of automation is a recipe for security disaster.
From the 257~ original ICS-CERT advisories, marine, medical, and a specific tractor-trailer advisory were excluded, and 248 advisories remained. Of the 248 advisories, they had a collective average score via the tagged CVSS of 8.0 (HIGH), and there were more than a single vulnerability assigned for each advisory.
In addition to the above summary statistics:
It is also important to note that a higher disclosure rate by vendors might be a sign of security program/development lifecycle improvement, and not the opposite – more reporting means more active & investment in security or a change in internal culture, which hopefully will result in improved products overall. It may also signal that the vendors not reporting issues, are not practicing sufficient security, and should be forced to do so by the asset owners (who have a decisive voice).
In addition, from a data perspective this chart has several caveats that a reader needs to be aware of:
Of course, patching all vulnerabilities is not a panacea, nor is the reporting of issues, but many ICS assets have updates that are not cited as security-related and improve the stability of various products due to the elimination of “errata” or “anomalies”; which can be useful for ensuring SRP over the asset’s lifetime.
Each of the ICS-CERT advisories had difficulty/vector/exploit and problem/issue fields. Low skill level to exploit, exploitable remotely, and stack-related programming issues are still prevalent. We saw an increase in the number of low skill/remotely exploitable vulnerabilities. Over 80% of all vulnerabilities have a low skill level required to exploit.
If an attacker can get access, most vulnerabilities are exploitable with the relatively low skill required.
Of the 248 vulnerabilities 86 unique vulnerability/issue values were found after sorting and counting. The top 20 issues and their frequency are as follows. In summary, they include:
The above vulnerability causes highlight the significant risks to operations that these vulnerabilities present. These are not “minor” issues. In critical infrastructure, these processes must operate at high reliability and unapproved changes can cause significant damage or downtime.
Over one-third of alerts involved the ability to get around authentication and validation requirements offering the potential for attackers to make changes to critical processes. Another 30% involved the possibility for buffer overflow where the impact could be to overflow devices with information thereby causing operational disruption. So two-thirds of all vulnerabilities involved those that could cause significant damage to the environments if unfettered access by a malicious actor was able to sufficiently breach security controls.
It is important to note:
In the 2019 causes, there may be lesser occurrence counts due to overall numbers of CVEs (~300 less), and advisories (~50 less). However, there are similarities to the 2019 advisories – programming errors, stack overflows, lack of sanitization, and improper use of security features/cryptographic techniques.
One of the most critical, but potentially overlooked areas of advisories are those related to supply chain. Beyond the newly appointed U.S. DoD CMMC framework for designated industries, NERC CIP-13 or BOMs for FCC certification by top tier vendors, there are a lot of misunderstandings and a lack of understanding of the security of packaged software products (including their embedded variants).
Regardless of not knowing what X organization may possess or even at what configuration it may be running in, there were several high-profile supply chain-related announcements in 2020 (e.g., Ripple20, Treck/IP, URG11, Amnesia:33, etc.). However, when you review the advisories, you find almost none of them are related to these “hot topic’ vulnerabilities. Many vendors argued that these were overstated for ICS devices. Some released private advisories on their websites for customers regarding potential devices. But finding these vulnerabilities within the ICS inventory of an organization is often very difficult without specialized tools.
In both 2019 and 2020, roughly the same amount of supply chain/component related advisories were issued. There are several issues with the reporting system, but the decrease in 2020, and the increase of products on the market hints at a hidden issue lurking below the surface; this is not limited to embedded systems, but software that has statically compiled dependencies, uses third-party libraries (FOSS or purchased/COTS), and more.
For example, many ICS devices (particularly those that provide networking) are often running very “old” components and are lacking (at a minimum), vulnerability disclosures citing CWE-1104: Use of Unmaintained Third-Party Components or CWE-398: Indicator of Poor Code Quality. This would indicate that new strategies are needed for managing components used within products, and the growing problem when OEMs are developing products using open source (FOSS) & are unable to handle the component’s astronomical update frequencies (that never are reflected into their actual product upgrade cycles).
To illustrate the challenge when reporting on components is illustrated below. This device is not by any means a cherry-picked lone example, thousands, potentially hundreds of thousands of devices are in a similar state with few vulnerabilities associated with them, but they have many components that likely are never upgraded, or have other issues.
The device above demonstrates other undocumented risks. It is a product still deployed at countless sites:
------------------------------ Countless device firmware dependencies are not tagged as an advisory on ICS-CERT, nor is the vendor (or many other vendors for that matter providing core OS updates). ------------------------------
To provide some context, the below chart is the number of Linux vulnerabilities in 2020. These components are present in thousands of pieces of software. Many ICS products would contain some of these components. But finding them requires a deep software bill of materials.
Many ICS devices have a very low product refresh rate, patching is difficult, products are abandoned by the vendor (e.g., end of life), teams have retired or shifted to other products, companies are acquired or sold, and embedded updates and changes can be complicated due to the proximity to the OT processes.
What then are the consequences or the results of a growing supply chain cyber security and interest area? It might just become a paradigm shift, and it might require some adjustment for security teams & product developers. Instinctively, with the embedded stack vulnerabilities released in 2019/2020 for URG11, AMNESISA:33, and Treck/IP – it would have been assumed there would have been a very large portion of affected devices & numerous advisories, but this was not the case.
------------------------------ "Branded" embedded stack vulnerabilities resulted in few advisories on ICS-CERT, and the claims of millions of devices affected is questionable. (E.g., if 33 CVEs are in the "vanilla" stack, why do only some apply? Are there differences in scores and applicability?) ------------------------------
With relation to supply chain issues: Verve predicts that over 2021 that many asset owners and researchers will begin to assess more device firmware for inherently insecure components and the results might be as follows:
The above summary should worry all of us involved in ICS (or OT) security. The number of published vulnerabilities is increasing and they are becoming more severe. In a way, this is good news. Vendors are being more transparent; researchers are producing more insights, and our joint institutions are sharing like never. But the news is a wake-up call to everyone trying to operate the critical control systems that operate so much of our economy. We need to do better to do identify the vulnerabilities in our environments and eventually remediating those risks.
------------------------------ Operators face a set of challenges in dealing with the data provided by ICS-CERT and other vendors. ------------------------------
First, they need to inventory all of their assets including 100% of the software on those devices as well as the firmware present on the embedded devices. This needs to be combined with ICS-CERT/CVE data as well as vendor-specific releases that may not be done publicly.
Second, develop a comprehensive risk assessment of each asset that includes the vulnerability as well as the compensating controls that are currently enabled on the asset.
Third, develop remediation strategies relevant for each asset given its criticality, risk score, and feasibility of patching/updating.
Fourth, monitor networks and endpoints for potential behaviors that may indicate exploitation of known vulnerabilities.
Fifth, ensure appropriate governance and procedures to both reduce risk and ensure a rapid incident response.
2020 was another year of increasing alarm bells for ICS security. As stated above, additional vulnerabilities are not “bad” per se. They likely indicate a maturing industry. The growth rates, however, should be a wake-up call that the time to act is now. Greater connectivity is coming – not to mention all the current remote access that already exists as evidenced in the recent attack on the water facility in Florida. As this connectivity increases, the access to these vulnerabilities increases. No longer is it enough just to argue that “air gaps” protect these endpoints.
OT security management (OTSM) is a necessary requirement of all ICS environments. The increase in vulnerabilities highlights the tip of an iceberg, and that iceberg is heading our way. The current approaches to OT security relying on perimeter network defense and monitoring, will not be enough. CISOs, boards, regulators, and other stakeholders will require that ICS apply the same endpoint management and security as traditional IT environments. As this happens, the burdens on already overtaxed OT teams will explode.
The time is now to begin to get ahead of this and develop a plan to identify and manage these vulnerabilities – and those to come – in a more holistic way than today’s primarily manual efforts.
There is much good news as well. The fact that more researchers and companies are sharing vulnerabilities means maturity is increasing. Although one might argue whether some have been over-hyped, the more “expansive” disclosures such as Amnesia 33 or Urgent11 have caused an increase in discussions about the risks of these embedded OT devices. We also see an uptick in overall recognition of the need for some visibility into OT.
However, the challenge will continue to grow over the coming years. We hope the industry meets the challenge with an increase in attention and focus at the same rate as the growth in known vulnerabilities.
Verve Industrial Protection assessed the history of OT cyber attacks and determined where they are likely headed based on twenty-five years of industrial cyber security experience in ICS environments.Learn More
Download our on-demand webinar to discover how to achieve the greatest risk reduction for the time and money available.Learn More