Colonial Pipeline Attack: Lessons Learned for Ransomware Protection
How to leverage lessons learned from the Colonial Pipeline ransomware attack to prepare for cyber-related threats in oil & gas.Learn More
Subscribe to stay in the loop with the latest OT cyber security best practices.
In early 2022, we released a blog on how to prevent ransomware in 2022, with a specific focus on OT and critical infrastructure. This followed a year (2021) in which ransomware exploded both in terms of actual attacks as well as the public awareness of the threats.
Between May 6 and May 12, 2021, Colonial Pipeline, owner of 5,500 miles of pipeline carrying natural gas, gasoline, and diesel from Texas to New Jersey, shut down its operations in response to what it said was a ransomware attack targeting its IT network.
After the Colonial incident, several other major ransomware attacks on operating entities have been reported: Martha’s Vineyard Ferry Service, FUJIFILM, and the JBS meat company who supplies 40% of all the US meat supply. These came on the heels of several other large public ransomware events at the second-largest paper company, Westrock, Molson Coors, and others in 2021.
We highlighted the risks to industrial organizations based on a report by Digital Shadows which found that industrial goods and services was the number one most targeted industry in 2020 at 29%, and the number of attacks was more than those on the next 3 industries (retail, construction, and technology) combined.
IBM X-Force research showed that manufacturing and utilities had moved from 8th and 9th most targeted industries in 2020 to the top three in 2021. Further, the 2021 SANS ICS security survey found that ransomware became the top threat risk according to ICS security practitioners… by a factor of 2x over other threats and moving to number one from sixth in 2019.
We offered a range of recommendations to limit ransomware attack in industrial environments such as prioritizing risks based on a “real-time 360-degree risk assessment”, ensuring updated backup and recovery capabilities, applying critical patches, ensuring intended network protections are enabled, etc.
We wanted to step back as we enter 2023 to: review how the industry did in terms of reducing the ransomware impact on the economy and individual companies, provide an outlook for 2023, and update the recommendations from earlier this year.
Ransomware is a form of virus or more commonly called malware. Essentially the bad guys find a way in (phishing, social engineering, etc) to first invade the target network. Their ‘software’ then runs around the network (traversing network shares, local drives, etc) encrypting everything it finds with a key that only the bad guys know. If you want to unlock your files you have to pay the bad guy to give you the key. The costs to get the key and decrypt files can range from hundreds to thousands or even millions of dollars depending on the specifics of the attacker and victim.
Ransomware has roots in the scam and extortion criminal world, but by nature, it can also be used to target larger asset owners and organizations or to mask other activities that might be more devious.
Let’s first look at why ransomware is becoming such a challenge for industrial organizations today:
To put the cycle into perspective the diagram below illustrates the typical path ransomware takes to get into a facility:
The forecasts at the end of 2021 and early 2022 all called for a continued explosion of major ransomware attacks across the developed economies. However, the reality is that the predicted explosion never materialized. There is some debate depending on the source around whether ransomware slightly increased or actually decreased during 2022. But regardless of which data set one looks at the feared ransomware pandemic never occurred.
Using the same data source as we did last year, Digital Shadows, Q3 2022 saw a slight (~10%) reduction in incidents. This followed a slight increase in 2Q and a basically flat 1Q. Although Q4 is not over as of this writing, there has been no obvious public increase to be seen. This is just one source, but it is somewhat typical showing slight increases and decreases throughout the year, not another massive spike.
Analysts have offered several possible rationales:
In this theory, the primary ransomware attackers learned from the very public reaction to the Colonial and other attacks on large infrastructure providers and refocused their efforts on smaller organizations, governments, etc. As we wrote in early 2022, “By Monday, the DarkSide attackers expressed contrition for the Colonial Pipeline attack. Perhaps in response to the international publicity and the focused governmental and law enforcement efforts spun up in the wake of the incident, the hackers took to their dark website to say they never intended to disrupt public utilities.” “We are apolitical,” the hackers wrote. “We do not participate in geopolitics, do not need to tie us with a defined government and look for other our motives. Our goal is to make money, and not create problems for society.” The resulting government focus recovered a significant portion of the ransom payments (although merely a drop in the bucket relative to the economic externalities) as well as an active pursuit which forced many Darkside to reorganize and dismantle some of its infrastructure.
So, in this theory, the attackers’ financial incentives drove them to adjust their targeting to reduce the potential size of ransomware events and impact on the economy.
As stated, the US and other governments took rapid (for the government) action against the ransomware groups involved in many of these large, public incidents. While we don’t know all that they did, it certainly included the active pursuit of the groups’ infrastructure, their payment forms and blockchain accounts, rapid response for organizations impacted by ransomware, indictments of individuals, increasing private sector awareness of the threats, etc.
In this theory, the government’s actions significantly impacted the ransomware groups’ ability to act as they had historically.
The war in Ukraine has not only impacted the physical space and human lives, but also the digital “battlefield”. As the war was approaching, the US government deployed forward teams into eastern Europe to support Ukraine’s efforts to defend against Russian attacks. In addition, Ukraine invested significantly in its own defenses, focusing private groups on national defense. At the same time, Russian groups who may have formerly focused on private sector ransomware or other initiatives, were likely repurposed by their government supporters to focus on the Ukrainian war effort.
In this theory, the combination of events surrounding the war significantly reduced the capacity of Russian (and other Eastern European) hacking groups to focus on the financially-motivated ransomware attacks as they did in 2020 and 2021.
Two significant financial dynamics impacted ransomware in 2022. First, the devastation of the Bitcoin and other crypto markets. The primary vehicle of payment within the ransomware community collapsed in 2022, taking away the stability of pricing of a ransomware attack…on Monday a 25 Bitcoin attack may be worth 20% less by Thursday. Further, it is known that ransomware groups maintain much of their capital in crypto. The collapse of the currencies could have had a major impact on the ability to fund future campaigns.
Second, insurance companies had been a key funder of ransomware payments up to mid-2022. New insurance policies limiting the payment for ransomware and public pressure on insurers not to add to the “moral hazard” of paying one ransom which might incite more both changed the calculus for the attackers to some extent.
In this theory, the financial benefit and overall architecture of the payment system shifted the cost-benefit of ransomware as a service business models.
In the aftermath of Colonial Pipeline and the emerging war in Ukraine, the US government (and others around the world) significantly increased its campaign to support private sector. This included the US Executive Order, CISA’s Shields UP program, etc. In addition, insurers, boards, regulators and others increased the focus on cyber in many industrial organizations. Many tried to respond to these initiatives as well as they could.
In this theory, organizations with help of governments were able to stand up defenses to at least reduce the massive increase that was expected.
So, no explosion in 2022…but do we expect to see the return of rapid acceleration in 2023? To begin, the lack of a rapid increase in 2022 does NOT mean that ransomware is still not the most significant threat to industrial environments. Again, depending on the analyst, ransomware was at least flat year over year in 2022 – flat at a very high level compared to historical data.
The attack landscape did change significantly in 2022. Lockbit2.0 remains by far the “market share” leading group and the mid-year shut-down of one of the most notorious groups – Conti – has meant an increasing concentration of share by Lockbit…as well as the rise of a multitude of new competing groups – Basta, Hive and several others. There is some belief that many of these different groups are re-banded groups of Conti as they separate and reform.
In 2022, Lockbit code was released which may encourage imitation, reforming, etc. as happened with prior ransomware groups. This release could create a rapid acceleration of similar, but evolved Lockbit core elements in 2023 as new innovation is applied to that source code.
A breakdown across Q2 and Q3 of 2022 shows the shift as Conti shuts down.
The bad news for industrial sectors is that it continues to be the primary focus of the ransomware actors according to Digital Shadows.
And this aligns closely with data from Abnomal, IBM, and other sources.
As we look to 2023, it is clear from the above that the industrial sector is still under significant ransomware threat. The question is whether we get another year of just sustained high-risk levels or one that looks more like 2021 where we saw a dramatic increase in ransomware activity from these elevated levels.
We believe this really comes down to 3 questions:
The speed of this likely has to do with the evolution of the war in Ukraine as well as the stability of the crypto community. The last several months of 2022 have created increased uncertainty in these two areas resulting in fewer attacks. In 2023, we would expect each of these factors to shift with attack groups finding alternative monetization vehicles. In addition, the war in Ukraine continues to drag on. Our view is that the Russian hacking community will re-find its commercial footing in 2023, if for no other reason than to retain talent and fund future operations, and will again increase the ransomware activity against the private sector.
Over the past several years, ICS security vendors have highlighted the growing number of ICS vulnerabilities – and discovered many insecure by design elements that became ICS-CERT advisories and CVEs such as the so-called “IceFall” vulnerabilities. And although these are significant (and get fancy marketing-oriented names), the real threat that would drive a renewed explosion in ransomware and other threats would be the presence – and exploitability – of something like the SolarWinds, Log4J, SMB, or EternalBlue hacks and vulnerabilities. These risks provide attackers with a broad target environment using similar approaches. The public emergence of such a risk would enable copycats and potentially release untargeted malware into the community as happened with Wannacry/NotPetya in 2017/2018.
The potential for just this type of situation was on display recently with CVE-2022-37958. This vulnerability is similar to EternalBlue in that it can leverage communication protocols to spread. The Windows vulnerability was patched with the September Windows update. But the challenges of rapid patching in OT make these kinds of risks ever more significant.
In the past couple of years, security vendors have highlighted a range of possible OT/ICS ransomware variants and possible strategies. To date, none of these have appeared to have any significant impact. Ransomware really has focused on the Windows OS devices in the OT environment – servers, workstations, HMIs, etc. We still strongly believe this is the 90% of the focus for ransomware in OT. OT-specific ransom focused on PLCs and other embedded controllers is more difficult to develop and deploy. This takes time, energy and money. None of which has been in large supply during the past 18 months. As we look into 2023, should the payment and Russian-Ukraine distraction get resolved, we could see a significant acceleration in deployment of such OT-specific attacks.
Given the current state of risk and the potential for a renewed acceleration in ransomware incidents in industrial environments, how should organizations respond?
To gather this picture, an organization needs to have three key pieces of information:
Too often we have seen organizations jump into a certain initiative as a way to make traction on reducing the risks from ransomware (and other potential OT attacks). For instance, a frequent starting point is a comprehensive network segmentation effort to reduce connectivity between IT and OT as well as segregation within the OT environment. This certainly is part of a robust roadmap. However, it may not be the most impactful first step in the overall program.
Understanding risks, but also a sequence of initiatives is key to making rapid, but sustainable progress. Verve works with our clients to create a “portfolio of initiatives” that build on one another. For instance, it is usually very difficult to conduct a proper network segmentation without a clear picture of the assets on the network and how they are communicating with one another. Therefore, a robust inventory of the environment accelerates the eventual segmentation efforts. Similarly, some initiatives may offer rapid impact – e.g., leveraging backup tools that may already in place, but ensuring they are used and updated. This sequence of initiatives at a site and enterprise level provides a roadmap that allows for near-term protections and recovery capabilities while building the longer-term foundation of protection and detection.
Then it’s time for remediation. One of the advantages of the “Technology Enabled Assessment” mentioned above is that the technology is already in place to be able to immediately remediate identified risks – from patching to configuration hardening to managing risky software, users & accounts, etc. The TEVA accelerates time to protection.
But beyond accelerating those endpoint detections, there will be a range of additional protections and response capabilities necessary. One of the biggest challenges is to determine the appropriate execution plan to protect the most critical sites and assets, while not getting bogged down on these large/complex sites and never getting breadth of protection to the “medium” criticality sites.
Verve recommends what we call a “bi-focal” approach to the execution. On one lens, we certainly would pursue a robust program deployment across the most critical sites. This would include a comprehensive scope of initiatives as listed below. However, in parallel, we would encourage a broad and shallow approach to apply limited protections to all sites at an enterprise level while the deeper efforts are occurring on the critical sites.
What this means in practicality is that while the “gold” or most critical sites may need comprehensive network segmentation, new infrastructure, advanced anomaly and threat detection, backups, patching, user & access management, etc. However, at the “silver or bronze sites which individually may be less critical, but together make up a significant risk, you might apply prioritized vulnerability management and backups while waiting on a more comprehensive network segmentation effort.
In many cases, the implementation of a security program is a resource-intensive task, but it is critical that the organization plan up-front for the maintenance of any improvements achieved during the program. In Verve’s experience, this includes two key elements:
As one of our colleagues says “security has a tendency to rot”. Network rules put in place initially get changed during maintenance windows, updated patches don’t get applied, AV signatures updates get delayed, new assets are added but never inventoried, backups fail and are not remediated. A maintenance program with robust performance targets is key to any successful program.
Perhaps this should be the number one item. We include it last because it is most critical in the maintenance period of the program. Certainly, the organization needs to be aligned initially. Without the buy-in from operational leaders, security programs cannot get off the ground. However, we see most of the commitment challenges happen once the program is launched and the hard work of maintaining begins. People are called back to day jobs, other priorities arise, budgets get reallocated, etc.
It is key that organizational commitment is more than a one-time effort. In our experience, the best way to accomplish this is through the alignment of balanced scorecards that include OT security as an element.
The below list provides some specific guidance on what we typically see as successful elements of a program.
Effective endpoint management begins with a robust asset inventory. As the age-old saying goes, if you don’t know what you have, you can’t manage the risks. A rich view of a 360-degree picture of each endpoint enables proper endpoint management.
Most threats enter through commodity systems such as Windows machines. You cannot patch everything in OT, but an end-to-end patch management program (i.e. automation and intelligent application of patches) is of great importance due to several environmental factors such as compliance, legislation, and risk management (e.g., patches on hosts with RDP or firewalls connected to the Internet should be prioritized over a PLC protected by several layers). The reality of today’s ransomware is that it focuses on OS-based devices (servers, workstations, HMIs). These are where the primary focus should be when it comes to managing patches to address ransomware. Where unfeasible, application whitelisting, and policy enforcement makes an attacker’s life very difficult to improve your chances to defend or deny a ransomware attack on your OT organization.
This begins with application control of new software that might try to run on HMIs, Workstations, etc. In IT, this solution is quite challenging to maintain given the breadth of new applications that are necessary. In OT, however, most systems should be “locked-down” and new applications are unnecessary. Therefore, DHS strongly recommends Whitelisting as one of the top 2-3 initiatives to take. Whitelisting extends to USBs, removable media, and transient devices as well, especially if your network is “air-gapped” or heavily controlled. Users WILL bypass your controls by way of removable media. As a best practice, system policies are easily deployed, whitelisting software used, registered secure drives, and other technologies such as 802.X ensure authorized systems are allowed on network segments.
Any security program will not be sufficient to stop every attack. Therefore, a comprehensive backup program is critical to ensure rapid recovery. This includes prioritizing systems to backup, ensuring timely backups, monitoring for failed backups (which seem to happen on a regular basis in many OT environments), and ensuring replication in an offline repository so that the malware doesn’t limit their effectiveness.
One key way to slow the spread of ransomware is to place network barriers between IT and OT (or even within segments of IT and/or OT) networks. This approach is a foundational element but one, because of its technical challenges, is often underutilized.
Taking these five steps reduces the risk and impact of a ransomware attack, leverages existing technology investments, and improves recovery in the event of a compromise. Each of these adds successive protections and safeguards against a possible ransomware attack.
OT-specific challenges are identified in this document not to show that a robust OT security program is unattainable or improbable but rather to help the reader identify key decision points that will help a successful program achieve maximum protection with minimal challenges.
The application of ‘IT-like’ security controls in OT is increasingly being achieved in numerous industries, companies and countries around the world. But the true measure of success is in the maintenance and monitoring of their initial efforts. The companies that are significantly improving their security posture are acknowledging the unique challenges of an OT environment and making decisions such as:
How to leverage lessons learned from the Colonial Pipeline ransomware attack to prepare for cyber-related threats in oil & gas.Learn More
Learn how CISOs and OT cyber security leaders should manage risk in industrial OT environments against key drivers.Learn More
Most attack surface management tools and approaches do not understand the technical complexities and operational requirements of these OT systems. But there is a way to effectively and efficiently conduct ASM in OT.Learn More