ICS system hardening required to improve operational resilience, boost overall cybersecurity posture

Increasing cybersecurity threats and attacks have compelled organizations to concentrate on implementing ICS system hardening into their operational environments. The approach works towards bolstering the security perimeter covering interfaces that connect the control system to external networks, device connections to ICS networks, and wireless device access points. It also encourages organizations to take preventative measures to protect their critical infrastructure, reduce the attack surface of the ecosystem, and get rid of potential threats to protect the assets of their business.

ICS system hardening aims to reduce organizational visibility to threat actors by removing unused programs, apps, account functions, permissions, ports, and unauthorized or expired user access. To meet system hardening standards, security teams must consistently commit to monitoring, detecting, resolving, and controlling vulnerabilities while eliminating potential attack vectors and condensing the environment’s attack surface. By removing unnecessary programs, user accounts, functionality, connectivity, ports, permissions, physical access, etc., malicious attackers and malware have fewer opportunities to gain a foothold within the OT environment.

Technology intelligence firm ABI Research forecasts that by 2030, connected machines and production systems within factories will exceed 1.2 billion connections. Industrial 4.0 and industrial Internet technology investments are expected to grow from US$41 billion in 2022 to nearly $200 billion by 2030. This burgeoning environment clearly expands the attack vector and given the assortment of networks, endpoints, and connection protocol heterogeneity, will contribute to a variety of security solutions.

To mitigate attacks against industrial organizations, ABI said it is necessary to adopt three major ICS hardening solutions in parallel with each other. These solutions are endpoint and network security, and securing ICS protocols. Basically, these are areas where hardening takes place.

ABI also outlined that most ICS-connected components, like PLCs, are still linked by cables and physical fixed lines, like Ethernet. It added that most of all other connected industrial applications, including industrial pumps, intelligent industrial electric motors, and connected robots are also connected with fixed lines, thus protocol cybersecurity and the use of firewalls, authentication technologies, and unidirectional gateways will see strong demand for the foreseeable future. 

Industrial Cyber reached out to cybersecurity experts to evaluate the strategic steps and tactical actions that the critical infrastructure sector must adopt when it comes to tackling ICS system hardening across cyber-physical systems. They also throw light on the benefits that ICS system hardening brings to OT/ICS environments. 

“While IT systems are generally subject to continuous daily/weekly/monthly changes including continuous installation of applications and system updates, OT systems are rigid in nature and are commonly subject to fewer changes compared to IT systems,” Ayman Al Issa, senior expert at McKinsey & Company, told Industrial Cyber. “Hence, hardening OT systems is more reliable, feasible, and achievable in OT than in IT systems. Hardening OT environment should be considered during the design, implementation, and operation of the control systems lifecycle.” 

He added that strategically, the hardening of the OT systems should be initiated by the automation manufacturers during the design of the ICS systems, be adhered to during the implementation and commissioning of the control systems, and be enforced through strict procedures for hardening the systems during the operation of the ICS systems.

“Tactically, OT system hardening is a continuous process that should be practiced and owned by all stakeholders of the ICS systems during the lifecycle of the systems, this includes OT system owners, system integrators, automation vendors, cybersecurity vendors, and all third-party vendors operating within the OT environment,” according to Al Issa.

He added that ICS hardening processes should at least include designing secure reference architecture specific to each OT environment that is based on zero-trust principles; designing secure identity and access capabilities to the control systems and within the control systems; and hardening the ICS systems is a process and practice rather than a technology, hence it should be a way of life during the implementation, commissioning, and operation of the ICS systems during all their lifecycle.

“OT environments need very high stability in the network and systems. System hardening is not only vital for cybersecurity enhancement but also for improving the systems and network performance within OT environments and maintaining production or operational services,” according to Al Issa. “System hardening and proper configuration of OT systems can significantly reduce the attack surface to a remarkable level and enhance the protection of critical infrastructure, maintain high level of system and network performance within the ICS environment, and serve as the first line of defense to minimize the opportunities for compromise to avoid operational downtimes that could lead to loss of operations or production,” he added.

Josh Beed, solutions architect for TXOne Networks, outlined that critical infrastructure operators must identify critical cyber assets, carry out network segmentation, patch, and vulnerability management, access control for critical systems, and bring about monitoring and detection of malicious or anomalous activity when hardening the ICS systems. 

“ICS system hardening will benefit the ICS environment by ensuring that attackers do not have an open door into the environment,” Beed told Industrial Cyber. “Each step to achieving system hardening in the ICS environment is designed to secure the system against an attacker or threat. Identifying and controlling access to cyber assets will give visibility to what cyber assets are in the ICS environment and who has access to the cyber assets.” 

Network segmentation helps control lateral movement between segments which adds benefit if an attacker does gain access to the ICS environment, according to Beed. “The damage is contained to a smaller segment of the environment. Patching ICS systems or having mitigating controls through virtual patching will close security holes in the systems before they can be exploited while monitoring and detecting any malicious and anomalous activity will help stop an attack in the event that one is able to get by the system hardening,” he added.

“To provide system hardening properly a few things need to come together,” Rick Kaun, vice president for solutions at Verve Industrial, told Industrial Cyber. “First would be a target hardened state – i.e., CIS benchmarks? DISA STIG (US gov so good for some, not for others). In other words, what is the standard you are trying to achieve? The second would be to have a confirmed method or technology to allow you to both measures and, preferably, apply/enforce system hardening parameters. Now in OT, some classes of asset (by fragility or operational impact) may fall outside of anything other than manual hardening and enforcement but tracking and reporting presence of configuration settings is key.”  

The final step would be to be able to use that report to track and eventually remove technical exceptions to hardening, according to Kaun. “We often find clients have pockets of older or less sophisticated assets that can’t apply some controls. Tracking these as part of your current risk profile as well as for planning future upgrades and security requirements to phase out exceptions is very important as well,” he added.

Kaun said that system hardening is probably one of the most powerful tools in an OT/ICS arsenal. “Since so many OT endpoints are ‘always on’ or are built on legacy platforms and can’t run the latest software or full slate of patches the protections fall to ‘compensating controls,’” he added.  

Ali Bozorgmir, global director for OT security services at OTIFYD, said that tackling ICS system hardening in OT is easier said than done. “There are a number of reasons behind prolonged process of system hardening or in general poor patching hygiene in the industrial environment. Impact on operations, rigid testing requirement, OT system complexity, slow security advisories response from vendors and limited window of opportunity during factory scheduled shutdown make system hardening a challenging task for the operation and cyber security teams,” he added. 

Bozorgmir told Industrial Cyber that is why more organizations are moving away from a maturity-based model and shifting towards a risk-based cybersecurity model that involves numerous steps including asset inventory by creating a comprehensive inventory of all OT assets; vulnerability assessment that performs regular assessments to identify vulnerabilities in the OT environment; risk assessment that conducts a thorough risk assessment to identify potential risks and prioritize security measures based on impact and criticality; and system hardening management and planning that establishes processes for timely hardening and tracking vendor security advisories and threat intelligence reports.

He also included a test plan that outlined the specific objectives, scope, and steps for testing the hardening process; test environment isolation that separates the test environment from the production to prevent any unintended consequences; compatibility and functional testing that verifies the compatibility and functionality of hardware, software, and configuration; performance testing that assesses the performance impact of the hardened assets and compares the system’s performance before and after; and documentation and reporting.

“Just like in the IT environment, ICS environments benefit from asset hardening to enhance overall cyber security posture and improve operational resilience,” according to Bozorgmir. “A proactive asset hardening can minimize downtime and operation disruption that could be the most important outcome desired by the operational team. Additionally, regulatory compliance, long-term cost saving, and maintaining trust and reputation are some of the other non-security related benefits.”

The executives examine how ICS system hardening helps lower risks related to ransomware, insider threats, and the rise in state-sponsored attacks because it is a crucial component of OT security. They also analyze whether businesses must incorporate ‘secure-by-design’ into their ICS system hardening strategy. 

Al Issa said that hardening the OT system borders helps to reduce the opportunities to conduct lateral movement to the OT environment, also hardening the borders between the OT systems within the OT environment reduces the potential of a lateral movement to multiple OT systems in case one of them is compromised. “Hardening the OT systems (servers, workstations, HMIs, PLCs, etc.) helps to reduce the opportunity to compromise the OT systems themselves. In the end, there is nothing such as bulletproof cybersecurity, and what can be done is to make it difficult for attackers to achieve their goals by combining hardening with effective security controls, including building advanced capabilities for enabling early detection of threats,” he added.

“Security-by-design or zero trust, whatever we call it, is the first step in OT system hardening. As discussed above, hardening should be considered during the design, implementation, and commissioning of control systems,” according to Al Issa. “In reality, this is easy to say, but in practice, it could take a year or even 5 or more to commission OT systems as it includes an EPC (engineering, procurement, and construction), then operation stages to get the OT system operating on the plant.” 

Al Issa added that during those phases, the control systems are subject to several configuration changes that could make them vulnerable by design. “Hence, hardening control systems should be practiced and trained for all parties that are involved in the design, implementation, and operation of control systems.”

“ICS system hardening can help reduce the risks of malicious lateral movement within an ICS network as well as ensure that malware does not propagate throughout the network,” Beed said. “With the rise in state-sponsored attacks and insider threats, it’s more important than ever to make sure that the ICS network is properly secured. ICS OEMs are already testing and building security into the design of their products to help meet the secure by design mindset.”

Beed added that the more secure the product is when coming into an ICS environment, the easier it will be to maintain the required level of security. 

System hardening is reducing the attack surface of individual endpoints – it takes away common attack vectors and access points from target systems, Kaun said. “We are strong advocates of ‘residual risk’ which is an overall or 3-dimensional view of an asset. This means a risk profile that incorporates the asset itself (patch levels, software, firmware, etc) coupled with third-party indicators of risk (vulns, exploits, etc), user-defined tags (asset criticality to operations, asset redundancy, zone of the Purdue model, etc), additional security protections (AV/whitelisting status, backup status) and finally system configuration.”  

Kaun added that it is the final line of defense and rounds out a fully informed, contextual view of risk as opposed to individual indicators like a vulnerability report without all the above context.

To answer this, Bozorgmir mentions that system patching and asset hardening are often used interchangeably, but they are different. “Hardening involves some additional steps beyond patching by disabling features and obfuscating system components to limit access for adversaries.”

“Compared to nation-state hackers and insider threats, ransomware actors’ TTPs (techniques, tactics, and procedures) are usually less sophisticated. They try to take advantage of unpatched systems by exploiting vulnerabilities to infiltrate and spread their malicious software,” Bozorgmir said. “Insider threats and state-sponsored cyberattacks are more sophisticated, and adversaries may leverage native control protocols and tools available in the control system environment.”

Bozorgmir added that systems hardening reduces security risk by eliminating potential attack vectors and condensing the system’s attack surface. “By removing superfluous programs, accounts functions, applications, ports, permissions, access, etc. attackers and malware have fewer opportunities to gain a foothold within the OT ecosystem.”

He added that if there is any lesson to be learned from Chernovite’s Pipedream toolkit/malware, it is that we can’t leave the end customers, as the last line of defense. “The principle of integrating security considerations throughout the entire lifecycle of an ICS system, from the initial planning, design, development stages, and implementation, is essential,” Bozorgmir added.

The executives also discussed how sustainable it is for OT systems to be permanently hardened, in the wake of rising cybersecurity incidents and volatile geopolitical situations. They also address how far OT/ICS environments have come in putting hardening measures in place.

“Definitely, ICS system hardening is sustainable in the long term. When we enable only the services, applications, communications, protocols, access, etc. that are needed, then we are reducing the attack surface to as minimum as possible, adding to these proper configurations and early detection of threats, we establish the core pillars of OT cybersecurity protection,” Al Issa said. 

Considering the increase of state-sponsored attacks, the protection of critical infrastructures is the responsibility of everyone, starting from the government/regulators, moving to ICS owners/operators, and including all the stakeholders who should enforce controls to harden the ICS environment, according to Al Issa. “However, as ICS environments have different specifics, the hardening procedures should consider the different environments and different ICS systems. This might lead to the need of specific hardening procedures for the different environments or systems.”

Looking into the progress made on this front, Al Issa said that the majority of the automation vendors started to issue hardening procedures for their systems, and the same for cybersecurity vendors, however, what is commonly seen on the ground is poor implementation of the procedures and best practices. Even legacy systems can be hardened, and in fact, hardening is the first step in the protection of legacy systems.”

One of the major challenges is the lack of skilled resources and personnel who have experience in hardening ICS systems, and how to achieve this without compromising the plant operation, Al Issa pointed out. “There is a lack of OT cybersecurity experts globally and hence we need to focus on developing educational programs that can help to develop graduates with skills and knowledge of automation systems, industrial processes, and cybersecurity. Developing ICS security labs in universities should be an enabler to enhance the protection of critical infrastructures.”

“ICS system hardening can be sustainable in the long term with a core investment in the security of the ICS systems. The standards and controls must be assessed and maintained to ensure that the system hardening endures with time,” Beed said. “With new standards and directives being released by the government, the responsible parties within the impacted ICS organizations are building playbooks and controls to meet the new standards within the timeline provided. As with all security, these standards and controls will need to be adaptive for new needs depending on the current risks in the world,” he added.

Kaun said that “absolutely – our clients see immediately how their OS and Networking devices are configured across a long list of hardening parameters. The status of the controls themselves is continually updated and reported (globally across all assets at all sites if required) so that current status, trends, changes, etc are all well understood and maintained.”  

“The continuous feedback loop just described is then paired with emerging risk and threat incidents. Hardening (under the above conditions) can then be continually evolved and adapted to the emerging risk,” according to Kaun. 

Coming to the progress that OT/ICS environments have made so far, and what remains to be done, Kaun said that “many are slow to embrace this OR rely on Active Directory as an approach which misses much in the ‘continual reporting’ and in the applicability when you consider many endpoints are not managed by a Domain Controller.”

Bozorgmir said that ICS system hardening is sustainable in the long term, but it presents a complex challenge. “It’s important to remember that ICS systems are designed to be operational for many years, and as such, they need to be able to withstand an evolving threat landscape. With this in mind, continuous improvement and threat intelligence processes should be part of any overarching risk management strategy. This ensures that system hardening baselines remain effective and proportionate to the assessed risk,” he added. 

The executives look into the role that zero trust architecture could bring about when it comes to working towards delivering ICS system hardening. They also assess the role played by IEC 62443 standards and NIST guidelines when it comes to ICS system hardening. 

Al Issa said that zero trust could seem to be an early topic to mention when covering the protection of ICS systems. “In practice, zero trust is vital to consider in OT environments. For green field systems, considering zero trust, security by design and defense in depth is a key player in the protection of the ICS systems. However, for brownfield systems, zero trust can be implemented at least at the borders of the ICS system especially at the firewall layers between OT and IT and for access to the OT environment from external networks including the IT networks.” 

He added that moving to zero trust within the OT environment could be achieved at the design phase of the control system, however, could be complex for systems that are already in production. 

“IEC 62443 and NIST standards contain security-hardening guidelines for industrial control systems (e.g., IEC 62443 4-1, SP 800-82), however, they cover high-level requirements that should be verified and tested by the automation vendors and ICS owners before they are directly implemented on ICS environments,” according to Al Issa. “Automation vendors and cybersecurity vendors should develop specific and detailed hardening procedures for the control systems. Finally, we can definitely say that the future of ICS cybersecurity starts with hardening the systems during the design, implementation, and operation of the ICS systems.”

Beed said that with zero trust, “everything must be assessed before being allowed. Nothing is trusted until the analysis is completed and that device or protocol has been allowed in the ICS network. Zero trust helps with the system hardening as each part of the ICS environment has been analyzed and approved prior to being allowed a network connection. This applies to network devices, protocols, services, and endpoint devices.”

He added that IEC 62443 and NIST set best practice standards to help achieve zero trust in the ICS environment. “These standards can be supplemented per industry with specific directives for a specific industry such as TSA 1580/82-2022-01 for the US Railroad.”

“Zero trust architecture is a key component but is just one component of actual endpoint and environmental hardening,” Kaun said. “They can provide ‘OT Safe’ system hardening guidelines and suggestions. Many hardening standards are not always ‘OT Friendly,’ and OT practitioners would benefit greatly from more ‘OT Specific’ guidance on this topic,” he added.

In the absence of a timely response for hardening ICS systems, zero trust architecture can be a valuable approach to protect OT networks, Bozorgmir said. “For example, it can be used to segment ICS networks utilizing micro-segmentation techniques. This provides a more granular segmentation model that places connectivity access controls east-west across an ICS network in addition to the more traditional north-south only segmentation model commonly found.”

“IEC 62443 standards and NIST guidelines play significant roles in ICS system hardening. They provide foundational best practices and guidelines that serve as a good starting point,” according to Bozorgmir. “However, due to an ever-changing threat landscape, an organization should take a proactive approach and consider threat intelligence to ensure their system-hardening baselines evolve themselves and remain effective and proportionate to the assessed risk,” he concluded.