Should You Start with Network or Endpoint in OT Security?

In Verve’s 30 years of experience in OT/ICS cyber security and reliability services, we often hear a debate about the best way to address cyber security risks. You hear about new vulnerabilities in ICS devices (40% more than last year). You hear about new threat actors targeting critical infrastructure. You hear about offerings from OT/ICS cyber security vendors.

And we’re asked questions such as:

• Isn’t it better to stop the attacker before they get through the perimeter into the OT network, rather than focus on protecting OT internal networks or endpoints?
• Isn’t it impossible to patch and harden OT endpoints, so network detection is really the only option?
• Once the attacker gets through the perimeter, how do you stop them?
• Isn’t the priority to identify malicious packets communicated to PLCs or controllers rather than try to monitor more IT-like devices in the OT systems?

Their real question they’re asking: Where should we begin in our ICS/OT cyber security program?

• Network Segmentation?
• Assessment?
• Asset Inventory (you can’t protect what you can’t see)?
• Vulnerability Management?

OT/ICS security teams hear different perspectives from different groups and are often left confused as to the best place to begin.

Where to start your OT cyber security program

The United States Cybersecurity & Infrastructure and Security Agency (CISA) recommends defense in depth as a way to protect OT/ICS environments. This includes items from policies & procedures to network, access, and endpoint protection. Through all of their advisories for new vulnerabilities and new threat actors, CISA is consistent in its advice. The good news is there are a set of programmatic things you can do to protect your organization.

Improving the reliability and security of ICS networks, Verve has successfully deployed in thousands of environments and conducted hundreds of site assessments of cybersecurity on OT systems. These deployments have called attention to how differently every organization approached their cybersecurity program, but it’s become evident the most successful programs start the same way.

Conducting a robust vulnerability assessment of the ICS/OT attack surface is key to getting your program right. While there is a temptation is to jump in to “do something” to demonstrate progress, organizations tend to neglect the importance of first assessing the situation because of how long it can take. While these companies jump to take immediate actions they know they will eventually need (such as network segmentation or threat detection), this is not the most effective or efficient approach. And may end up costing you in the long run.

For these reasons, we recommend beginning with a Technology Enabled Vulnerability Assessment (TEVA).  A TEVA gathers detailed visibility of the risks of the environment and prioritizes those risks. Unlike a traditional vulnerability assessment, it uses technology that gathers detailed asset inventory directly from the endpoints. This results in prioritizing the risks and remediations to gain the greatest maintainable security in the least time and cost.

Benefits of a technology-enabled assessment for endpoint-enabled security:

• TEVA provides an accurate asset inventory with data directly from endpoints in the network rather than relying on spreadsheets or scans for immediate response support or vulnerability assessment. A robust asset inventory is critical to the entire program and acts as the foundation to build upon.
• TEVA has several advantages over the traditional manual/survey-based assessment approach because it leverages technology, such as:
  • Lower cost because of the automation of the data collection and no need for travel to each site
  • Fact-based data on endpoint vulnerabilities rather than relying on surveys
  • Accelerated time to remediation using the technology employed in the assessment
  • Real-time updating of the assessment so that as remediation is conducted, the assessment updates risk scores
  • The technology enables a detailed view of current network rules and protections by analyzing ACLs and configurations of firewalls and switches
• Although network segmentation will certainly be part of an ICS/OT security program, starting with that effort often leads to challenges due to the time it takes to deploy new hardware and the moves of systems required. In addition, many segmentation efforts stall because of a lack of visibility into the endpoints that you actually want to segment, which the assessment provides.
• Network anomaly detection is not usually a good place to start given the cost and time to deploy those solutions in most OT networks with the need for span ports, taps and other infrastructure necessary to provide the level of depth required.
• TEVA provides the foundation necessary for deployment of endpoint protection not only because it identifies all of the assets and their OS versions and software in the environment, but also because it provides the right access to deploy those tools once the technology is enabled.

The tech-enabled vulnerability assessment is defined by a 360-degree view of the most significant threats to the environment are, such as …is the biggest threat the communication at layer1 between two PLCs or between an HMI at level 2 and a PLC at level 1, is the biggest threat that an attacker will reach the IT system and ransomware will spread into OT because it’s connected to IT, is it that a targeted attacker will find a way through the perimeter and run malware or gain privileged access to HMIs and servers to conduct the rest of the campaign, is it that they will use that access to take advantage of a vulnerability in a PLC or controller to cause a DOS or other impact on that device?

It also determines what the organization’s starting point is, whether IT/OT separation exists, is it well configured, do you have significant software vulnerabilities, etc. Then consider what is most timely to execute: How long does a particular project take? What enablers are required to do the first thing?

FAQ about endpoint detection

With this recommendation, we often get the following questions:

Aren’t perimeter network protection and detection going to be the first thing to do in any remediation program?

We are in no way recommending that an organization ignore its perimeter security. In fact, when we use the TEVA, we find that the most important effort is to improve the perimeter network security.  However, the TEVA provides a strategy of the best way to achieve this. For instance, an organization may already have a design that provides for perimeter security, but it is in the configuration of the network protection devices and the evolution of things such as dual NICs, remote access, etc. which degrade the designed security posture.  On the other hand, some organizations (or even sites within an organization) may have no clear process control network and their OT networks connect to various connections of the business network and require comprehensive hardware deployment and movement of system connections to achieve the perimeter protection. A fact-based perspective on the actual devices, how they connect, and the rules in the various network devices is critical to establishing the roadmap.

Since we can’t patch regularly and can’t use modern EDR tools because vendors don’t approve them, isn’t the best place to start always network anomaly detection since we really can’t “protect” our endpoints?

To paraphrase Mark Twain – the report of the death of endpoint protection in OT has been grossly exaggerated. The reality is there is a significant amount of endpoint protection possible in OT/ICS devices. Moreover, many of these can be accomplished more efficiently than trying to deploy the necessary infrastructure (spans/taps/collectors/etc.) to conduct packet analysis for network anomaly detection. In most OT/ICS environments, our 360-degree TEVA analysis discovers hundreds or thousands of software vulnerabilities, missing patches, users and accounts that are insecurely managed, insecure or unnecessary software, insecure configuration settings, weak firewall ACLs, out-of-date anti-virus signatures, etc.

Each of these offers ways to rapidly improve the security of the environment, with no need for additional hardware or infrastructure. Patching, for instance, is NOT impossible. Many of these patches are already approved by vendors and in many cases, an “80-20” rule can be applied whereby by applying just a handful of patches, the organization can address 80% of their vulnerabilities. Similarly, by focusing on those more critical devices – domain controllers, servers, HMIs etc., a significant improvement in overall risk is possible. Further, by eliminating dormant or insecure accounts, changing default passwords, eliminating software that shouldn’t be on OT systems such as TeamViewer, hardening configuration settings, or even just tightening the ACLs in the existing network infrastructure, organizations can make rapid progress in securing their endpoints.

Many OEM vendors have approved Anti-virus solutions and, in most cases, the updated signatures are regularly approved. Furthermore, application whitelisting is a very effective form of OT/ICS endpoint protection that requires no signature updates. In fact, CISA has recommended in prior releases that effectively deployed whitelisting is the #1 protection for OT systems.

Why use a technology-enabled vulnerability assessment rather than conduct a traditional assessment which would require no technology deployment?

Although we understand the attractiveness of a manual/survey/questionnaire-based approach because it sounds less intrusive, the results are much less effective and efficient than when leveraging technology. TEVA advantages include:

1. Lower cost by using technology rather than human resources with all the attendant travel, etc.
2. Real-time assessment data. A manual assessment is only as good as its data which decays quickly as changes happen. Further, this allows the organization to track as the OT/ICS system goes from “Green” to “red’ as remediation is implemented.
3. Accurate, fact-based risk information. In most cases, we have found that survey or questionnaire-based data is incomplete at best and completely inaccurate at worst.
4. Ability to immediately take remediating actions rather than waiting months after the assessment to deploy the solutions necessary to remediate anything discovered.

What makes Verve an expert?

The final question that often arises is “why should we use Verve”?

Our 30 years of experience means that our cybersecurity technology is OT/ICS-specific and has been proven to work in every different type of control system environment. It means that our services team has deep OT/ICS expertise to help ensure alignment between IT and OT, but also provide practical efficient results. It means that our clients deliver security rapidly at a lower cost than by using other providers or technologies.

There are three key differences to Verve’s approach:

Lower Total Cost of Security Ownership

Cost to security has three components: cost to prioritize risks, cost to remediate those risks, and cost to maintain the security over time. Verve’s approach delivers lower costs at each stage:

• Lower cost to prioritize risks: By using the Technology-Enabled Vulnerability Assessment, Verve reduces the usual costs of gathering data and perspectives on risks in distributed OT environments. It eliminates labor, travel, and data gathering by using technology that can capture that data directly. The technology itself is also the lowest cost to deploy as it requires no infrastructure – span ports, taps, collectors, cabling, etc. – that the network-anomaly tools require. Finally, because Verve captures the 360-degree view of risks, the organization can use technology to automate risk prioritization.
• Lower cost to remediate: Verve’s technology enables the remediation of many of the critical risks found during the assessment – from patching and vulnerability management to configuration hardening, locking down whitelists, user and account management, and software reduction, etc. Further, Verve’s 360-degree risk prioritization significantly reduces the cost of remediation by focusing the organization on the most critical risks – all the way down to the endpoint level. Finally, because of Verve’s integrated services capability, the cost to remediate things such as network segmentation, lack of backups, etc. can be accomplished with a single team, reducing the cost of labor in the process.
• Lower cost to maintain security: Verve’s Think Global: Act Local architecture allows organizations to reduce the cost of security maintenance by 70%. By aggregating all of the security data – patching, configuration, software, backups, anti-virus, threat detection, etc. – into a single enterprise database, the organization can reduce the headcount necessary to continually monitor and prioritize new risks. Further, Verve’s “act local” approach to remediation allows for central playbook development, patch review, etc., but ensures “local” – i.e. by ICS/DCS/OT engineers – control over the actions necessary to conduct that ongoing maintenance.

Improved ability to stop an attack

The primary alternative to Verve’s approach is the network anomaly detection tools on the market. Although these tools provide a specific security use case – i.e. detection of anomalous communication within the OT network – they do not provide the best PROTECTION to stop an attack. Verve’s solution approach not only provides protection at the network and endpoint level, but also detection at key points in the attack sequence to create responses using Verve to rapidly deter the attack with the “least disruptive response”. Protection and Detection, together at both the network and endpoint level:

• Network: At the network level, Verve’s software analyzes the configurations and ACLs of critical network infrastructure to identify potential attack paths that need to be closed. This is very different from just monitoring the traffic itself. Network-based anomaly detection tools that only look at the traffic itself, may miss significant potential threats lying in the ACLs and configurations of those switches and firewalls. In addition, its visibility into the endpoints and networks enables organizations to develop the right rule sets and design for further network segmentation. It also monitors network traffic for active threats at the perimeter as well as by establishing virtual sub-segments using the latest Verve threat intel to enable rapid response to active threats.  Finally, Verve’s integrated services teams go beyond just software to provide network segmentation support to redesign and deploy network separation and segmentation.
• Endpoint: Verve is unique in its ability to conduct integrated endpoint management and protection in OT. Through the use of our proven endpoint technology, Verve hardens endpoints across various types of OEM systems – from patching to configuration hardening to software and user/account management. Verve gets directly to the endpoints – SAFELY with no need for risky scanning – to gather data as well as to provide ICS-controlled remediation actions. This technology approach delivers robust endpoint protection as well as ongoing detection of threats at the endpoint, rather than only relying on network traffic as a data source.

OT/ICS expertise to ensure safety AND security

ICS/OT security requires that our first mission, like doctors, is to do no harm. Any security solution needs to start with the foundation of operational resilience. Verve has been designing control systems across a range of industries for 30 years. As a result, our team of experts brings a deep understanding of these systems as well as the security knowledge necessary to protect them.

To a CISO trying to create IT-like security in OT, the challenge can be daunting. Scanning systems can cause them to crash. Deploying modern EDR creates challenges with cloud connectivity and false positives that can stop production. Control systems engineers are rightfully cautious to employ new software on a system that needs to run 24X7 on technology from the last century.

Verve’s team of OT/ICS security experts helps bridge this IT-OT divide, both with our technology that enables IT-like security in OT, safely, as well as our services that can be a single partner to help bring the organization through the various challenges of getting IT and OT to understand each others’ needs and challenges.

Our services team helps companies conduct assessments, design and deploy network segmentation strategies, deploy appropriate backup and restoration, ensure systems are patched appropriately, hardened, etc. We strongly believe that protecting OT/ICS environments requires a combination of technology and services that work together to achieve measurable results for our clients.