OT Systems Management Whitepaper

What is OT Systems Management (OTSM)?

Over the past decade, IT Service Management (ITSM) leveraging CoBit, ITIL (IT Infrastructure Library) and other standards have become a proven, rigorous process in most large enterprises. The basic components of ITSM include designing, planning, operating, and controlling IT services provided to users.

In practice, this includes the way hardware is configured, patched, managed, and deployed, the way software is developed and deployed, and the way teams respond to incidents.

These practices are critical to improving overall IT cyber security posture. In fact, according to the National Initiative for Cyber Education’s (NICE) Cyberseek database, over 50% of job openings in cyber security are related to operations, maintenance, and provisioning – jobs often contained within ITSM functions.

Moving away from traditional IT realms of PCs, laptops, cloud-based servers, and mobile devices into the world of Operating Technology (OT) found in manufacturing plants, the power grid, building controls, and other cyber-physical systems, the role IT plays in managing these devices is less clear.

OT is often controlled by manufacturing engineers, process control engineers, or instrumentation and controls technicians. The systems are critical in functioning physical processes, such as power generation and transmission or manufacturing production. These OT systems are built to last ten to twenty years, as opposed to the five-year lifecycles of traditional IT equipment.

OT systems contain many embedded devices running on firmware developed by specific OEMs that are not built with open management interfaces. Programming these systems often means accessing OEM-specific tools to update or reconfigure them.

Over the past decade – and in the decade to come – corporations connected OT systems into the traditional enterprise IT infrastructure to drive greater efficiency and effectiveness of operational processes.

Greater connectivity offers the hope of leveraging the cloud for advanced predictive maintenance analysis, improved operational efficiency by adjusting control parameters, and more efficient use of labor by managing sites through centralized, remote access. These financial drivers are significant, so much so that this idea has been branded as Industry 4.0.

As these systems connect, security becomes a much greater issue. Systems formerly “air-gapped” or “islanded” from enterprise IT and its access to the internet and communication applications, such as email and cloud interfaces, are accessing the enterprise infrastructure to take advantage of scale and the power of big data analytics.

But with this added benefit comes risks from IT networks: ransomware, hacking for espionage, and potential disruption of physical processes to cause physical damage.

Chief Information Security Officers (CISOs) and Chief Information Officers (CIOs) are asked by their boards of directors to secure these systems to ensure they have the same level of security as the rest of the devices within the enterprise. As a result, IT leaders want to increase the integration of IT and OT in driving standardized cyber security across all endpoints and networks.

However, in most organizations, OT assets such as HMIs, servers, PLCs, relays, RTACs, and other intelligent electronic devices are excluded from ITSM processes for a variety of reasons.

From organizational boundaries to lack of skills of IT personnel on OT systems to regulatory requirements, ITSM practices do not extend to these systems. Further, OT staffs are already under headcount pressure to increase efficiency.

Many foundational elements of cyber security are not present in OT. Inventories are not accurate, configuration and patch databases are out of date, and account and user access management are poorly executed. In part, this is because tools have not been available to automate these processes given the sensitive, unique, and embedded nature of OT assets.

This gap means IT OT convergence or integration in cyber security has no foundation to build on. The C-suite and board of directors are left in the dark about real risks on cyber-physical systems because IT leaders (CISOs and CIOs) cannot measure progress or risks the same way OT can due to the weak foundation security tools and processes are deployed from.

For instance, if a company deploys a detection platform into an OT environment without foundational elements, it is never certain whether all hardware and software assets are accounted for. They cannot effectively protect the systems because of a lack of active device management.

It would be like installing locks on all your doors but ignoring the dozens of windows in the house that do not have locks and are easily accessible on the first floor.

To develop robust OT cyber security roadmaps and foundations, organizations with OT systems (everything from manufacturing process controls to building control systems to security access systems) should embrace the concept of OTSM, paralleling their ITSM practices, but within the unique environments of operating systems.

Achieving a mature level of OTSM is critical to improve overall ROI from increasingly connected industrial systems and to ensure foundational elements of OT cyber security are in place to protect critical infrastructure from targeted and untargeted attacks.

5 Benefits of a Robust OT Systems Management Program

• Insight into all hardware and software in the network to ensure vulnerabilities are identified quickly
• Properly updated and configured systems to reduce opportunities for cyberattacks
• Operationally efficient systems update to provide automation on key operational tasks
• Consistent reporting and monitoring across IT and OT for simplified progress documentation
• Effective advanced security controls built with proper visibility and access to the underlying endpoints and network data

Systems and Security Management is Critical for Cyber Security and Reliability

Rigorous systems management is a foundational element to ensure secure and reliable systems. With almost every major cyber incident, the analysis report calls out the importance of maintaining updated patches, secure configurations, limited access and privileges, and updating antivirus signatures.

None of these grab headlines quite like the advanced threat hunters and analysts who dig deep to identify how hackers made their way into the systems and exfiltrated data. However, they are foundational elements that make cyber security much more effective.

NICE focused on cyber security workforce development, breaking U.S. cyber security job openings into seven types: Operations & Maintenance, Provisioning, Protect & Defend, Analyze, Oversee & Govern, Collect & Operate, and Investigate.

Of the 1,115,000 cyber security job openings as of April 2021, half fall into the first two categories, which largely consist of roles closely aligned with ITSM. 14% are in Protect & Defend, including management of infrastructure hardware, software, and vulnerability management, which are also closely aligned with key ITSM categories.

These workers and the processes they manage are the backbones of cyber security. They ensure systems are provisioned for security when moved into production. They monitor changes to configurations that do not align with secure baselines. They confirm passwords meet organizational standards. They monitor and deploy software patches necessary to maintain the security of systems in the field.

This is not intended to understate the importance of other roles in analyzing or investigating. We tend to overlook this fundamental practice of reducing attack surfaces, keeping up good cyber hygiene, and executing the most important asset-level protective functions.

ITSM Often Does Not Extend to OT Security

In most organizations, the procedures, policies, and service agreements managing IT systems do not extend to the Operating Technology environment. This results in functions normally associated with ITSM (asset inventory, provisioning management, patch management, configuration management, disaster recovery, and incident response) to either be unmanaged or applied at a local or business unit level without the same level of rigor, process, or consistency you would see in IT.

This is not a blanket statement. In some organizations, IT absorbs the OT function and employs similar systems management across both environments with the necessary customization for OT requirements. In other organizations, robust OT Systems Management is deployed as a result of the regulatory compliance requirements, such as medium and high impact assets within the NERC world. Overall, an ad hoc approach to OTSM seems to be the most common.

Ad hoc programs often delegate responsibilities to either an instrument and controls technician who has tuned the DCS in the past or to a plant IT representative or chemical engineer running the manufacturing system. In most cases, these individuals were not trained in systems management or on the IT equipment at all.

Most follow processes developed by operations engineering or locally for an individual plant, hospital, or facility. They do not leverage the same toolkits as their IT counterparts, due to the difficulty or risk associated with deploying and accessing IT tools with the OT environment.

These same individuals are usually asked to pick up new tasks in conjunction with their day jobs – build an inventory, keep it up-to-date, patch systems on a regular basis, ensure password policies are enforced, confirm firewall rules are properly configured and don’t trip the plant in the process.

Launching a New Discipline Called “OTSM” 

If an organization integrates IT and OT, there will still be a need to customize the policies, processes, tools, and the team responsible for the sensitivity of the OT environment. This is a new discipline called OTSM.

4 Elements to Develop a Robust OT Systems Management Process

Establish policies and procedures that match the specific OT environment for the organization.

The great news is most organizations have a base of IT policy and procedure templates to draw from, such as SANS or NIST. The key is taking those guidelines and building the specific elements necessary for the unique OT environments.

For instance, in a pharmaceutical company, the patch management policy for a production line may differ significantly from the R&D lab where the product is tested in small batches. Procedures for configuration changes must reflect the different regulatory structures within each industry and geography.

Additionally, considerations need to be made for DCS vs. SCADA deployments. The geographic proximity between the team, tools, and assets in scope makes for very different dynamics in the execution of OTSM functions.

Develop your talent and workforce. In most cases, the personnel responsible for OTSM will be techs and local IT staff.

In IT, most systems management functions are centralized and executed remotely. With the growth of the cloud, this becomes even more true.

In many OT environments, the actioning of systems management requires local resources (or at least local oversight) for patching, configuration setting changes, and incident response.

The downside risk of a patch deployment taking a machine, and therefore the plant process, offline is too great to do remotely. Similarly, a false alarm in a manufacturing facility is significant, and in most cases, incident response requires a local, or at least an OT-trained staff member, to evaluate potential risk and remediation steps.

As a result, workforce development around key OTSM concepts such as patching, configuration management, and password management is necessary. We applaud the significant training available around cyber security analysis, investigation, and threat hunting, but at least that amount of focus should be placed on the other half of cyber security – the foundational elements of Systems Management.

Identify relevant tools and automation.

To date, the prevailing approach to OT cyber has been passive tools that can only gather data from the wire and manual processes using multiple OEM tools to manage the assets. Not surprisingly, the lack of real visibility to the underlying endpoint information and lack of automation – both of which are available in IT – make this difficult in OT.

What is needed are OT-specific tools that provide the same functionality IT teams are familiar with. This is vital for effective management and reporting functionality that is necessary to ensure all levels of the organization are accountable for security.

These tools are unlikely to be the same used by IT, which were built for traditional and emerging IT devices. These tools consider the unique characteristics of OT devices. But for true IT OT security integration, they need to provide similar automation and informational capabilities as their IT relatives.

Align leadership on priorities.

OTSM requires a significant change effort. Traditionally, industrial control systems have been long-term capital investments that last fifteen to twenty years between major upgrades.

OTSM requires regular management: updating, configuration management, access management, and vulnerability management. In many cases, this requires changing mindsets and behaviors of team members and the more functional and procedure requirements. Senior leadership is key to making effective changes within already stretched operational organizations.

We have seen several companies adopt this OTSM approach to manage and secure their OT assets and networks.

Power Utility Case Study

This North American company operates dozens of facilities across a wide geographic range. The CEO established an objective that IT and OT would achieve the same cyber security standards.

The mandate was to find ways to build systems management foundations across IT and OT. So, instead of saying ITSM and OTSM, they refer to it as SM or Systems Management. IT was well ahead of OT when they started their journey in 2017.

There was no accurate inventory of hardware or software. They had no visibility into the configurations, user accounts, password settings, or backup statuses across their facilities. Where they did manage devices, they did so in an ad hoc way with manual processes and dozens of OEM toolkits.

Over a two-year period, the utility company built an OTSM process to mirror their ITSM process, using a range of standards from ISO to CIS to establish basic objectives and scoring mechanisms to track progress. This created clear executive alignment on objectives and processes to manage different environments.

They deployed tools and automation relevant for the environments and trained resources to assess, manage and remediate endpoints across their systems.

As a result, the utility company significantly increased its cyber security maturity in both IT and OT. They built a strong foundation for additional controls and operational consistency. They effectively communicate their progress and status across all networks to the C-suite and board of directors, and they measure traction against security improvement on a regular basis.

Establishing OT Systems Management with Verve Industrial

Success in OT cyber security and reliability requires a new foundation in OTSM. Systems management is a critical element to ensure connected systems are protected and managed appropriately.

OT must take a page from the IT playbook and deploy processes, tools, and training that enable the 50-60% of cyber security functions that are foundational in systems management. Without it, IT OT integration and true OT security will be difficult, if not impossible, to achieve.

Verve Industrial Protection has worked with many clients seeking OTSM capabilities in our twenty-five years of business. Leveraging the Verve Security Center (VSC), a true endpoint management platform built for OT, with the support of our OT expertise, these companies doubled their cyber security maturity with measured results to track progress.

The vendor-agnostic VSC allows OTSM to be possible and efficient through automation and visibility. Its closed-loop approach provides visibility and tracking of assets and vulnerabilities, as well as actionability to manage patches configurations, user accounts, etc. from a single platform.