How to Close the OT Cyber Security Talent Gap

The shortage of people with the skills and knowledge of OT cyber security is a huge challenge for the cyber security industry. In many cases, we hear that it is not budget that holds organizations back, but the ability to find people to fill the slots they need to achieve the objectives they’ve set. One would think that the COVID-19 pandemic and ensuing economic crisis would have reduced the challenge, but it hasn’t.

According to Cyberseek.org, the database established by NIST’s National Initiative for Cybersecurity Education (NICE) to track the number of open cyber security jobs, the gap continues to widen even during the downturn. Some have argued that the challenge is not in skilled resources, but in the salaries organizations are willing to pay for the talent.

Certainly, higher wages could help reduce the gap over time as people shift from other careers into cyber security, but higher wages alone do not close skill and knowledge gaps. This talent shortage is even greater in OT security, where individuals must understand cyber security, as well as the impact on sensitive control systems.

So, where should an organization look to find this unique talent? How should it go about building the right skill set necessary to protect its control systems?

We begin with the question of “what skills are most needed?” As a cyber security industry we tend to focus on the more advanced and analytical skills and roles such as threat hunters, advanced data analysts, SOC analysts, and architects.

However, according to NIST’s Cyberseek database, approximately 60% of cyber security-related jobs are in operations and maintenance and secure provisioning – things like patch, account, admin, and configuration management. This group is approximately 300,000 of the ~500,000 cybersecurity jobs.

Successfully and safely executing these tasks requires an understanding of the operations processes, OEM architectures, and regulatory requirements. If one includes the foundational elements of “protect and defend,” which includes vulnerability management and Antivirus and whitelisting management, the “management” tasks reach over ¾ of all jobs.

So, as we think about closing the talent gap, it is these skills which should be top of mind. This is not to say the more “advanced” skills of threat hunting, etc. are not important. But the biggest gaps are in foundational “systems security management” tasks.

In OT, this talent gap is even more pronounced as many of the foundational elements of Systems Management are not followed today in OT. Conducting these activities on sensitive OT systems is potentially operationally risky.

We all have stories of IT patching systems and taking plants offline. In the pharmaceutical industry, any changes to the manufacturing process may require revalidation of systems, so it needs to be closely managed by engineering and quality teams.

It is not impossible for an IT person to learn, nor that it will be a breeze to teach I&C techs about Windows patching or appropriate secure configuration management. However, these management, operations and provisioning tasks do not require the same level of cyber security training as threat-hunting or evaluation of attacks, while trying to impart the knowledge of 25 years of industrial operations experience about how the OEM systems are architected and how configurations are designed to an IT staff member through classroom training and some role-playing will be more difficult than we imagine.

For over 25 years, Verve has supported industrial customers in these types of OT systems management functions leveraging the Verve Security Center to automate and reduce necessary labor requirements. We have found that the best ways to find and develop this talent includes:

• Leverage internal IT resources who have depth in foundational elements of vulnerability management, configuration hardening, etc. First, by sheer numbers, there are more IT workers than there are industrial engineers and technicians, by a factor of 5-10x (depending on how each is specifically defined in BLS). Second, the skills needed to operate and manage IT and OT HMIs, switches/routers/firewalls, and other computing equipment are similar. Third, the functional requirements (i.e., understanding correlations, how to use the latest analysis tools like Splunk, how to define patch requirements, etc.) of cyber security are similar between IT and OT, even if the specific threats or incident response actions might be different.
• Tap into this IT resource pool by centralizing the analysis of cyber risks leveraging OEM-vendor-agnostic technology so that the organization does not need to build all of these cyber security expertise areas in each plant or site.
• Integrate OT experts into this central team. Although general cyber security knowledge such as vulnerability management is key, how to address those items within the OT context requires people who understand what is feasible and operationally safe within the OT environment. This blending also enables cross-learning over time.
• Invest in training of site/plant level OT resources in key OT Systems Management functions. Here, specifically, we are calling out the management tasks identified above such as patching, configuration hardening, etc. It is key to the safe deployment of these security actions that local OT resources are involved and understand the management tasks taken on those systems. This training should also include key incident response activities.
• Leverage technology that enables these local teams to automate the actions that they will take across vendor systems to reduce the labor burden on them. One of the key challenges we often see is the dependence on OEM vendors for this management function which places the risks in the hands of third parties…and in most cases, multiple third parties as most plants have multiple OEM vendor equipment.

Some have asked the question: I have a corporate SOC, so should we rely on that group for security operations and analysis/alerting?  We absolutely believe in the importance of an aggregated view of threats across IT and OT.

However, once an alert is identified in OT, OT expertise is usually necessary. Based on statistics from a range of sources (Ponemon, Fireye, Advanced Threat Analytics, IDC, Bromium), there is a 50% chance that the alert is a false positive. We have many experiences where a corporate SOC has called a plant and raised a flag on an incident, which has required a local I&C tech to research the issue, only to find that an operational change which the SOC was not aware of created a false alarm.

If the alert is in fact a true threat, the incident response to that threat will usually require local knowledge about what is happening in the plant at that moment, what processes can be stopped, what risk to quality or production will result. While we would all like detailed incident response plans for every type of scenario, we also know they don’t exist. Decisions and trade-offs are made in the moment. A local operations resource – who also understands the needs of cyber security – will be a key participant in this effort.

We cannot be narrow-focused in our search for cyber security talent. We absolutely need to recruit from the huge pool of IT talent and find ways to introduce them to the complexities of industrial operations. But, we also need to transform yesterday’s I&C techs and engineers into OT and OT cyber security talent.

Finally, we need to expand our ability to attract a new, younger generation into the industrial technology world. The debate of whether to  start with IT or OT/Operations talent is a false dichotomy. We need them both – and we need to think about how we train and develop each group differently for the role that they will likely need to play.

And perhaps one resultant benefit of this approach is that we provide roles, jobs and futures for many of the communities that have been hardest hit by America’s industrial shift over the past thirty years. These communities have schools, resources, and a passion for manufacturing and industry.

If we develop the right approach to OT and OT security, we can develop tech jobs where we might least expect them, and sustain these communities in the future.