The Ultimate Guide to Reading ICS Cyber Security Advisories Like A Pro

While people think traditionally about cyber security as being hackers in hoodies, ransomware, complicated, or something foreign altogether – it does not have to be. Fortunately, cyber security can be approachable and reasonably easy to consume. But above all, nearly everyone plays a part in securing organizations and reducing their risks.  So, we are here to help break those barriers!

Regularly, Verve and many others that support asset owners hear a variety of common questions regarding CVEs or product cybersecurity advisories.  Some of these include:

• What does cybersecurity have to do with process control and OT?
• What are cybersecurity advisories & vulnerability disclosures?
• What is the difference between erratum and vulnerability?
• What level of cybersecurity knowledge is needed for review?
• What is a CVSS score, and how is a score or CVE applicable to my assets?
• What are the weaknesses with CVSS, CPEs, CWEs, and CVEs?
• And what is the approach to understanding advisories & mitigating any identified risks?

Without any further delay – let us dive in while keeping to what you need to know.

What does cyber security have to do with process control and OT?

Cyber security generally is thought of as hackers, credit card breaches, protecting passwords, applying updates, avoiding phishing scams, and encryption for data or its transmission.  This is a common viewpoint, and while enterprise or Informational Technology (IT) in nature, much of it will overlap with systems or technology used in Operational Technology (OT) environments and industrial control systems (ICS).

In the IT world, most of the models rely on what is called the “CIA triad”, which is short for:

• Confidentiality – protecting data from unauthorized viewing.
• Integrity – ensuring data is as expected and without tampering.
• Availability – ensuring systems, applications, infrastructure, and data are available when needed.

It is debatable on whether keeping liquid in pipes flowing, ensuring electricity is being generated, or ABC product is being manufactured, but clearly, of the three components in the CIA triad – only two really apply: availability of systems, and the assurance that they are running as expected (e.g., integrity). And so, the industrial world focuses on a different paradigm – SRP:

• Safety – ensuring safety for humans and the environment.
• Reliability – ensuring processes and systems are operating consistently & as expected.
• Productivity – ensuring assets, processes, and persons are completing their activities as expected without disruption or downtime.

Now whether IT systems such as Windows, or Cisco routers are IT, OT or IT/OT convergent is out of the question.  These are systems that provide critical functionality or host it, in industrial environments.

Therefore, any emails answered in these environments, files downloaded (including device manuals, or engineering workstation (EWS) application installers), changes made (patches, updates, configuration alterations), backups, remote connectivity (VPNs), logging & alarm management – any of these can affect the security of the organization, and if improperly secured, the malicious activity can take advantages of any flaws – putting asset owners at risk by obscuring operator visibility, compromising the integrity of systems or devices, disrupt/halt production, and in the worse case, result in a Human Safety Environment (HSE) event.

Cyber security in OT for asset owners is about maintaining SRP and keeping the business feasible while managing risks appropriately.  It is not just a “geek” thing and can traverse all aspects of a business including how to manage Operator workstation accounts, their passwords, limiting access to ICS devices, maintaining assets, programming devices, managing alarms, and governance (policies, procedures, failure recovery, and more).

What are cyber security advisories and vulnerability disclosures?

Regardless of the word used – a vulnerability is a flaw or a condition, that if realized can result in something negative. 

Most consumers see a flaw as something like an issue resulting in a product recall, an update to your car’s computer at your next dealership visit, or a “bug” that requires a computer to reboot.  All those examples use different terminology, but they are conditions that we want to carefully avoid or weigh the pros and cons for.

Information is not necessarily a bad thing, and when it comes to safety – advisories are not a joke.  Unfortunately, advisories and cyber security are not written in plain language, and not all of us have adequate training in this field, but it all stems from some core concepts that are probably “good enough”:

• A vulnerability is a type of flaw that if exploited/executed upon whether by accident or intentionally, will result (potentially) in X occurring.
• A vulnerability has a vector or a point of entry and may only affect specific components. Every vulnerability may be different from another, and the consequences, or how it is exploited may also differ.
• A vulnerability is a flaw that is measured (scored) in criticality using several criteria. It is a metric and relies on the Operator to account for other considerations that raise or lower its severity or determine its applicability to their environment.

In other words, a vulnerability is an issue that if disclosed, will be referred to as a Common Vulnerabilities and Exposures (CVE). It will be assigned a label, products affected, high-level descriptions of the flaw(s), contain information necessary for initial analysis, and potentially, remediation steps.

Usually, CVE records have minimal information that would be relevant to an end-user but are an identifier that points to information that your security products use (e.g., vulnerability scanning).

What an operator or non-security layperson needs to know is that CVEs are often grouped with supporting information into a report called an “advisory”.  Advisories may also contain:

• High-level risk assumptions and a score (e.g., is this critical?).
• More than one CVE (and a snippet of related information).
• List affected products and versions.
• Other affected vendors/products
• Information about the researcher and/or reporters.
• Contact information & remediation/fixes where possible.

Generally, advisories are complete summaries and refer to a single product, however, cases may exist where the advisory is referring to a component or product that is used by other vendors.  In this situation, these types of advisories are harder to determine if they are applicable to deployed assets.

Advisories can be found in several places, some are public, and some are not, but assuming your organization has a detailed inventory of its assets already.  It is best that an operator or site owner is monitoring CISA’S ICS-CERT and vendor security portals.

Please note that depending on your industry – there may be “closed-door” advisory portals or websites, and these may require access credentials or membership.

What is the difference between erratum and vulnerability?

Erratum, vulnerability, anomaly, and flaw?  What is the difference?  Well not very much at a superficial level, however, it really depends on context, and the asset owner ideally is in the best position possible – they know their environment, and how they protect their assets.  Unfortunately, we humans do not always think the same as malicious actors as our adversaries do.

On one hand, engineers might know of erratum as hardware related flaws or idiosyncrasies in software that need to be worked around.  And on the other, technicians may note known issues in release notes or also known as anomalies, and those too are to be avoided.  However, in both cases, they might not be labeled as security-related, and this becomes a communication and syntactical problem.

Therefore, a vulnerability from an OT security standpoint is any issue that affects the security of a device or the process it controls/contributes to.  “Bad people” can think of any number of ways to make a device or application misbehave, and so generally, any issue that can affect SRP – is a cyber security vulnerability, but you – the asset owner – will need to determine the level of tolerable risk; even if it is a known issue or a setting that will allow someone to nonsensically change a devices IP address to 127.0.0.1.  In other words, anything that can affect SRP – can be a vulnerability.

What cyber security knowledge is needed to review an advisory?

Anyone can become “comfortable when being uncomfortable”, and nearly anyone can be proficient or play a fundamental role in an organization’s cybersecurity.  As said earlier, everyone ranging from a contractor replacing valves, to an Operator in a control room are critical – even if they are not certified, well-studied in cybersecurity, or even “their job”.  It is just a part of it.

However, cybersecurity should not be a burden, and it can be a learning experience.  Ideally, if an organization is providing adequate training, and implementing a concrete cyber security program – technology would do everything, but that is not the case.  It still needs a human in the loop.

To answer the level of knowledge required to assess a cybersecurity advisory, or a CVE to determine risk – it really depends on your role in the organization.  However, there is a myth in cybersecurity where everyone must be a trained expert.  Expert-level knowledge is not required (but recommended) for all tasks or employee roles, and a rote learning approach can be highly successful in increasing an organization’s security.

For example, I do not need to know how SSL/TLS works, but I do need to know that it provides encrypted network traffic so my secrets are safe, relies on an algorithm or cipher for security to make network traffic look like “gibberish”, that credentials may be used in or with TLS to assure allowed logins, and that these should be kept safe.

Or another example is:

An ICS device is like my laptop, it has an operating system (OS) like Windows or Linux, but it’s specialized to operate in OT environments. The device probably communicates using XYZ protocol, has USB/serial ports, and can be configured.  I should protect those because this device might have vulnerabilities inside of it, and I want to prevent anything from affecting SRP.

These examples are common scenarios and should not require sheer “cyber skill” or imagination to dissect.  Advisories on the other hand might require additional info, but core concepts can be linked together via “chaining” – e.g., for example, persons with memory impairments or students wishing to study & recall information faster will often use “buckets” and “associations” to link items together.  A cake means a celebration or a treat, so whose birthday or event are we celebrating?  Will there be candles because there is a cake?  You can see that this works well for initial range associations and can be sufficient in many real-life cases.

As part of that rote learning process, I recommend developing:

• A vocabulary or mental list of what an organization or site may have. If I know my operators use X SCADA software to manage alarms, and this software is on several PCs or servers and talks to these devices – then I have some understanding of the devices or technologies in use.  Once you have seen a PLC, DCS, SIS, EWS, Historian, HMI, programming software, router, or switch – you basically know how they fundamentally work, and therefore, conceptually can imagine their interfaces, configuration, and function.  The implementation will change by the vendor or by technological era, but the core pieces remain the same.
• A model of how the technology works, and what is required to some extent at the most basic level to operate it. At first, you might say wow, I do not know.  But truthfully you do – a car with a conventional combustion engine needs fuel, a battery to start/spark, and keys.  Assuming there is not a failure – you already know some basic inputs, and these inputs can be translated to even a light switch in a home.  To turn on the lights, you would need access to the home, electricity, and a working light switch.
• High-level mental data flow models. Even if security might be “out of your wheelhouse” – storyboards and stick-figure/doodles with lines linking elements, or even arrows pointing to notations such as “Bob or Jane logs in here with their security token/card”, then “an authorized user will download logic from a device using X application”.  There will be nuances, but, with a bit of patience, some note-taking, and Googling – a little work can make the learning curve far more tolerable (and maybe enjoyable).
• A process of creating building blocks and strengthening them – just as you would nearly anything else in life, practice makes perfect, toolsheds/garages will grow, and new knowledge leads to new understandings. An electrician could plumb if they wanted to, and a carpenter can do wiring, tiling, and plumbing (albeit with certification), but to my point – many do-it-yourself (DIY) enthusiasts do this nearly every day and you can find them at Home Depot or on YouTube.

It might be intuitive, or common sense, but security is multi-discipline and multi-domain.  This is one approach that I have seen results with, but others may work as well or better – find one that works for you.

Another piece in this last part is understanding terms and filling in those terminology/concept buckets. Nobody should be expected to know exactly how a buffer overflow functions, or to even have hands-on experience, but it is helpful. Regardless, the following figure highlights some of the buckets.

If someone were to look at each of those sample advisories, being overwhelmed might be a fair response, but the key is to look for terms and remember associations.  For example, if one says ICS protocol affected, this means network access to the device needs to be considered, that abuse might result in a shutdown, and this should queue a discussion.

Others might be relating to configurable vulnerable interfaces: Telnet, DHCP, FTP, and HTTP.  Similar thought processes should be applied, but through “bucketing” – nobody should need to be an expert so here is a brief tabular analysis to get someone started:

Review of advisory issues for 2020 and their “buckets”

Synthesizing and building lists/matrixes of security issues seen in ICS/OT assets or systems can make quick work of complicated terms, provide the path forward for others, and create material to be re-used for training or organizational standardization.

What is a CVSS score, and how is a score or CVE applicable to my assets?

TO re-iterate, a CVE record (vulnerability) has various attributes surrounding the issue being disclosed, and one of those attributes – is a score.  The score or Common Vulnerability Scoring System (CVSS) is a value calculated using several criteria to provide a metric on the severity of the flaw.

It is a scale between 0-10, but it has had variations due to standard changes.  The important thing to note though is that scores are:

• A combination of several factors including skill required to exploit, whether an attack is remotely exploitable, and if proof of concept/exploit code exists.
• The worst-case and do not account for compensating controls (or things like firewalls in front of a device).
• A guide, and a metric that needs some analysis by the asset owner to determine the true severity and impact on an organization.

To summarize, if all things were equal, the severity score would be a primary factor when making risk-related decisions, and a sole metric for discerning impacts.  Unfortunately, most organizations focus purely on the CVSS score as part of their Key-Performance-Indicators (KPIs) but do not utilize other factors sufficiently, and so they scramble to remediate risks almost solely on these scores when they may or may not be accurate or even applicable in their environments. This can result in needless fatigue, the incorrect expenditure of efforts, and unmitigated risks sneaking under the radar.

This generally works well in IT environments, but in OT environments – patches and vulnerability management often become a risk management activity:

• Is this asset truly at risk?
• Are there compensating controls in place that reduce the risk to tolerable levels?
• Does this risk require immediate treatment or remediation?
• Can remediation be scheduled? And with what urgency?

For an onlooker, this again sounds complicated, but humans make decisions like this on a regular basis: should I drive in the snow? Do I have winter tires?  Do I lower my speed? Etc. Furthermore, to apply a score or to determine if a CVE applies to my assets, an asset owner needs to:

• Determine the presence of potentially affected assets.
• Validate applicability (configuration, versions, logic).
• Determine compensating security controls and protection layers.
• Determine severity, risk exposure, and potential impact.
• Use the original CVSS of the CVEs, and corporate-aligned risk matrices, and determine actions.
• Create a remediation or risk management plan and follow due process for execution.

CVSS scores in OT/ICS need to be taken with a grain of salt, but after reviewing enough advisories and by knowing your environments in-depth – it can be useful to coordinate a response and engage security teams.  There is a lot of other research out there and a few great presentations on this topic are available here and here.

What are the weaknesses with CVSS, CPEs, CWEs, and CVEs?

No system or metric is perfect, and so the CVSS and CVE system is constantly changing and evolving.  The scoring is largely well defined, but some scores may pose controversy in OT.

E.g., why would a CVE that can result in an unauthorized HALT on a Programmable Logic Controller (PLC) get a 6.0 or a 7.0 score, and a webserver running on a device has a 9.0?  Clearly impact and severity are some areas that might need some verification.

For the uninitiated, managing vulnerabilities and reading advisories is already difficult enough – especially when vulnerabilities are labeled with changing Common Weakness Enumeration Specification (CWE) labels or there might be challenges matching affected products with “magical” Common Platform Enumeration (CPE) vector strings.

For the most part, anything relating to an advisory should be reasonably well structured and mostly correct. Using 2020’s ICS-CERT data, Verve discovered approximately less than 1% of the advisories had issues with CPE matching – so it is a safe bet to largely trust supplied information from MITRE, NVD, or the vendors.  Regardless though any usage of a CVE requires the reviewer(s) to apply their own logic, risk matrixes/process, and to use the information to the best of their ability.  In the cases there might be a discrepancy or a concern – asset owners are recommended to speak to their vendors and gain clarification.

After all, the CVE/CVSS and advisory system is both advisory and home to potential Indicators of Risk (IoR); it is not definitive nor absolute.

What is the approach to understanding advisories and mitigating identified risks?

To be fair, the best way to approach reading an advisory is to have adequate knowledge already of your environment and to never make these types of decisions alone.  However, advice is best served with supported practice – so let us walk through a basic process in the following figure.

First, if we obtain an advisory, or get sent an alert (e.g., from a vendor or a CERT), we need to read the assessment briefly and compare the affected products to our inventory.  If we are without an inventory, then we might be able to make do, or we will have to confirm applicability.  The closer the asset is to the Internet, or to a system that has frequent user “churn” – any further delays increase risk of exploitation.

Second, we need to determine if the vulnerability affects the systems at a deeper level, and conduct workarounds and remediations exist such that the organization remains within its threshold?  Most organizations have risk guidance already defined for their IT/enterprise side and may already have similar frameworks built for OT/disaster/incident planning (it may require some rework for cyber though). Many organizations have some degree of compensating controls already – it just needs to be clearly documented and validated for accuracy.

Using the information contained in an advisory such as the one below, and leveraging our organization’s risk management structures & asset inventorying information, yourself or a team can arrive at a basic analysis regarding:

• Whether we have affected assets operating and  how many are deployed
• Nature of the vulnerability, how an attacker may take advantage of it (remotely exploitable, and the skill level required to successfully execute an attack (low)
• Versions and products affected.
• CVEs noted for this advisory (only one with a raw CVSS score of 9.1, and it affects device authorization)
• Component affected (e.g., libssh – a third-party, open-source library)
• Upgrade paths or available firmware updates

Combining the above analysis with the detailed information provided by the vendor and CISAshould let us prepare for change management and other risk management activities in accordance with our organization’s policies. However, before committing to immediately rectify identified issues, we must first be cautious, and confirm if the affected assets are protected & to what level.

For example, if affected systems have a firewall in front of them, and access can be assured to originate only from the secure endpoint, or network, then the urgency, and potential impact/severity of this vulnerability may be degraded to the point where remediation is scheduled (fix later), deferred (fix never).  Otherwise, it may require an urgent or immediate fix (fix now) – subject to the organization’s policies though & SRP.

In other words, advisories, CVEs, and product notices drive risk management activities, but they require well tested and greased infrastructure.  This includes all aspects of people-process-technology (PPT) and being able to act upon alerts vs. defer indefinitely (or until an incident occurs).

Next steps

If you made it this far through the article, I am sure many readers may agree that cybersecurity and managing risk is a complicated topic full of nuances.  This article was written to be a 101-level primer on the subject, but my belief is that it’s not as arcane as commonly believed; humans are resourceful.

Through simplification and deconstruction of “complex” topics through divide and conquer strategies, relating concepts to commonplace items or subconscious daily activities, and vendor/industry relationships – even a layperson can get a grip on vulnerabilities and advisories (even those for ICS/OT).

Unfortunately, understanding advisories is one of the areas that begin to scratch the surface for contemplating risk. But cyber security is more than advisories, and a security professional or even those in operations should possess some degree of familiarity with the following because they can have a positive impact if used correctly:

• Asset inventory and management
• Network security, segmentation and design
• Secure configuration of devices, systems, infrastructure and software
• User access, permissions, roles and management
• Log collection, alarm/alert management, and overall analysis
• Vulnerability (flaw) and patch/update management
• Change controls and related processes
• Endpoint security (for all configurable devices, laptops, servers, workstations, virtual machines, and more)
• Policies, procedures, overall governance, and specialized standard operating procedures
• Awareness training

The good news is – not too long-ago formal degrees and certifications did not exist.  Most people were self-taught or taught on the job, and this is evidence by poor correlation to my point: anyone can “do cyber security” if they put their mind to it, but today – there are a ton of courses available, and free and public information available. 

Verve’s mission is to help our clients ensure the security and reliability of their most critical assets: their industrial control systems. Verve Industrial brings over 25 years of ICS/OT experience or what is possible to bridge the IT OT challenges of securing these environments.

We are a partner to our clients in their security and reliability journey. That journey is not solved with a single project or the one-time sale of a tool.  We walk alongside our clients in helping them increase the maturity of their systems and processes over time.