By Rick Kaun, VP Solutions at Verve Industrial Protection
October, 2021
We hosted a (CS)²AI Online™ seminar on September 22, 2021 that focused on Navigating the New TSA Directive for Pipelines.
Here is a bit about the event:
Navigating the new TSA directive for pipelines (and other future industry targets) – Lessons learned from a regulated industry.
The recent increase in ransomware events coupled with one of the targets being a large pipeline company has compelled the TSA to issue a new cyber security directive. This means many OT organizations are now scrambling (some more or less than others) to stand up a multi-disciplined security program for a very diverse, distributed OT environment. This looks and feels a lot like the Power Industry was confronted with when NERC CIP was first introduced and so we, as security practitioners, can learn a great deal of lessons from an industry that has already run down this path. Challenges in understanding scope, standing up multiple security initiatives, organizational changes for responsibility, maintenance and response activities and most notably day to day maintenance and compliance can be significant obstacles for operating companies to overcome.
Join us to review a number of security learnings around setting up and maintaining an OT security compliance program such as:
• A multi-disciplined approach is key – treating individual security tasks as silos will create gaps, increase effort and decrease efficiency
• Remediation is a key consideration – simply mapping vulnerabilities or enabling perimeter/network monitoring is just a drop in the bucket – need to be able to reduce risk and attack surface as well as react to emerging situations
• Monitoring – as risk is reduced and new threats emerge the current risk status is always in flux. Being able to monitor and report on current status, changes to the threat landscape or show progress/compliance are key components of a sustainable program
• Automation – as many of these tasks and insights that can be automated the better. OT staff is spread too thin and traditional OT risk reduction approaches are far to manual to provide meaningful and consistent risk management
As always, we encourage our audience to participate throughout the event by contributing feedback and questions for our speakers. We weren't able to answer all of your questions, so we have asked some of our speakers to answer a few of them. Below, are some answers to a few questions posed during the event.
Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions.
*******************
QUESTION:
What are the Cyber Risks facing Phasor Measurement Unit (PMU) devices in the power sector and how to mitigate these cyber attacks?
ANSWER:
This is a very specific question that I do not have a very specific answer for other than to suggest that ALL OT/IIOT devices are gaining much more scrutiny from both ‘good’ and ‘bad’ guys. Any class of ‘embedded’ asset (or asset in general) can be a target for anyone looking to do harm. The explosion of IIOT devices and connectivity in general have supersized our technical footprint and therefore our security debt as well. Most IIOT devices rush to market with ‘problem solving’ and ‘ease of access/connectivity’ as primary promises to the consumer. What is a distant (if even present at all) consideration is security. So we see many organizations wanting to stitch data together but do so in a less than secure way. So to answer your specific question – what can be done? Work with your vendor to provide secure by design solutions. Add data (if possible) from these systems and the networking gear connecting them to an SIEM or alerting tool set. Use this feedback to tune your end point deployment, incident response programs and, above all else, security language into your procurement language and projects as well as back to the original vendors.
QUESTION:
Awareness of Zero Trust and Secure Access Service Edge concepts are growing across industries. What parts of the TSA directives can be addressed through Zero Trust/SASE implementation (likely in the 2nd generation approach)?
ANSWER:
Interestingly enough the second TSA directive relative to cyber security explicitly calls out the need for Zero Trust. The specific ask is generic – they say ‘Implement a Zero Trust policy that provides layers of defense to prevent unauthorized execution’ then points to specific details in the directive document. And Zero trust (according to Palo Alto) is “Zero Trust is not about making a system trusted, but instead about eliminating trust.” But what the end user needs to know is that this is a concept that can/should be applied to as many policies, procedures and technologies as possible. Specific examples include network segmentation and firewall rules, remote access technologies and policies (ie, implement 2 factor authentication, minimize applications/permissions on the ‘jump server’, include logging and monitoring of that system, the subnet it is on and the logged in users, etc. This concept needs to be incorporated into every layer of security throughought an entire program for inventory to protection, detection and response/recovery.
QUESTION:
This is a great model and case study. What happens when the OT team and the IT team do not get along? Have seen client environments where the two teams do NOT collaborate = Demark at the air gap. Tips on getting past that situation?
ANSWER:
This is not an ideal situation but is unfortunately all too common. Worst case is you can have a ‘negotiation’ whereby you do a RACI chart or an accountability matrix that shows who gets to (has to) do/decide on what topic. For example, IT can monitor the landscape for Vulnerabilities (ie, what is in the wild) but just for identification. OT then needs to take the baton to map known risks to their assets and come up with a plan. IT can be expected to help understand/navigate the actual risk and possible compensating controls (ie, what can we do if we can’t actually patch?) but then OT needs to roll out the remediation. Ideally this type of forced collaboration *should* lead to a better working relationship over time as both sides get to know one another better. Maybe starting with an exercise to understand the other perspective would work too? We hosted a whole webinar about it here if you want to dig in more. Reality though is you need organizational support. Meaning the senior management needs to identify, facilitate and, if necessary, support a collaborative environment. We wrote about that recently too since so many organizations are currently examining their manual/patch work efforts and wondering how to plan a better solution.
QUESTION:
Any specific thoughts for medium to smaller companies who may not have nearly the budget of enterprise sized companies? It can be a bit daunting on smaller budgets.
Why do you state that moving IT to an OT security role will not work?
ANSWER:
There are two questions in here! First – small and medium companies can absolutely participate without enterprise wide budgets. Though I must put the obligatory plug for educating the finance department that our mission is to ensure the safe, reliable, expected operation of the facility. The very facility that makes the company money. We can get money for spare equipment (sometimes very expensive equipment) because it relates to uptime. Why can’t we see the same correlation to cyber? Anyways. Small/medium companies can be smart and build automated tools into a multi-phased approach over multiple budget cycles. They can also find a growing list of professional (managed) services that cater specifically to OT. These options help to move security along in increments but not break the budget. Really what you need is a view of what you are trying to achieve (ie, what does ‘done’ look like) and then break that journey into multiple, achieveable phases. And where you can invest up front in tools that provide returns on multiple fronts or that introduce OT safe automated remediation and management then you really get the benefit. Remember a technology without an owner is a waste of money!
B. Second question. I did not necessarily say IT to an OT security role would not work, what I wanted to point out was that both disciplines (IT and OT) are very specialized and can be very technical. To expect a single person to be an expert at both is asking too much. When I built my team at Matrikon I only had traditionally trained IT types to draw from and had to teach them the basic nuances of OT. Essentially – do no harm and do not touch unless OT says you can. My point was simply that if you want the best possible solution you are better off pairing an IT expert with an OT expert to work together towards an OT safe program.
QUESTION:
How has the OT Organizations reacted to the TSA Cybersecurity Directive? Acceptance? Reactive?
ANSWER:
The answer to this is yes. All of the above and some others inbetween. Like most regulatory initiatives there are leaders and laggards. For a select few they are looking at this as acceptance and building appropriate responses. (read: Not just knee jerk reaction to tasks like “Within 120 days, implement and complete a mandatory password reset of all passwords within OT systems, including PLCs” While this does need to be done, it is not a one time task (or at least in the context of building a repeatable, sustainable program) it is part of what should be a new policy/procedure. And to incorporate this, along with many other exisitng or pending control, the opportunity is now to build a proper maintenance program. Unfortunately some do take the approach of ‘I don’t want to be first, I don’t want to be last – I just want to put this behind me and move on’.
QUESTION:
What are thoughts on the deployment of NERC CIP versus the Pipeline Guidelines? Do you see any possible lessons learned? Specifically the prescription of security functions.
ANSWER:
I indeed do see many parallels. And the biggest challenge NERC CIP had (and Pipelines likely will have) is in maintenance. (There are a host of other challenges NERC CIP had around enforcement, language, in or out of scope, etc. We saw large power companies physically and logically split their generation units into smaller, independent units to avoid megawatt thresholds!) But the single biggest lesson Pipelines needs to learn is that whatever path they choose to meet these requirements they really need to build in a maintenance plan. Expecting already overburdened OT staff to add inventory, user, software verification and patch/remediation tasks to their list is not sustainable. Automated tools, management support and budget are all fundamentally required. Just like a safety program!
QUESTION:
Is it advisable to have a in-house central team or third party?
ANSWER:
In house is good if you can afford it and find the team. Third party expertise and offerings continue to grow so are a good option if it is a better fit for your organization. However be sure to be articulate in your procurement and contract language just exactly what is or is not being provided and know that in the NERC CIP world, in house or contracted, mistakes or oversights are the fault of the owner/operator.
QUESTION:
What is Rick's take on alternative measures on Section 5 of the directive and how we can use that if we can't meet the intent of a directive?
ANSWER:
Alternative measures at first glance appear to me to be what NERC CIP built in for ‘Technical Feasibility Exceptions’. Essentially this is a ‘we cant technically do this on this specific asset’ exemption. Usually you need to prove what/why you cant do something, for example legacy or low functioning networking gear not being able to provide logging data. The second thing these types of exemptions require is a plan/path to remediate these exceptions at the next possible opportunity. You also need to submit what you are doing as compensating controls in lieu of the intended directive. So if you can’t patch a system for a specific risk but you can block the port/service it targets upstream at a firewall or layer 3 switch then that needs to be submitted as an alternative.
QUESTION:
What is the consequence of non-compliance to TSA directives for owner operators?
ANSWER:
At the moment there is not a formal fine/violation/audit function for pipelines but given the TSA is already a regulatory body and has other powers it can’t/won’t be long before some form of enforcement is added to the mix.
QUESTION:
What help if any on due care due diligence on TSA directives can help an owner operator, where we are not able to fullfill all directives but have a program in place to fix?
ANSWER:
I think this is an extension to the question about alternative measures listed above. In general more regulatory programs are looking for taking the intent and doing your best to achieve it. Specific, granular tasks/technologies are not typically prescribed but doing nothing due to a lack of budget or support is not an option. For those who show due care and due diligence and tie back existing measures as well as alternate measures back to directive specifics should be in a pretty good spot to pass a ‘compliance’ check.
QUESTION:
My question is on interpretation of TSA fines if we show due diligence on following TSA directives though not able to fullfill all of directive, what is Rick's take on that?
ANSWER:
Again this is about alternate measures. What I can say I have seen from NERC CIP is that if your organization is doing all that is reasonable (meaning you do free up someones time to change all passwords or to research, purchase and implement multi-factor authentication technologies) and have a timeline for completion you should be OK. What is not likely going to be well received is if you were to try to defer multi-factor (as an example) to Q1 or Q2 of next year because there are immediate operational projects underway. They have given direction and timelines and only technical limitations are likely acceptable excuses for non-compliance.
QUESTION:
Does Rick envision that TSA will issue more directives taking cue from NERC CIP and it will become a 2nd generation compliance program from the regulator?
ANSWER:
Crystal ball type of question here. My bet is yes. My sense is the TSA is merely starting with this specific set of directives. You can see within them (like documentation of privileged accounts) there are requirements for regular review. This is indicative of the expectation that pipeline companies need to be building a repeatable program. Not just change passwords and add multi-factor authentication once and never look back.
QUESTION:
Patching timelines are very aggressive on 35 day testing, 35 implementation for security updates on OT assets, how does Rick sees that being achieved keeping operational disruption/outages in mind?
ANSWER:
For NERC CIP it is a bit more nuanced than that. And not everyone even patches. If you dig into the details of NERC CIP language it can get complex but the general process is as follows: Within 35 days of a patch being released you need to first review it for applicability and assess its potential impact on your specific organization. You then have another 35 to either deploy it or deploy compensating controls in lieu of the patch. All of this needs to be documented. Now some NERC CIP entities deferred all patches and pointed to strong network controls (ie, data diode, physical security, etc) to justify the lack of patching but that is not a long term view (in my honest opinion).
QUESTION:
Which is the importance or the place of OPC UA for systems and security convergence across the industry?
ANSWER:
Any security tool or standard is welcomed and would be an improvement. And historically OPC broke a lot of security by virtue of its function to tie together otherwise unconnected systems. However as technology continues to be expanded upon and proliferates within operating environments technologies like OPC or IIOT appliances and apps absolutely need to be selling their virtues alongside their security capabilities. For all of you owner/operator types reading this PLEASE put minimum compliance security language into your procurement language and your project specifications going forward. Otherwise minimum compliant functional bid will be cost effective but not likely secure.
QUESTION:
There is no one-size-fits-all "standard" solution to security, so what does Regulation Do except add change to already difficult job?
ANSWER:
Great question. And all security practitioners always say ‘Compliance is not Security’. I am not a favor of ‘the stick’ and would prefer ‘the carrot’ myself but the challenge is that many organizations just do not take security seriously. So does regulation help get more organizations moving towards improved security? I hope so. But what it can do is at least provide a guideline as to what they *should* be doing within security. Again, it is not ideal but I have been doing this for 20 years now and for every 1 proactive security organization I see is see 3 that are half as good, 3 that have not started but are dipping their toes in the water and another dozen who have not nor will not do anything. (Until they are hit by ransomware perhaps?)