In our second episode, we ask the question: “what is OT?”, and attempt to provide an answer. Join host Ken Kully, Cyber Tech Lead at Verve Industrial, and his guests and colleagues Lance Lamont, Tyler Bergman, Doug Artze, and Drew Wintermeyer as they discuss what characterizes an OT environment and OT equipment, the impact of IT/OT convergence on distinguishing one realm from the other, and how a device’s usage is often the deciding factor as to whether it is an IT device…or an OT device.
Key Takeaways:
- OT is the collection of hardware and software that is principally concerned with process availability: monitoring the process, providing safety oversight, and the automation, supervision, and control of the core processes for any industry.
- The same device can be both an IT device and an OT device, depending on usage.
- OT systems are often – though not always – characterized by device isolation, and can also be spread over a large geographic area.
- There are some industries – healthcare is an example – where the distinction between IT and OT becomes very blurred.
Timestamps:
00:00 – Introduction and sound check
00:35 – Welcome to OT After Hours
00:43 – Host introductions and background
03:40 – What is OT?
04:32 – Parallel terms: IT and OT
05:56 – Other terms: DCS, SCADA, etc.
11:43 – Hardware differences between IT and OT
14:20 – The role of context in defining OT devices
17:34 – The evolution and convergence of OT and IT
19:52 – The CIA triad and its different priorities in IT and OT
22:36 – The timeliness element in OT systems
23:30 – Engineering workstations and their classification
32:36 – The criticality of availability in OT environments
34:30 – The sensitivity of OT control systems
37:41 – Examples of OT in non-industrial settings
44:27 – The importance of physical access in OT security
46:55 – The unique challenges of healthcare OT
50:59 – Concluding thoughts on defining OT
Guest Information:
Lance Lamont: Creator and Explorer at Verve Industrial Protection, leading the research team in exploring OT devices and their security.
Tyler Bergman: Experienced in utilities and OT security with over 20 years in the industry.
Doug Artze: Operations team lead with experience in nuclear power and wastewater treatment.
Drew Wintermeyer: Research team member and overseer of Verve’s internal labs of OT devices.
Subscribe
Get in Touch
SPEAKERS
Ken Kully, Tyler Bergman, Doug Artze, Andrew Wintermeyer, Lance Lamont
Ken Kully 00:00
My Dear future Ken, sorry. Regards past Ken.
Doug Artze 00:05
you present Ken right now?
Ken Kully 00:07
Well, not anymore
Tyler Bergman 00:10
sounds like a Spaceballs movie.
Ken Kully 00:11
Dear past Ken, you jerk.
Ken Kully 00:21
Welcome back to OT after hours podcast about industrial cybersecurity, why it matters. And all of the challenges that it presents. We actually do have a few returning names on the podcast and I’m again gonna go left to right across my screen. We have Lance Lamont, Tyler Bergman both back from episode one. And also a couple of new voices Andrew winter Meyer, and Doug arts. So, Lance, why don’t you say hello, of
Lance Lamont 01:03
course, I was the person that everybody was hearing in the first episode, apparently. So I lead the research team member of industrial protection here. And we’ve tried to figure out how to talk with all sorts of cool devices. I’ve been involved in embedded devices for many, many years and love being part of that industry. Already.
Ken Kully 01:24
Tyler.
Tyler Bergman 01:25
Hey guys once again. So Tyler Bergman been working on stuff since nine year old on my trs 80. And doing different consulting here and there for a while working in utility profession for about 20 years in different capacities from engineering design to now doing cybersecurity stuff. So, pleasure to be here with you guys.
Ken Kully 01:46
Brilliant. And yeah, like I said, new to the podcast, Andrew Andrew wittmeyer. Hey,
Andrew Wintermeyer 01:51
my name’s Andrew. Um, let’s see, I am the research lab manager here at Virg. So I manage and maintain about 500 Different ot assets. And yeah, just happy to be here.
Doug Artze 02:03
Thanks for having me.
Ken Kully 02:04
Cool. Doug. Is your iPhone working? Is Can everyone hear me? Hey,
Doug Artze 02:09
there you are. Hey, that’s Doug. Hey, hey, thank you. Appreciate the patience, guys.
Ken Kully 02:14
No worries, why don’t you give us an intro blurb and then I’ll go back to rambling. Absolutely.
Doug Artze 02:18
So I am coming too close to a decade now working in the OT space ot cybersecurity, worked on everything from design implementation, especially, I mean, mainly focusing on the power plants, nuclear power plants and wastewater treatment facilities. Brilliant.
Ken Kully 02:35
I’m Ken Kully. I guess I used to call myself a cyber team lead. But since the last episode, my roles changed a little bit. Now I’m the Operations Support lead with verve and those to the podcast, and you know, kind of like Tyler, I’ve been doing this for a long time. And just as kind of always seem that my career through engineering kept me in the orbit of OT, and I’m doing ot cybersecurity. So that’s pretty cool. And on that note, as promised last
Tyler Bergman 03:00
segue you Yeah,
Ken Kully 03:02
well, how many? How many wheels is on a Segway? Because that one only has one to two there. Okay. Right. So it’s not a person riding the whatever the one wheeled one is like the little you motorized unicycle but not like an actual like with a seat, just the wheel with like, platforms on either side, I forget what those are called.
Tyler Bergman 03:21
It’s called like, one wheeled skateboard or something. Yeah. And
Doug Artze 03:25
I think they’re moto wheels.
Ken Kully 03:26
Yeah. Okay, good to know, I don’t, I don’t know, electric, electric scooters or anything like that. It’s not really my scene, just throw me on a bicycle fallen off. But as mentioned in a previous episode, the second episode was, I wanted to focus on kind of a question of what is O T, because we throw around terms like OT and a parallel term ICs. OT stands for operational technology, and ICS stands for industrial control systems, although funnily enough industrial cybersecurity, also abbreviated ICs. But that’s neither here nor there. We mentioned these terms a lot. And I mean, most of us just kind of fall into the habit of, you know, just using the acronyms because we deal with it so much every day, it’s just kind of part of our lexicon now. But my goodness, getting an actual definition of what this stuff is, is kind of a challenge. We have these terms, ot ICs. And if you’ve dealt with them at all, you can probably intuit that they’re not, you know, ot has, I think kind of come to the forefront because it parallels it. And we know, most of us, I think, listen, you know, hopefully all of you out there listening have a decent sense of what it is beyond just oh yeah, they’re the guys who told me I can’t do that all the time with my computer. But, you know, like we kind of grasp I think now and it’s become pretty apparent the role of it of information technology and that group within a company. They’re the ones who aren’t charge of maintaining the fleet of hardware, servers, laptops, desktops, the operating systems that run there upon updates for those, if there’s antivirus in play, that’s usually their responsibility as well software updates of various sorts, providing software. But also things like you know, data management, data confidentiality, we talked about some of that last time as well, the CIA triangle, maintaining connectivity, because it is almost always Internet facing. And so if we think of OT as like a parallel to it, we can intuit that some of those roles, probably crossover. But at the same time, OT is kind of a different beast, we can kind of grasp that, because it’s a different environment, the requirements are different, the constraints are different, the challenges are different. But even so, what is OT?
Tyler Bergman 05:53
Yeah, so don’t forget the SCADA systems out there, too, you know, yeah, ICS, DCS, SCADA, all kinds of different terminologies out there, right. So I think it breaks it down into, you know, some of the major aspects of it, when I look at across the board is just, you know, consolidated systems that are controlling some type of process or something of that nature out there, right. Something that’s, you know, manipulating the physical world as far as a process or something of that nature. And I think that’s kind of, I’ve done some soul searching there for a while. And I think that’s really it, it really kind of consolidates down into like a singular set is something that’s controlling the process, and the physical world has real world impact as far as moving parts and machinery, things of that nature. And it just seems to me to be a little bit more broader of a brushstroke across the board. Because you know, you can have manufacturing, you can have industrial processes, you can also have wide area processes that are going on for like a SCADA system within a utility environment, you know, electrical and mechanical style devices and forms of protective relays in the electrical system for controlling energy usage and things of that nature out there as well. So it has no real world large impact. And sort of like on the IT world where it may be an individual machine or server. And I don’t, you know, don’t negate the overall scope of those systems. But typically, you know, you have individual user stations, that if they go down, they only impact like one person to, whereas if you have a station go down for like an operator, it has, you know, could be major impact for that operator. And they do have redundancies in those nature for multiple different workstations to add, to make sure that they maintain visibility, but you have reduced functionality set when it comes into those that have impact in the real world aspect that’s out there. So that’s, I think that’s kind of what it really boils down to me. You know, you’ve got the IT side, you have servers out there. Typically, if you’ve got like, large email facilities out there, you have a lot of redundancy built in to where if you do lose like a single server, and like a server farm for email structures, and things like that, you’ll have a little bit of a reduced capacity. But a lot of times they tried to design those two are as you know, a lot less impactful on those. Whereas on the OT side, there’s a lot of other considerations that go into it. They do have redundancy built into it on a lot of cases, but it just really depends on where they are in nerve, their venture, as far as you know, from an operational standpoint. And that’s not even talking to the next bridge over of the cybersecurity standpoint where you’re positioning this, as well,
Lance Lamont 08:54
as a quick question, when you send operator out what did you mean by an operator? So,
Tyler Bergman 09:01
I really stereotypically go towards like a control room. Right? So let’s talk about like, energy production facility. So you have a control room, you have an operator there that’s monitoring the processes, the little dials and gauges and things of that nature to make sure that something not is not going astray, and what is typically a fairly automated process. But there are like non automated functionality, that you have to have that you know, human component there to be able to monitor and adjust things as the process is ongoing. So like a colo facility, you have to monitor like effluent gases and things of that nature. Make sure you’re not going over permitted limits and things of that nature. So you have that operator component that will monitor that and Then if you start butting up against a certain limit or something, they can dial back certain things. So that’s part of that operational aspect from a control room operator perspective.
Lance Lamont 10:11
Sure. I was trying to understand the concept that wasn’t sure if you’re talking about operator as the factory or power plant as a whole, you know, the person operating the power, or the company, or if you’re talking about an individual, individual person monitoring things, thank you. Ya know, and
Tyler Bergman 10:30
it’s there’s certain different aspects out there, right? So you got SCADA systems all that but yeah, I digress.
Ken Kully 10:36
I would, I really,
Andrew Wintermeyer 10:38
I agree with Tyler’s definition of OT there have really with the parallels to it, where it like you see, like the output of any IT system is usually something that you’re going to see on a screen, either your application is going to work, you’re going to get access to something, you’re going to be able to log in somewhere, but it’s going to be very screen based. Whereas an OT the output of computation is usually an action in our physical world. Being that I work inside the research lab, I don’t get to see these devices out in the wild as much. It’s more like, I’m putting them into a zoo, and they stay in their cage. And they do exactly what I want. But I would say it’s definitely like, you definitely know when you’re working on an OT system, when to set the IP address of a device, you have to use a wrench, I think, yeah, right. I mean, that’s just that’s the kind of stuff you see an OT, like you’ll you’ll see. DIP switches are you’ll see tiles and stuff like that built into the devices. It’s really like computers for like electricians and stuff like that. It’s a, it’s, it’s a real different type of technology.
Lance Lamont 11:43
I always found the, I agree with these definitions, I always found the interesting space where there’s that sort of transition space, the network switches and all of this other infrastructure that’s required for these specific devices that are interacting with the physical world, to be able to interact with each other appropriately, you can find them both in the IT space in the OT space, I always found those interesting, too. So I find the definition gets a little blurry when we get into those because, you know, is this Cisco switch or other companies switch? Is this an OT device or an IT device? And you really have to look at it in context, not in isolation? You know, let’s think of a simple device like a printer. Right? You’ve got it sitting right at the end of your desk. Is that printer an OT device? Probably not? Yeah, I think the answer is probably not it’s your printing out your presentation you need to give to your manager the next day, or just notes from a meeting or something like that. Probably not a critical device. Now what if that printer same exact printer is printing out calibration data for the things as your manufacturing as they come off the manufacturing line? Isn’t that is it now an OT device? I think the answer is yes. Because it’s critical to the success of your manufacturing facility. If you can’t print out their calibration data and include it with the packaging, you probably can’t ship that product. So it’s it’s not a cut and dry thing for a device to be either an OT device or not ot device, you have to take it in context. And that’s where things get really fascinating to me. But even if we just talk on the hardware side of things, a basic desktop printer, we’ve just said it’s probably not ot unless it’s printing something specific to manufacturing. What if it’s a whole office printer? What if it’s the ones that can print and collate and do all those multifunction things that 40 people utilize? Is that an OT device? I think the answer is still no. I think there’s a timeliness and Criticality aspect. That is, we’ve been talking about what happens if that device is a printing press, you know, giant printer that is printing off, you know, magazines, is it? I believe that would be an IoT device. But I need to figure out where’s the where’s the threshold? What’s different between that printer that’s in the office that can print out, you know, 1000s of copies? And that printing press? What’s the difference?
Ken Kully 14:20
Well, I mean, I think you’re kind of getting at it, when you’re, you know, sort of looking at the the the T loss the end that it is that it is being operated toward right. Because, you know, yeah, if it’s just, you know, printing out, you know, the occasional email and the occasional silly photo and, you know, business documentation, all the usual things that you would expect an office printer to print out. Well, it’s, you know, it exists to support the operations of the office, which in most cases is in the IT realm, right. If it’s printing, though, configurations, it’s printing out, production reporting, then it’s being utilized to support the operation of some kind of process within some kind of industry. And that was sort of one of the, when I was, you know, again, like reading some of these things from like Cisco and Red Hat and Palo Alto and just other sources of, you know, trying to opine on what is the difference between it and OT, one thing that kind of kept popping up again, is that O T, which is kind of a blanket term, right? I mean, cover ICS DCS, supervisory control and data acquisition SCADA. I mean, all of those systems exists to monitor and automate industrial operations, whatever those those might be, usually in real time, or quasi real time. And oh T, really is, you know, sort of that I think Palo Alto use the word integration of hardware and software to effect that management and monitoring and automation of industrial operations. It’s, you know, the cloud of devices and programs that serve the purpose of controlling and monitoring and interacting with and programming, the different devices that are used in industry. And, you know, actually defining and controlling and regulating and shifting the processes that are the production, you know, whatever the production is, right, whether it’s a pulp mill, or a paper mill, or, you know, our plant, refinery, whatever the case may be, just, you know, it’s that cloud of stuff, hardware software, that enables the effective monitoring, and controlling and manipulation of that entire process, environment, whatever it is, which, you know, so when we talk about things like it and OT integration, right, I mean, a lot of the times, that can mean two things, but I think it’s maybe better to separate out, you know, the idea of integration from convergence, because those are two different things. Integration means that you can use a lot of things like, say, a desktop printer, in both realms, and to serve both ends, right, you can use a desktop printer to serve the needs of your front office. And you can use that same desktop printer to serve the needs of your production floor. And, you know, is it the same printer? Absolutely. But yeah, one is being used in IT capacity, one’s being used in OT capacity.
Tyler Bergman 17:34
Yeah, I’d like to expand on that a little bit, too. So you got a lot of parallels? Well, so I’ll even extrapolate that some more, you have a lot of more recent parallels, as far as you know, ot equipment out there. Because, you know, there’s a lot of times where, you know, at one point in time, this was like, largely analog, or, you know, relays and things of that nature, a lot of hard wired devices, or controlling a lot of these components. So as we’ve continued to evolve into more electromechanical style devices, and more intelligence built into them, you get into computerized components that are out there that it’s like, oh, my gosh, we can now securely connect, and have all this data come out of it through the serial connection at 9600 baud, and all this other good stuff. So it’s been an evolution out there. So we’ve finally gotten to kind of the tipping point where you have a lot of large convergence. And they talked about this convergence of OT and it because there’s a large lot of large parallels to the same style of equipment that you would use in the IT space on the OT space as well,
Doug Artze 18:49
you have one of the van deltas, comparing it cybersecurity professionals with OT is that in the IT world, they seem to be more siloed into that space, where as an OT, we also, we crossover fairly frequently into the IT space. And from my experience, one of the biggest Delta’s I’ve seen is how, on the IT security, there’s more focus on protecting data confidentiality, privacy, similar to what Tyler mentioned, when he was breaking down his definition of it. While an OT, there’s always even though there’s a focus on those that I mentioned, there’s a priority and also keeping these processes running safely and reliably. That’s a line we’re constantly flirting with, right? How do we secure these systems, but still operations being so important for the from a business side and that’s one of the biggest customers concerns that the customers tend to bring more frequently? Any thoughts on that?
Tyler Bergman 19:52
Well, that was actually a huge topic on our first episode, where you know, you have the CIA triad of the confidentiality, integrity. unavailability, whereas, you know, we kind of flip that dynamic and it’s availability is like King on the OT side, and then you go right out and integrity and confidentiality, because a lot of these systems are very localized. So you’re not really concerned about the confidentiality of it. But you’re concerned about the availability and the integrity of it. Absolutely. All the way through. Yeah.
Ken Kully 20:22
And yeah, and again, you’re right, we did talk about that, like in the previous episode, but I think maybe, for me, like, coming back to what you had talked about Tyler and then Andrew as well is, but just thinking like, on my own experiences with OT, and Tyler, you really got at this, I think with with your explanation is that you know, this is, it is essentially just mostly about the electronic world. But ot really is sort of, it’s about where the digital touches the physical. And we could say analog digital, and to be fair, like, the whole concept of digital analog interface is a core component of almost any ot system, because at some point, those digital signals, hit some kind of device that is then going to be manipulating voltage on a wire. And that voltage on a wire might assume you know, one of a couple of discrete values, or it might cover, you know, every step of voltage in the range. And then at the end of that wire, there’s something whether it’s a pump, whether it’s a valve, and it is not going to have discrete states, I mean, in theory, a block valve, you either want it fully open or fully closed, but any valve is not going to have just those two discrete states, a valve is going to be partially open, partially closed, there’s going to be a transition state that comes up, pumps can adjust the, you know, the pressure that they are putting down a line, motors can assume variable speeds, right? Different RPMs, depending on need, and demand. So all of that lovely, happy digital stuff suddenly becomes real world and analog and very, very variable. And I mean, that’s, that’s where the big challenges come from, because you know, if those things and again, we discuss some of this with regard to like Stuxnet in the first episode, if those things go a little too far out of their normal range and their normal tolerances, then you know, that those real world processes can start to generate real world smoke. And that’s not a good thing, either.
22:36
That makes a lot of sense. That was their timeliness element to this. For instance, you mentioned that it’s the devices that are used to monitor and configure. Now, in a manufacturing sense, there’s an engineering team, and the engineering team is using their computers to design new versions of the widget you’re making. And then that is you. Their designs are then used to reconfigure what’s happened in manufacturing for the next generation of the widget. I don’t think we would call those engineering workstations, ot devices. So I think there needs to be some sort of a timeliness element to this, it needs to be a, you know, the loop must be not in order of weeks or months or years, which is the engineering time loop, it needs to be talking about, like real time or semi real time monitoring and control to get into the OT realm.
Ken Kully 23:30
I mean, we have we have an issue there with terminology, though, right? Because sure, when we talk about, you know, an engineering workstation, yeah, we could mean something on the front end where they have CAD installed. And they’re, you know, tweaking the design for a widget. But right, and that was my, yeah, but going back to Episode One, you know, we also use the term engineering workstation in a different context, which I think came up in the Stuxnet discussion, right? Because that was the whole thing about STUXnet is inserted itself on what was essentially an engineering workstation, or typically termed as such, within an OT environment, which served the express purpose of being the home for the software used to configure the controllers hooked up to the high speed motors. Right. Right. So you know, in that sense, it’s an engineering workstation, you know, what we would call an engineering workstation in that context, is definitely an OT device. But that’s a terminology problem, right? So you know, I would, if we’re talking about something where they’re pulling up the CAD diagrams for the widget, and you know, like tweaking it, and, you know, maybe doing some like airflow modeling, or whatever the case may be, that they need to model to determine how effective a widget it is. I maybe call that a design workstation. That’s probably a better term for it. Right? And I guess that kind of gets us back to the that CIA triangle too, because at the end of the day, what is the you know, what is the thing on the workstation that has CAD and has the designs and has all of the you know, For materials testing and whatever other modeling, what is the thing that we need to worry about protecting most, we need to ensure that we’re protecting the integrity of that data. And we need to ensure we’re protecting the confidentiality of that data. We don’t want anybody stealing our widget designs, nor do we want to lose our widget designs. Right? Right. If that station goes down for a day or two, that might be inconvenient. But as long as the widget designs are secure, and backed up or otherwise still available, it’s not the end of our world. Right? Right. Whereas, you know, if this is, you know, if we’re talking about a workstation, where the availability of that data, and sometimes it might be the same data, it might be a CAD file in each case, right, because maybe, you know, that’s getting fed into a CNC machine. But now, when it’s the availability, and still the integrity, but the availability becomes king, because that data needs to now be fed into the process. And the process is actually, you know, converting that data, that electronic file, electronically stored information is now being converted into, by whatever means something that’s actually happening in the physical world, we are taking the CAD file that has a picture of our widget, and it’s going through some wires and some controllers. And all of a sudden, there’s this spinning machine over here that is literally like polishing widgets out of blocks of metal. Now, having that availability of that data is key. Because if we don’t have that data, well, then maybe all of a sudden, our C and C program interrupts in the middle. And if the data integrity is not there, then maybe we make a bad widget. But either way, now I think we’ve crossed into what is arguably, OT,
26:47
that makes a lot of sense, it almost sounds like a good general definition for ot would be devices where availability is
Tyler Bergman 26:56
the highest priority, I just had a thought out there to see, you know, when you were talking about the real time that you know, you you’ve got real time processes are out there, too. So these things have to react in a certain, you know, state, so like predictive relays, you know, you’re talking about 60 cycles on your electric grid, so they have to be very responsive, and their capabilities to be able to respond for particular incidents on the wire. Yep, so you’re talking about, you know, the sub, like, millisecond type of things that you’re trying to react to, or anything like that, whereas, you know, typical, and I’m not deflating anything. on the IT side, I’m just saying that, you know, these are very well, specific purpose style of devices that are built to provide specific apparatuses, you know, or it’s like, the IT world is more general purpose computing type of things, you do have some more generalized things for like networking and ASICs and everything like that are out there as well. But, you know, typically, on the OT side, you have a lot of like, very specific devices that were built to achieve a very specific thing out there, you know, and are continuing to grow and all that good stuff. So it was like a real time OS that is like, monitoring that electricity line off of different signaling aspects for the voltage and current that’s coming off of it. And then sending out the st send signals to a, basically a circuit breaker, which is, clinically linkage switch, all it is to be able to do protection on the, like electrical grid.
Ken Kully 28:35
Yep, same ideas, like you know, pressure relief valves, various, you know, Safety Instrumented Systems as is that, you know, monitor plant processes, often involving, I don’t know, pressurized steam, or a big fireball, or things of that nature, right things that are important to producing power or, you know, like, could be a different mix of like toxic chemicals important to producing whether that’s pulp and paper or whether that’s polyethylene glycol, or you know, you name it all of these things that are like so critical to what we kind of know as just you know, our orderly day to day lives there’s a lot of intensely
Tyler Bergman 29:20
making widgets out there just
Ken Kully 29:22
right like you know, to put in your putting your cans of Guinness like whatever the case may be all of these things you know, that we kind of almost just think you know, I turn the light switch on and a light comes on cool, I’m happy you know, I’m not thinking about a bunch of natural gas on fire 100 kilometers away from me creating steam to turn a turn a generator, but that is happening. And there’s, you know, all kinds of you know, you want to be able to monitor those things. You want to be able to very quickly like you say shut those things down. If there’s anything that happens anywhere in that system that could endanger the physical hardware this system endanger the lives of the people. Working on the system. And they do need to be able to respond with that, like sub millisecond latency. And it very, you know, it’s, it becomes it’s that bridge between like, you know, the digital, the computer over here, and the physical, I’m literally slamming that Valve shut, or forcing the valve open, depending on the situation. At the end of the day, you know, to keep your process running, everything that is integral to that process needs to be online, which means availability. You know, if your controller goes out, if your motor goes out, you have a problem, if your controller on the motor goes out, you might have a problem depends a little bit on its fail condition, fail, open, fail closed. But, and sort of everything up the chain from there. Now, in some cases, you know, which gets us back to that loss of view discussion, right. But in some cases, you know, you might be able to actually, as long as the motor is still turning, and it has its program locked in, as long as there’s no need to make any kind of changes for the next however long, you might actually be able to just let that run for a little bit, while the controller power cycles and everything else, as long as the failure condition for the controller doesn’t automatically bring the motor to a halt. Right. And similarly, you know, if you lose view on it for a few minutes, as long as you know, okay, well, the program is locked in and it doesn’t need to change for an hour. So we can take 10 minutes to reboot this workstation. Okay. But yeah, what, if you’re even having those conversations at all, you’re probably in an OT realm of some.
Doug Artze 31:32
Absolutely. Versus
Ken Kully 31:34
like, oh, shoot, like, you know, the accountants laptop blue screen, and we have to reboot it. Right. You know, I mean, that sucks if there’s a report due in the next hour, but it’s not, you know, if the reports 15 minutes late, there may be some knock on effects, you know, that are kind of outside of our ability to perceive here, maybe, you know, okay, you’re 15 minutes late filing that report. So you know, maybe there’s an extra form you have to fill out for the stakeholders or something, right, maybe your stock price dips a couple cents, because you were a little bit late in posting your results, or whatever the case may be. I don’t really delve into the stock market. And it’s not that that is not an impact. It’s not that it’s not a bad thing. But again, like, you know, that’s it’s a different category of problem, versus I don’t have the data I need to keep this power plant running,
32:27
potential injury or damage to health or happiness, or all those sorts of things as well. And safety as well. Right? Yeah, absolutely.
Ken Kully 32:36
Yeah. Any kind of instrumented system that exists to preserve life or preserve physical integrity, very arguably, ot because, again, availability is King there, you really don’t care if you know your fire suppression system is confidential, you just care that it turns on.
Lance Lamont 32:55
Yep. So yeah, as you as you look at each device around you, if you were to say, Okay, what, what’s the most important thing is it confidentiality, integrity or availability, if you look at that device and say, this needs to be available beyond all else, all the other things might be very important as well. But if availability is key, it is probably an OT device. To
Doug Artze 33:17
quick paint a quick perspective also, on the sensitivity of some of these control systems. I’ll make one quick one, was working back in LA. And when we were doing some some network hardening on a power plant, there was a failed load, and they’d created a trip they turbine, which is one of the biggest nightmares for anyone in the IoT space, securing power generation sites. And in order to protect it, all of the assets for that specific customer or operator, they created a chain reaction. So in order to protect their physical hardware, it has the system in place to trip the other sites that are nearby. So again, just painting a perspective of how dangerous sometimes working on these systems can be and the impact it may have. I think that incident itself led about 250,000 people with no electricity until those system were brought back up. So just just to paint a perspective there on how you know sensitive and high stress working on these operating environments can be
Tyler Bergman 34:30
scheduled Blackplanet plants.
Doug Artze 34:33
The famous the famous topic, I know the famous theory, zero megawatt club, the club, no one wants to be in.
Ken Kully 34:41
Yep. i And again, a lot of issues we talked about in the first episode, right? If you have a loss of view of it, are you under requirement to shut down? You know, are you going to be able to maintain control of the process if you don’t have someone who can have you know, a hand on the numbers for 1520 The minutes, all of these interfaces between, you know, what you see on a screen and what’s literally happening physically in the world somewhere often close by sometimes not so close by. And it is all just like, a lot of it is just very finicky and very sensitive. And I mean, sometimes that’s an accident of design. And some of that is very deliberate part of the design. But the end result is it does make, you know, working in OT environments, incredibly challenging, incredibly sensitive. And it does just seem strange to you know, come across from an IT environment where I mean, not that, you know, not that it environments are, you know, completely forgiving of these things, right. I mean, there’s lots of situations where you don’t want a production server to come down, if Amazon US one West goes down, a lot of the continent is going to have a bad day. That is true. And even within, you know, an individual company, if you lose, you know, certain critical servers, you know, maybe your billing systems offline for a while, maybe your, your record keeping offline is for a while maybe some other aspect of you know, your day to day, business operations are affected. And that’s not good, either. But it’s not really the same category as literally having to shut off 2000 megawatts of power generation for however many hours it takes to cycle the whole system, and reheat the boilers and bring it all back up. And the impacts are much further reaching probably in that latter case. Because, you know, again, yeah, it sucks for you as a business if some aspect of your business operations goes offline. But if the operations that are offline are providing power to 750,000 people, the sense of scale is wildly different. And I mean, we do usually think of like, you know, operational technology, ot as being the province of industrial environments, it’s certainly like, if you, you know, Google or Bing it and look for you, what is OT, you will tend to see that the discussion gets centered on operational environments, process environments industry. And to be fair, that’s where most of it is, but not necess. You know, again, it’s, you know, one of those terminology things, right, is the H vac system for a building industrial, we probably don’t think of it as such. But that is an operational technology system. Right? Are the X ray machines at a hospital? An industrial system? I mean, they’re certainly, they certainly have what we might call industrial scale, but we wouldn’t necessarily think of them in the same category as a paper machine, or a boiler. But arguably, those are an OT environment as well. Well,
37:41
that’s actually a really interesting example. The X ray machine, I’m not sure from from the definition I posited here, the availability is most important. I’m not sure, obviously, we want the x ray machine to be available. But I think even more important than availability on an x ray machine is integrity, it has to operate correctly, whenever you operate it. So that that that sort of pushes against the definition we just created, or we just created. Because if that X ray machine operates incorrectly, that’s even worse. That’s true.
Ken Kully 38:22
And it’s kind of weird, because if, again, just you know, all the different articles I read, like trying to narrow down how people define OT, a lot of times and it does kind of break along the lines of confidentiality versus availability, I tried tends to be concerned about data confidentiality, oh, t tends to be concerned about equipment availability. Whereas integrity in that, well, its integrity is kind of on both sides of the line. I mean, I guess it depends on, you know, I guess we have to, you know, do some failure analysis on the X ray machine, right. I mean, obviously, you know, if the X ray machine, what is the failure condition of the X ray? Or what is the failure that we’re dealing with on the X ray side, right? Is it putting out way too much X ray? Well, that could be harmful to people, right? In addition to producing incorrect imaging, that could actually be harmful to the patient. It’s not putting out enough X rays. Well, that is impacting imaging, which has the knock on effect of you know, impacting the patient, right, if we can’t see clearly what where the fracture is on the bone, how do we know how to deal with it? Right. So it is a good point, but like, I would still tend, I think, to think of that hospital equipment as being a form of OT, because availability still is fairly important to it. And I mean, maybe we could step beyond the X ray machine and just go to you know, like the mobile works and you know, the mobile stations that are heart lung machine or even just like the mobile patient care workstations that a lot of clinics All the nursing staff used these days, right? Like those things need to be operational. So you can access pertinent patient data in a timely fashion. And I mean, you would hopefully have that, you know, there is a confidentiality concern there. So this is maybe where we get to the IT ot convergence. Maybe its strongest example is in healthcare, because obviously you care about like patient privacy, confidentiality of patient data. But, you know, we’ve seen the case of like ransomware attacks against hospitals limiting the availability of patient data. And that’s a huge concern. And it becomes a huge concern very, very quickly, especially for patients who are, you know, in ICU or in trauma care? And, you know, oh, shoot, I can’t look at, you know, oh, I don’t?
Doug Artze 40:43
What’s his medical history?
Ken Kully 40:45
What’s the medical history? What are this person’s drug interactions? Blood type, right, all of this, they’re unconscious, they can’t tell me any of it. And a lot of that data does need to be delivered in near real time. Yeah. So I do agree that, you know, the lines blur. I think if it were, you know, if it were me making the call, I would probably say that, you know, a lot of that stuff in hospitals still probably falls more on the OT side than on the IT side, again, just because of the need for that, you know, the availability, the potential harm to human life, in the absence of availability, and the need for real time or quasi real time access to the data.
41:26
That makes a lot of sense. The, the, I think availability is still probably the most important thing for defining an OT device. Yeah, boy, that’s a, it’s just an interesting thought, talking about an x ray machine or other medical equipment, X ray machine is one of those things on the cusp, but then the heart lung machine that’s like, no, it better work, as long as we need it as soon as we need it. So that’s definitely qualify as an OT device.
Ken Kully 41:54
And I mean, you know, you can wander elsewhere into the hospital, and you’re elsewhere down the chain of devices in the hospital, right and get a little bit closer to the patient. Well, what’s the patient hooked up to right, probably a heartrate. Monitor pulse oximeter. Essentially real time monitoring of, you know, vital body statistics, could argue that’s an IoT device. Now, that might be you know, that unit might not even be online in the sense that it’s connected to a network, I imagine. Many of them are, but not necessarily all of them. Right? Some of them are just local to the patient. They just live on the on a pole by the patient’s bedside, there’s no right cabling other than what’s going to the little pulse oximeter sensor or to the blood pressure cuff. And everything is just recorded on box. And I mean, from a, you know, well, how do I defend this perspective? Cool, great. That particular device is as robust as it’s going to be, because it’s not connected to anything other than itself and the patient? Is it still an IoT device? I would argue that it fits the definition for an IoT device, it’s just not a connected IoT device. So you know, trying to devise a cybersecurity strategy, then it’s like, okay, well, we know we have those devices out there, but they’re literally never hooked up. So don’t need to worry about them. Except, you know, to make sure that only people who should have physical access to them have access to them. The physical access is definitely one of those things to take into account. Yes. But you know, in other in other cases, I mean, those, you know, the workstations that the nurses need to move around, are, they’re going to need network access, they’re going to need to be able to pull up patient data from different points, and then from different workstations. Arguably, some of the bigger stuff that you know, the heart lung machine, maybe maybe not necessarily the portable ones, but you know, the larger ones, the MRI devices, the X ray devices, a lot of those, there’s a demand. Now, I think, for network connectivity, because again, that information doesn’t just need to stay locally within the machine or the you know, just the computer that’s hooked up to it, that needs to then be able to be added to the patient’s files sent to the doctors, all the rest of that. And again, you know, there’s still the confidentiality concern, which puts it, you know, which tips us towards the IT side, but again, it’s that data availability, you know, making sure that I can actually get at all of the pertinent information about the patient that I need when I need it. I think you know, that that tip suspect towards the OT side, so maybe that’s like maybe maybe healthcare is like converged. Like when we’re talking about convergence, maybe that’s where health but healthcare is. Yeah. The more
44:27
we talk about this, the more I have more of my respect for everybody that works in the health care, Id OT, and obviously everybody, doctors, nurse and everybody just that is a unique and very, very challenging space to be in. I definitely have to respect
Ken Kully 44:43
all of them. Yep. It’s keeps you on your toes. Definitely.
44:49
I can’t even imagine what a what a normal day is versus versus even an abnormal day. Yep. So so we’ve been talking You mentioned a little bit when we’re talking about hospitals, the physical access to these devices. I don’t think it’s pertinent necessarily for this discussion, but we should definitely talk about from a security standpoint, there’s all sorts of factors, not just, you know, do we have a firewall? Do we have the latest updates? There’s like physical access. There’s lots of interesting aspects that we can explore. But there are two.
Ken Kully 45:22
Yeah, I mean, I guess it’s worth, you know, on that it is worth noting that usually, it is characteristic of an OT environment for things to be a bit more isolated. And not just electronically, but also typically, physically there. That’s not necessarily to say that they are under lock and key, often they are not, but it’s usually not, you know, it’s not the same as walking into an office. And you know, it’s just rows and rows of desks with computers on them, that tends not to be how ot works, you know, there’s, well, okay, so in that cabinet over there, there’s a switch there. And then in underneath that boiler over there, there’s a little controller shed, and inside there, there’s a cabinet, and there’s another switch in there. And there’s also a workstation. Like it’s, they, you know, the physical locations, and a lot of these things are just they’re more obscure. And that doesn’t necessarily mean they’re more secure. But they’re definitely more secure. They don’t, you know, sort of conform to what we tend to think of as being characteristic of an IT environment. Computers on desks, yeah, there’s a server room with everything nicely wrapped up, you know, the printers in one convenient spot next to the coffee room and whatever else. The positioning of devices, the ability to access devices, physically, is a little bit more, shall we say, ad hoc, and the environment in which those devices are situated is not necessarily, you know, as clean as we would typically expect an office environment to be very true.
46:55
Also, some of the IoT devices are like monitoring pipelines, where they, they are hundreds of miles potentially from any human, right, it’s
Ken Kully 47:05
literally in a shack next to a bog with a solar panel on top. No, that’s very true. Especially when you get into things like SCADA, right? Because, yeah, you’re distributing control over a large geographic area at that point. And so yeah, it literally is the case of oil. Yeah, well, we have, you know, six black belts on this pipeline, and there’s a little shack, you know, next to where each one is located. And that’s where the, you know, that’s where the instrumentation is, and the controller and the little HMI out there, like so. Yeah, and the last two are only accessible in wintertime. Because otherwise, it’s all like swamp land. And we actually have to like, wait for it to freeze over to drive out there.
47:45
I hadn’t heard that one before. But that’s interesting.
Ken Kully 47:49
That’s, that’s not? Well, no, I mean, that’s, that’s environment, like literally in the sense of like the, you know, the, the natural environment in which a lot of these things operate, right. I mean, up here and up here in Canada, and, you know, in northern Alberta, that’s not uncommon. It’s not to say that it’s ridiculously common, but it’s definitely not unheard of, I’ve worked for companies that have had sites like that my sister works kind of in an oil field adjacent role. And she’s, you know, been in situations like that, where, you know, she has like, and some in in her case, sometimes, because she does, like environmental sampling and stuff. In her case, sometimes it’s even, you know, more pressing, and that’s Yeah, okay. Well, we can go on site now. Because, yeah, we have an ice road so we can get there. But, you know, like, it’s, it’s March. So like you have, you know, two weeks, unless the temperature goes above this, then you need to leave that day. Wow. Because otherwise it’s a helicopter to get you out. And there’s nowhere to helicopter to land. They’ll just drop a ladder.
48:57
Talk about availability being critical. This fail, we can surface it in four months. Yeah.
Ken Kully 49:05
But we need it tomorrow. Okay, well, then it’s a helicopter. Wow. And I mean, sometimes, you know, that’s the call that gets made. Right. But that’s, you know, that’s a very expensive block valve service at that point.
49:19
Okay, do we want to try and circle back here and just come up with like our conclusion, I think we have sort of a core definition of OT, we’ve we’ve come up with a bunch of different ways. What do we think that what is OT? What is an OT device?
Ken Kully 49:37
Like I said, I think I’m gonna go with sort of where I was going at the a little bit ago there, you know, where it really is, you know, it’s that collection of hardware and software that is principally concerned with process availability, whether that’s monitoring and controlling the on different devices that affect the process, or, you know, providing safety oversight to keep the process within its safe parameters and bring it to a halt, if it falls outside of those parameters. It really is just about that management and monitoring and automation, all of those different industrial operations in real time or near real time to monitor and control, whatever the process, the core processes are for any particular industry. That’s my,
50:33
I think it’s a great definition. I can’t disagree. I can’t disagree with that at all. That’s, that that feels like where we’ve been heading this whole time. And it’s, it’s interesting, because devices can be, you know, it can be a standard IoT device, but it’s in an OT world, and thus it becomes ot covered. So part of the IoT world. So, yeah, it’s a really interesting to world to work.
Ken Kully 50:59
Yeah. Well, I mean, I think it’s a good definition, because it’s really just focusing on what the devices are used for, right? I mean, not that I’m, I’m probably never going to see, you know, your average, you know, PLC is usually going to see though is in an IT environment, there are some types of devices out there definitely that are, you know, this is more or less the province of OT, this is more or less an OT specific device. But it really does come down to, I think that’s why I like that definition that I offered is that it really comes down to a usage focus, not a device type focus. It’s what and which we got, you know, which we touched on earlier talking about the printers, what am I using this thing for? Because yeah, some stuff I’m really only ever likely to see in an OT environment, your typical Windows workstation, I can see that anywhere. Typically, you know, printer, I can see that anywhere. But what am I using it for? Yeah, context, the context of the device is really important. Cool. Well, unfortunately, we’ve lost most of the rest of the discussion group. But yeah, apart from that definition, oh, you got any other closing thoughts, Lance? Well,
52:05
I really liked that definition, I think it’s really important to pay attention to the context more than just the name or title of the device, or even the label of the device. And it’s this has caused me to reflect even around my house, you know, the, the connected garage door opener I have is like, Wait a second. That’s that’s probably not it device anymore. So it’s caused me to do a lot of internal reflection, just my surroundings and tried to categorize things less has been fascinating. So thank you very much for the mental exploration there. Oh,
Ken Kully 52:42
happy to see. And this is why I tried to not have anything critical to my house is operations have network connectivity? Yeah, I’m gonna go cut that wire right now. Yeah. Hey, do you want to do you want to switch to our services? It comes with a free Nest thermostat. Let me stop you right there. That is the last thing I want. Okay, cool. Well, hopefully, after all of that discussion, we’ve come away with a good definition of what it is. You know, if you agree or disagree, by all means, feel free to reach out to us on LinkedIn. You can either reach out to Verve directly on LinkedIn or to myself. My profile will be in the show notes as well. Happy to hear from you either way. If you have any additional thoughts you’d like to share on what OT is, we would be happy to hear them might even find a way to incorporate them into a future show. But until next time, thank you for listening, and we will talk to you again soon.