SPEAKERS
Ken Kully, Dylan Stencil, Lance Lamont, Tyler Bergman
Lance Lamont 00:00
everybody hearing me?
Ken Kully 00:01
Yeah, you’re good. Yeah.
Lance Lamont 00:02
Yep. Glad to hear that
Tyler Bergman 00:04
said a little bit louder for the rest of us in the back of the room.
Lance Lamont 00:08
Is everybody hearing me?
Dylan Stencil 00:13
Oh, you merely adopted the dark.
Ken Kully 00:16
I ever want to do a podcast trailer. I know who I’m coming to.
Tyler Bergman 00:21
Yeah, that new setups treating you well, Lance.
Lance Lamont 00:24
Thank you. Yeah. In a world without security.
Ken Kully 00:29
Well, that’s the world we live in. Do
Lance Lamont 00:34
we got a little Yeah.
Ken Kully 00:36
Okay, I’ll just put it on a different port. They won’t find it
Tyler Bergman 00:40
till somebody drops it on showed in and you’re like
Ken Kully 00:45
What do you mean capital A SDF? 1234 Bank isn’t a secure password. It ticks all the boxes. It’s got all the requirements Welcome to OT after hours podcast about industrial cybersecurity, why it matters, and all of the challenges that it presents. And this is the first podcast episode two, the pilot episode, the inaugural episode, the let’s figure out if we even know what we’re doing episode. I don’t know. Yeah, well, yeah. This is where we, this is where all the bugs come out. And then, you know, we’ll we’ll patch them in future releases, of course, and introduce new bugs.
Tyler Bergman 01:27
It’s not a bug, it’s a feature.
Ken Kully 01:33
Well, before we dive into the topic, we should probably go around and introduce ourselves as being the pilot episode. It’s still too early to say whether any of us will become regulars on the podcast, but I’m gonna go just from left to right across my screen. We have with us today, Lance Lamont, and Tyler Bergman, and Dylan stencil. And why don’t each of you starting with you, Lance? Why don’t you just take a minute to introduce yourself? And tell us a little bit about who you are what you do?
Lance Lamont 02:04
Absolutely. So I first of all, I think I’m a creator, Explorer, I love taking on new challenges, whether it be hardware, software, anything, anything of the sort, I with verve industrial protection, here, I am leading the research team where we take on various ot devices and try to understand how to communicate with them to work and gather as much information on the devices as possible. For this discussion, I think I’ll be talking about some of the adventures some little things we found related to the security of devices.
Ken Kully 02:39
That’s it brilliant, and how about you
Tyler Bergman 02:41
don’t know if I want to date myself that bad. So yeah, I’m Tyler Bergman , and you know, doing utilities for the last 20 plus years, from a distribution cooperative to a Lucanus Aldi, and did a lot of various things. Well, you know, started at a very young age of nice, tender age of nine working on my trs 80 computer with that I have very fond memories of, you know, writing and basic code back in the days, and that kind of fell forward. And now I’m here with you guys talking about duty security. Nice. All right.
Dylan Stencil 03:22
So my name is Dylan stencil, part of the research team under Lance Lamont and kind of had a interesting start in the industry I kind of grew up with my dad owned a CNC shop. And basically, I’ve been interested in technology since a young age. And I’ve started with doing pretty much controls, work hands on and then move kind of through the years and to the security space now that it’s becoming evermore important. So just a little bit about me.
Ken Kully 04:02
Brilliant. And I’m Kenneth Kully, Cyber Team Lead with verve. And then before that, I’ve worked for a few other companies pretty much it’s kind of funny, because even when I think back to like, you know, being an engineering student and grabbing summer jobs with whichever engineering firms were hiring, through the you have a jobs board, I kind of always ended up in the OT space, whether that was doing, you know, lightweight SCADA programming for for oil and gas companies or pipeline companies or, you know, working with industrial cybersecurity groups at different companies. And so yeah, I’ve kind of just always been in the orbit of OT, but it’s really only been since about 2008 that I’ve kind of formally gotten into the industrial cybersecurity track and have kind of been with it at a number of companies ever since Verve being where I’ve ended up And in a lot of respects, Verve is kind of like the culmination of all of that other stuff, because it was all always about like information and getting information on systems and like figuring out what do we even have in the environment, and like Verve literally has the whole solution already built that I was trying to build at my previous employer for like the last five years. So it was an easy transition to make. So as, as the pilot episode for the podcast, I wanted to explore something related to like I said in the intro, why it matters, and not just in like a generalized sense. Hopefully, we all know that we shouldn’t, you know, reuse passwords, or use 123456 as a password. Most of us probably have experienced with ID departments and maybe malware infections. So maybe we kind of understand why the Internet facing stuff at least needs security, I realized that I’m making a lot of assumptions there. But you know, my hope is that, you know, if, if you’re specifically going out of your way to look for an OT security, podcast, and industrial cybersecurity podcast, these are concepts that are probably not strange to you. But it is a little bit different in OT, isn’t it, which ideally, is not supposed to be Internet facing OT is supposed to be the stuff on the plant floor hooked up to that paper machine or the boiler over there, ensuring that you know, whatever process is important to a facility keeps processing along. And I mean, yet, we’re going to actually have to do a whole nother episode on what ot even is. But Lance, you actually pitched the idea that I thought would make a good topic for this first episode. And it really kind of gets more towards the question of why industrial cybersecurity even matters because a lot of the connected devices that exist in OT environments do have vulnerabilities, not always ones that have been published, let alone patched. But it can be difficult sometimes to really communicate to people who work in those ot environments. Just why it’s important to worry about that. Yeah,
Lance Lamont 06:57
absolutely. I, the OT environment will definitely have to dive into that podcast, or you will at least have some really unique requirements, right, we want to make sure our ERP systems are up and running all the time, we need to make sure the manufacturing plant is working. So it’s a much harder environment to make sure it stays up to date. And it’s actually can be risky to date things in the OT environment. The case of I believe it was a Windows patch to port which meant it took months to get the systems back up and running after that patch was installed. So there’s lots of cases where there’s a strong drive to not make changes, not making any updates to an OT environment. But from our experience, and we’ve got lots of experience on this podcast today. There are many, many scenarios where that the obvious risk of updating is much, much smaller at the risk of not updating from a security standpoint. So I thought we could start out a little bit here and talk about some of the interesting public OT, insecurity, incidents, incidents. I think almost everybody’s familiar or of our listeners will be familiar with Stuxnet. So does anybody want to have a little refresher on what it was and talk a little bit about how it happened?
Ken Kully 08:23
It’s probably worth diving into just because, you know, we think of it as recent but we’re old. Some, some people are coming to this industry who, you know, maybe were, oh, gosh, you know, even in high school when STUXnet was a thing and it might have escaped your notice at the time.
Tyler Bergman 08:40
So I mean, Stuxnet, I mean, it was like the first case of kind of an industrial espionage and I think it was against Iran for their the centrifuges that was going on over there. So it was basically like a piece of like, Mauer that was delivered on had dropped USB into the system to try to really grenade these systems that was trying to do the, the, yeah, do the management process for these centrifuges and all that. So that was kind of one of the first case in this instance of, you know, the industrial espionage and that’s more nation state type of things of that nature going on at the time. And, and I’m sure those of you guys out there that are readers and things of that nature can go out there and read the book sandworm goes into great detail on lengths of you know, how all of that really kind of happened and some of the the technical aspects around it. Very interesting read, especially if you’re just getting into like the OT cybersecurity side of aspects. And as almost, it’s almost kind of, I don’t know, what do you guys think is it more of kind of the the initial foray of like an industrial incident type of thing where it was a malware type casement, that drencher? Well,
Dylan Stencil 10:00
it’s actually probably one of the first cases of IoT device attacks. It originated with targeting a specific controller that was used in their facilities. And that’s really never been seen before. I believe it dealt with just manipulating the process values of the controller. And I believe that’s through some firmware vulnerabilities that they found. You know, regardless of the countries in question worked with the manufacturer on that or not, you know, that’s to be determined. But regardless, you know, regardless, we’ve, we’ve started to see an uptick in those type of issues. While it’s not the most common attack threat vector, it’s definitely one that we’ve had be more aware of, in the coming years, especially with internet connected devices,
Lance Lamont 11:03
and systems. And one of the things I find particularly interesting about this one is, if I remember correctly, their devices were not directly internet connected. And so they dropped the USB stick, somebody put it in one of the computers on their network, and it managed to infect the OT devices through that. So even when we’re talking about the context of OT devices in a internet connected scenario, we have to keep in mind that these vulnerabilities or the things we’re talking about, can happen even on an entirely disconnected network, if one person doesn’t follow reasonable security protocols. And if the network itself is not architected in a way to protect the devices within the network. So there’s a lot of additional context past, just Hey, okay, I’m not connected to the internet, I’m fine, we have to think beyond the little bit of that. But Stuxnet, Stuxnet is a great example of some of the things that can be done. And there’s lots more that we can be talking about. There’s just tons and tons of reasonable examples on this, we should probably bring a few up if we want. But it’s been interesting, because a lot of the devices we’ve explored here, when we’re doing research, end up having vulnerabilities that we don’t even find any public disclosure on it at all. Some of these are really, really fascinating, we found one device that if we sent a specific packet to it would literally erase itself and reset. So if you were to be able to send that packet to that device, you would be down until you can recreate the information on that device. So it could take a long time. But anybody else else have any interesting examples of other vulnerabilities that we’d like to bring up?
Ken Kully 12:52
Well, just not so much vulnerable? I mean, it isn’t vulnerability, right, since it is a vulnerability in a sense, but I think, you know, you’ve teased out something really important, which is that, and this is something that, you know, I continue to see in engagements with our clients. It’s something that I definitely remember encountering, during engagements with clients, at previous employers. Because when you’re getting onto that ot network, and you know, like, what, what is the core of our software, right? Like, we’re trying to do device inventory, we are trying to find what is out there on a network environment, specifically an OT network environment. And there’s kind of an attendant nervousness that a lot of clients have, when it comes to doing something like that. Because a lot of the devices that they have out in the field out in play hooked up to their processes are, shall we say, not the most sophisticated, not in the sense that, you know, they don’t do sophisticated things to make to monitor and control the process. But the back end implementation through which all of that happens, might not be as robust as we would expect. You know, if I have a Cisco switch deployed in a network environment, I can be reasonably sure that I can subject it to just about any network traffic imaginable. And it has a fully implemented network stack and it can handle it. If I send it bad packets, it can handle it, if I send it malformed traffic, it can handle it. But I can’t make that same assumption about that PLC over there. You know, it probably the reasonable assumption when you encounter some of these critical devices, and OT environments, PLCs and controllers and things like that, is that they probably don’t have a fully implemented network stack. They probably only have just enough to handle the traffic that they’re expecting to handle and to send the traffic they’re expecting to send and if you send or request anything weird, there’s a very good chance that you will tip them over or like you pointed out in the case of whichever device that was As Oh, shoot, I sent some malformed traffic to it. And now I’ve just erased all my device firmware. Did I have a backup of that? Jeez, I don’t know. Exactly. And that’s the real problem with ot environments. Is that just the, the sensitivity of the devices?
Tyler Bergman 15:14
Right? The secure by design concept type of thing, right? Yeah, that’s, that’s very absent. So I got another prime example of that as well. So the so the example was given was there’s this robotic controller out there that if you send it a single ping, it automatically sends a robot into the home position. So for those of you uninitiated, as far as you know, robots in the home position, so there’s like, different knuckles on the robot. And there are
Ken Kully 15:43
You’re talking like the sort of things that would be on like a vehicle assembly line, right, all the articulating joints and Right, right, right, right, right. So
Tyler Bergman 15:49
robot control arm basically has multiple, different articulation points. So like an elbow, wrist, fingers, things of that nature. And by homing a robot, it goes to a known position. So that way, like say, if it gets kind of out of step, for one of the articulation points, you can like, hit a button on the robot and unlock the the joints. So you can match it up with the hash marks for all the particular joints. So that’s, that’s basically send it home to, you know, what it’s known status for what it’s calling, like, zero reference for all the joints, right. So just by sending a single ping out to this robot controller, it sends this robot into this home position, which may or may not be a good route for the robot to go depending on where it’s at, in this process, and things of that nature. But I was just thinking in the back of my head, like, because there’s multiple times I think, in the cases of ours, where, you know, we’re trying to do like either ARP sweeps, ping sweeps, Port discovery out there, things of that nature, and trying to be not nefarious about it, and that type of way. But we’re trying to figure out, what are some of the systems and aspects out there for inventory and things of that nature. So we got to resort to the methods that we have, ping is one of them, you know, and it’s utilized across the board and it space, ot space is not really seen as much because of, you know, those types of specific examples that are out there, and they’re still to this day, you walk into an OT landscape, and there’s like, well, you’re gonna do port sweeps across the board cares, like, oh, no, not exactly, not the ones that are used to, you know, no, we’re not gonna go in there and hit all 65,000 ports for all these devices out there. You know, there’s a couple of very miniscule ports that we’re looking for just for the specific devices that are out there. But yeah, that’s definitely not the whole range of 65,000 ports. It’s just super intrusive. devices can’t handle it, especially in the OT landscape. So yeah, it’s it was. But yes, that’s, that’s just my example of just kind of the weirdness of some things that I’ve read out there. And it was just like, wow, who does that?
Ken Kully 18:10
Which, you know, goes to show that you? This is I think, one of the reasons why, you know, having a robust industrial cybersecurity program is kind of important, because it’s, I mean, Stuxnet is kind of an outlier in some respects, right? Like, it’s an interesting example of one too, but it’s kind of an outlier, because like, it was very specifically crafted to do a very specific thing to a very specific control system implementation. And, I mean, you know, like, that’s it, it’s the, it was the sniper bullet of, of, of OT attacks, because he, you know, highly, highly targeted, designed to affect one particular thing in a very particular way. And not that, you know, those kinds of threats don’t exist, but the OT space, just the types of devices that are out there, and the way they’ve been limited in the way they’ve been implemented the sort of decisions that have been made in their implementation. And I have all kinds of thoughts about that, which maybe I’ll joke about later. Because I’m an engineer, and I know my people, but the, you don’t have to go out of your way to cause havoc in an OT environment. Oh, I
Lance Lamont 19:21
definitely agree. The example I mentioned with the incorrect packet that was sent to the device, and we would erase it, that was not a purposeful attempt just to discover vulnerabilities of the device. It was purely a one of the steps in our discovery process of interacting with the device and trying to gather information on the device. So we’ve met up with numerous others that are similar. One device we interacted with, you could send it a totally correct packet for that device. No, no nefarious aspects of it at all. But if one value was incorrect in range, but incorrect, incorrect for what that specific device was expecting able to crash the device until the device was power cycled? So, yeah, exactly. Yeah,
Dylan Stencil 20:08
I mean, there’s been cases to where just normal operations that you’re trying to perform, if you don’t perform it in a downtime window, you could possibly halt production for an extended period of time. Like there’s controllers out there, where if you’re just performing a firmware update from one version to another, you could lose your whole program. And, you know, if you didn’t save that somewhere, you’re dead in the water?
Ken Kully 20:38
Yep. And again, you know, like, we can kind of make a joke about like, oh, well, do you have a backup of that config? And that’s one thing, if it’s like a Cisco switch, but it’s, you know, these device configs? Can you even back them up? Sometimes? I mean, hopefully, you can. But
Tyler Bergman 20:50
are you? Well, that also depends on the implementation, you know, was that a third party that brought it down as a turnkey system? Or, you know, have you guys been maintaining it, maybe there’s been a bunch of program changes, you know, undocumented program changes from the initial, you know, stand up of the system to like, performance tuning and things of that nature, too. So are they doing the good etiquette of making sure that they’re backing up those changes as they make those adjustments a lot, too. So, yeah, it also kind of rolls into so you know, the ICT, the IT CIA triad that’s out there, you know, yeah.
Ken Kully 21:24
Availability, integrity, confidentiality? Yeah. Well,
Tyler Bergman 21:29
you know, we say availability first and the OT side, but on the IP side, we usually talk about confidentiality first, right. So that is the always kind of the big topic out, there’s, you know, confidentiality, integrity, and then availability, whereas like, we kind of twist it up a little bit on the OT side, saying, Well, we’re going to do availability, integrity, and then confidentiality, because we’re more worried about the availability of the system as a whole, because it’s affecting the real world processes that are out there that are going on. So that’s kind of the general dynamic that you’re running into, you know, like these controls, engineers are wanting to just go out there and buy a device, off the internet at large to put it in to solve a particular problem, and increase the availability output of their system. Right. Yeah.
Ken Kully 22:14
I mean, that was something that I remember dealing with working in, you know, a controls group for a pipeline company, up here in Canada. And they, you know, like, it was a regulatory thing, like, if they lost, I mean, sometimes it did happen that, you know, like, radios to the field are not always 100% reliable. And so you lose communications periodically. And that’s fine, as long as it’s a short outage window. But, you know, they actually had like regulatory stipulations that if they didn’t have that availability, right, if they lost visibility into what was happening in the pipeline, for longer than 15 minutes, they had to shut down and to shut the line down. So it’s kind of an there’s, you know, other industries under other regulations that impose those similar often very short time limits on just how long you can have a loss of view situation before you have to assume the worst, and bring it all to a halt. So it’s easy to see where availability becomes king in that scenario.
Dylan Stencil 23:22
Not even on that point, but like Colonial Pipeline was a great example of how even associated systems with ot devices can cause downtime just because you need, like, what happened there was that the billing system went down and just through indirect costs, and they had to decide to not use a control system right away for that ransomware attack. And, you know, that basically caused downtime, and it wasn’t even an OT device.
Ken Kully 23:56
Well, and that, I mean, that’s going to be maybe almost another episode entirely, because yeah, the the way that it’s like second order effects, right, you know, you maybe have a security incident on the IT side, but because of legal reasons, regulatory reasons, financial reasons, all of the above reasons, you still have to shut down your ot side, I totally get that. There’s a gas plant I used to work at. And it was operated by one particular company, but it had about five different companies feeding natural gas into it from their different field extraction operations. And anytime that we lost, and I mean, again, you know, up in the mountains of up in the foothills and mountains of Alberta, not the most reliable communications anytime you had a communications outage for longer than however many minutes, we had to start slamming blocked valves closed, because you don’t know who is sending you how much product and you don’t know how much product you’re shipping out to each person in turn right? So you don’t know how much to charge people, you don’t know how much to pay people, that becomes a legal quagmire very, very fast. And I’m sure colonial was probably looking at something very similar and thinking, you know, geez, like we can’t, can’t even take this risk. That’s
Tyler Bergman 25:15
all a part of those ancillary systems. Sorry, Lance. Go ahead.
Lance Lamont 25:19
No, I was going to slightly segue into a different point on the availability. Sometimes it’s just technically or mechanically important for that availability of the systems. For instance, like glass factories, where you have molten glass running through pipes, what happens if what happens if the system has failed, or they stopped pumping the glass, all of a sudden the glass starts hard to get those pipes and now you have to spend millions of dollars to manually remove or remove all of those pipes from the glass factory and then reinstall them because you can’t rebuilt the glass of those pipes.
Ken Kully 25:55
Yeah, it’s like all of a sudden, your five minute outage became a multimillion dollar capital project. Absolutely. Like
Dylan Stencil 26:01
in food and beverage, which is my main industry of expertise. Like, in the dairy industry, specifically, if you cut the tags on your panel for like, let’s say the pasteurizer, you have to get a an inspector from the state to come in and recertify it before you can even operate that equipment again. So I mean, it could be as simple as just someone randomly cutting a lockout, essentially, it’s just a wire tie. But, you know, that causes downtime, too. And I mean, people don’t even think of that as a threat vector, which is, you know, lack of training. Yeah, exactly.
Ken Kully 26:44
They’re just, you know, I’m just trying to tidy it up a little bit in here. This was messy. Well, yeah, but also just cost us a lot of money.
Dylan Stencil 26:51
Yeah, or you’re troubleshooting an issue and you know, someone’s not trained, hey, do not cut that tag. And next thing, you know,
Ken Kully 26:58
exactly, gotta shut down the equipment. And that comes in costs, because, you know, if you ain’t producing, then that’s, that’s just money you are not making and that can’t I mean, you know, that can be depending on exactly what your industry is like, that can add up to significant cost if you trip a boiler, and it’s going to take 36 hours to recycle that boiler. Well, that’s, you know, the per megawatt hour price of power times what the megawatt output of that boiler is times 36. And you might not make that much in a year. Right. So, and it’s so the, the other thing, too, that kind of gets me is it’s sometimes really easy to even in the course of like normal operations or balancing between, like, what’s a normal operation and corporate requirements, and maybe even a regulatory requirement, you sometimes come into these situations where you end up causing issues for yourself, just by trying to be in line with all of this, it’s less of a problem. Now more and more control systems are going virtual, more and more control systems are going to high speed gigabit or more networking. But you know, even as recently as 10 years ago, and 10 years is a long time in an IT sense, but in an OT sense, it’s barely, you know, it’s barely starting to mature stuff will stay in operation and IT environments for a long, long time. If you installed that system 10 years ago, well, it’s probably still there, it’s still doing its job. And it’s doing it just quite nicely. Thank you very much. But, you know, I can think of control systems that didn’t have a virtual implementation. So they’re all on, you know, physical servers, physical workstations, and for which network communications are locked to 100 megabit. Now for controls traffic, normal operations, that’s totally fine. But now what if you have a requirement whether it’s corporate or regulatory, to capture periodic backups, so now you have to situate you know, a backup server somewhere that can reach all of these network devices, all of these endpoints, and capture full or incremental, or, you know, both depending on your scheduling backups of those devices, over the same 100 megabit pipe that all your controls, traffic is moving, doing that without stopping on your controls traffic, and essentially, inadvertently causing yourself an outage, a self imposed denial of service can be very, very tricky. And I’ve seen it happen more than a few times where, you know, I’ve been in a facility and they’re like, Well, we have this issue. And this alarm set comes up where we start to get like, you know, our primary and secondary servers fall out of sync, and we start to like, lose bits and pieces of control data. And it always happens on this particular schedule. And then you kind of do some digging and you’re like, it happens to exactly align with the schedules that have been defined on the backup server. Well, geez, I wonder.
Dylan Stencil 29:51
Yeah, I’ve, I’ve had experience and backup and recovery tuning before and it’s interesting to see more Most of the time, when it’s implemented, you don’t have the correct configurations for the environment. And you got to kind of feel it out, try to play it as safe as possible. But like, one good case is SQL database servers, you know, we’ll get into the IT side sometime down the road, but like, basically that could, that has all your historian data and everything like that associated with your processes that are being used every day for internal and customer facing applications. And, you know, you have to just watch out for these do’s and don’ts. Like, that’s a touchy server to deal with, or like, one of the scatter servers, you got to watch out because the network traffic can just get monstrous,
Tyler Bergman 30:48
and I’m just over your thing. It’s like, man, 100 bags, I was a luxury back in the day. Radio, radios, yeah, I’ve
Dylan Stencil 30:58
been having issues even with 10 Gig networks, you know, just with the traffic, if you have a large set of applications you have to deal with. I mean, there’s, you have to like stage them correctly. And everything. Also aggregation.
Tyler Bergman 31:14
Yeah, no,
Lance Lamont 31:14
yeah. Oh, sorry. You go last? Oh, related that some of the devices we’ve researched within the research team here have been probably 90s and 1990s. Era? Oh, no surprise. Yeah. And they’ve been devices that don’t have a network port. So it’s just, yeah, we’re just using the serial port through an Ethernet adapter. So both we have fun bandwidth challenges there. But also, we’re now connecting devices that have no concept of the Internet to a network. Encryption.
Tyler Bergman 31:49
Oh, by the way, we’re just going to tunnel this serial session over Ethernet. And it’ll be fine, of course.
Lance Lamont 31:54
And
Ken Kully 31:56
it’s like, well, we’ll just put some bump in the wire encryption in there. And I’m just and it’s like, Well, okay, but you know, like that will introduce, you know, a millisecond of latency. And our process is not even that tolerance of latency. Ah, okay. How do you deal with that, jeez, I’ve been in facilities where like you couldn’t even use like you actually, they specifically had to order like, those Panasonic Toughbooks, that you can still get with the serial port, because certain devices that they needed to connect to, like, if you use the USB adapter, just even going through the step of you know, like the USB, essentially, the USB driver was existing communications latency between the laptop and the end device.
Tyler Bergman 32:45
We had a lot that on the protection relays, you had to buy a very specific chipset out there that would allow that. Otherwise, it did that all day long. Like you’d be in a protective relay, and it just kick you out. You’re just sitting there going, what’s going on here. And that’s when we figured out like yeah, there was like specific serial chips that you had to do on those USB adapters because of the conversion process. Like wow, yep. Unreal. So,
Ken Kully 33:09
you know, just goes to show again, that your, your average ot environment, you know, like, I mean, yes, if, if you need, you know, if you are a nation state, and you need to very specifically target one specific process, obviously, you’ll use something finely crafted like Stuxnet. But there’s a, there’s a lot of damage and chaos that can be caused with not even a 10th of the sophistication of that particular piece of malware.
Tyler Bergman 33:34
And that’s kind of the thing. Yeah, it’s like across the board, you know, so realistically, and this is kind of what I’ve seen, you know, over a course of a decade is like, you got all these IT folks, as you know, finally gotten up to very, very maturity levels of their security. So they’re trying to take these larger security tenants over to the OT side, right, that they’ve been running for, like the last decade, that they just kind of figured out through like the 90s, and 2000s, everything like that. Hell, I went to school in the 2000s. And we still have like a lab that will have all public IP addresses on it to be more to well, it was fun, because you know how I could connect to a printer up in the computer lab and call it the lab monitors like, Hey, can you staple this and deliver this over to Professor Chris and all that. So it was cool back in the day, but like you think about that today and you’re just like wow, that was wide open. Terrifying.
Dylan Stencil 34:37
Commonly nowadays, what you would find that similar in my opinion, is just every OEM that comes into a factory you know, they have some sort of remote access solution. Now if you get into the more security mature companies, they normally have a VPN provided and authentication everything tied into, you know, each contractor with their permissions that they need to have for their devices. But otherwise, you just have a bunch of random VPN servers running on your network. And I mean, that can cause all sorts of issues. As in Oh, sorry. laughs
Lance Lamont 35:20
I was just gonna say, hopefully, we you know about all of the VPN servers that are running on your network. You
Dylan Stencil 35:25
might not even know that or something’s running like a cellular BACnet. There’s all there’s all
Tyler Bergman 35:31
this box.
Lance Lamont 35:34
Oh, that’s just the one I were playing WarCraft on. Well, I’m waiting for the Union finish.
Tyler Bergman 35:40
Oh, well, the prime example. I think this was one that was gonna go out there is like the the whole VPN or cellular everything like that, you’ll go around and be like, Hey, what’s this box? And nobody knows. And then you eventually find out like, oh, yeah, that’s the cellular backhaul for the vendor and all that. It’s like, well, we didn’t know that was even there. Wow. Yeah, that’s a fun one, too. And those
Dylan Stencil 36:00
devices, especially in industrial, if you don’t know about it, it’s not getting updated, either. And a lot of these devices aren’t really built with security in mind, because it’s coming from a vendor in the space where confidentiality isn’t the priority. And that’s
Tyler Bergman 36:19
when you go out and find it on showdown or something like that. Right?
Ken Kully 36:22
Yeah. Oops. Yeah. Well, I mean, it doesn’t help to that, you know, like we’re seeing, it’s definitely gotten better over the course of, you know, my time in, in the industry, you know, specifically in ICS. Since about 2008, it’s definitely gotten better. But it is still very much the case, unfortunately, that, you know, the people who are in the plant, and who are, at least on the org chart, responsible for the system, and increasingly for its security, they’re not necessarily, you know, cybersecurity experts, that’s not their training, maybe they were just a plant electrician before, right. And, honestly, they really just wish they could go back to being the plant electrician, changing out the light bulbs, and whatever else. But they did something that kind of got them flagged as okay, you know, computers. So great. Now, you are our NERC CIP resource, you are our plant, ot technical person, whatever the title might actually be, whatever the designation of the role might actually be.
Tyler Bergman 37:28
You fix the time on the VCR? Sure, exactly.
Ken Kully 37:32
You know, Bob, the electrician is my, he, I’ve met him so many times, right? He’s the plant technical contact, but honestly, you know, you catch me in a private moment, he wishes he could just leave it all behind and go back to, you know, doing electrical work in the plant.
Dylan Stencil 37:51
And a lot of times, you’re doing that on top of your truck or two, right? I mean, the workload you have is just crazy. And that’s why a lot of this work has to get farmed out to integrators, machine builders, etc. Because, you know, the, the plant engineers are having to deal with a totally separate set of problems. So you have to be a project manager on top of that.
Lance Lamont 38:15
I do want to point out that Bob, the electrician, at least the ones that you meet, can are doing something, right. They’re looking for help, they’re talking with people outside the company, and recognizing that they can get help, they can contract with others, whatever they need to do to get things done. Yes.
Ken Kully 38:33
And they’re also being supported enough that, you know, they can actually be having those conversations, which, you know, I mean, they’re usually, at least, when I’ve tended to have those kinds of conversations, they’ve usually been in the context of, you know, me showing up on site. And not just when I was, you know, not just as a verb employee, but you know, past past careers as well past lifetimes as well. I’m showing up because enough people have said yes, that there’s budget to bring people in to, you know, either take what’s existing and try and do the best you can with it, or implement a, you know, some sort of security solution implement some sort of assets discovery solution. They’ve, they’re willing to have the conversations, they’re willing to talk about it, they’re willing to get that help. And then they have the people who actually control the purse strings, willing to listen to them, and say, Yeah, okay, we can approve some budget for this.
Tyler Bergman 39:27
And that’s a totally different conversation to like, when you’re walking into a plant, and it’s like, well, you’re not from it, are you? That’s probably a whole episode on its own. stories a
Dylan Stencil 39:40
little bit better when it’s not it walking in there.
Ken Kully 39:44
Definitely helps if you can speak, if you can speak the OT language a little bit, you know, if you can actually communicate an understanding of some of those sensitivities that we’ve been talking about all episode, right, the fact that you can’t just bombard every device with traffic.
Dylan Stencil 39:58
Yeah, and I think that’s definitely one of the Strong things about a lot of us, here’s that we have. We’ve been in that spot and we know how to deal with those situations.
Lance Lamont 40:11
I was actually going to say maybe a future podcast could be, how do you have that conversation with management? How can you invite the whatever contractor you select or help get that selection? I think that could be a really good, you know, how to approach your management to, to pursue better ot security?
Ken Kully 40:28
Absolutely. That’s, that’s probably a podcast series. Because there’s so many different conversations, right? Like, you know, how you have that conversation to, you know, say, going from your plant technical contact to your plant manager, like, that’s one conversation, but then the plant manager, going to whatever director needs to actually sign off on the expenditure is a different conversation in a different language, again, absolute into
Tyler Bergman 40:56
Season Two.
Dylan Stencil 41:00
They all have different priorities, when you talk to each one of them, that’s definitely one thing you end up finding out.
Ken Kully 41:07
Yeah. And then also navigating the, you know, navigating the discussions about like, well, who’s even going to handle that, right? Because I mean, different companies have different organizational structures, sometimes OT is its own thing. Sometimes OT is, you know, essentially a sub department of it. And so in symbol, do we want to bring in outside expertise, do we just want to use our in house expertise, and, you know, not that you can’t use your in house expertise, but the rules are very different in it versus OT. And so, you know, that is something that needs to be, and that’s another episode again, is just very carefully, because man, like, you know, if you just come into the OT environment, and try to do it the way you do it on the IT side and apply all the things you would on the IT side, it can get disastrous quickly. That’s why
Dylan Stencil 41:57
you’re mostly seeing, like, ot managed services starting to become a thing to where it needs to rely on groups that have that specific experience. And, you know, it’s becoming more and more common for places that don’t want to build up that internal infrastructure to, you know, rely on having a group bridge that gap for them,
Tyler Bergman 42:24
or they just don’t have a group that has that conceptionally. You know, and I’m sure you guys have ran across the same thing as I do, you have these CISOs, that are just, they may be laser focused on patching, you know, across the IT organization. And patching is great across the IT organization, because if you reboot a computer there, it’s not as impactful as if I’m going to go restart my server that’s controlling all of my processes in the plant, you know, it’s a totally different concept of, of, and how you convey that message out there. And the implications of those things to write. And a lot of them is just knowledge that like transfer,
Dylan Stencil 43:04
like endpoint protection is a good one where, you know, someone might have a standardized Endpoint Protection application to deploy. But if you’re deploying it in an OT environment, and you’re not just, if you’re doing the act of protection, I mean, you could hamper production with some of the software. So it’s just, you know, understanding those nuances.
Lance Lamont 43:26
And even beyond that, many it situations, you can just simulate into the small test lab someplace like this is one of our typical installations. This is a typical server, we’ve spent X number of 1000s of dollars on this system, but it’s normally in the, you know, five figures, not seven figures. And you just can’t, in many cases, you just can’t afford to build up a full test network. Unless you’re for ot scenarios unless you have a very large budget. Individual dice devices we’ve done research on can take over a million dollars for a single device to be set up and configured correctly. So it can be super expensive, and you just can’t do it to test those patches before you put it in production. That
Ken Kully 44:12
even presumes that they have a test environment.
Lance Lamont 44:17
argument saying they probably don’t have a test environment. I mean,
Dylan Stencil 44:23
I mean, emulators can only do so much. Absolutely. You’re lucky to even have one device. Yep,
Tyler Bergman 44:31
of course, I’ve got a test environment. It’s also called production.
Ken Kully 44:36
Engineering workstation over there. Yeah, right. Exactly.
Lance Lamont 44:39
And I think we could do this whole thing in one hour of downtime, so don’t worry about it.
Ken Kully 44:47
Yeah, that’s another thing entirely.
Tyler Bergman 44:50
Yeah, that does happen out there. Well, and it’s just kind of the example of those like low bandwidth radios out there. You know, you’re trying to do like a firmware update on them or something like that, you know? I typically try to typically have a spare one in hand out there. So if it does go brick your radio, you can swap it out really quick and all that. But yeah, it’s there’s just a lot of moving parts and pieces that kind of go into it, especially the regulatory piece. You know, if you lose visibility out there on a particular systems or regulatory requirements as far as what you need to have on hand, when you lose visibility, I know like wastewater, there’s DNR things just like what Ken was talking about earlier, for the components that if you lose visibility on it, you must maintain power at that local facility to be able to maintain, you know, the effluent out for the wastewater facilities. The same thing across the board, multiple different industries, as well as different applications out there. So you got a large regulatory aspect that kind of rolls into it as well, that you got to keep cognitive. But that ultimately drives all the availability of all these things all the way around. So it’s, they’re very, very cautious when it comes to any type of change out there. I think they kind of coined the the phrase of you know, if it ain’t broke, don’t fix it type of mentality. And that’s the way they operate. Well, and
Ken Kully 46:11
just to kind of bring it back to that opening example, right? Like STUXnet was was tax, that was an interesting example, too, because fundamentally, it was an integrity problem, right? Because the whole idea was that it was it inserted itself between the the centrifuge controllers programming workstation and the centrifuge controller. And it was designed to, in such a way, if I’m remembering this correctly, was designed in such a way to basically take whatever the program that was being given to the centrifuges, and just kind of scramble it, randomize the commands, because you know, when you have a high speed motor of any kind, whether it’s you know, centrifuge or your car, paper machine, whatever, the usual way you want to run that is you want to just sort of nicely ramp it up to its max RPM is run at at max RPM, for however long you need to do the thing that you’re trying to do, in this case, settle out uranium 238, from 235, if I remember correctly, and then ramp it back down, the last thing you want to do is have it like be instant on then like at half speed, then kind of stop for a second, then shoot back up to half speed, then stop full speed. Right, that will get very smoky, very quickly.
Dylan Stencil 47:26
Tuning pump curves comes in.
Ken Kully 47:30
But that is the point that STUXnet was, you know, designed to basically do this all silently. So the commands that the operator put in, were not the commands, at least not in the same order that reached the centrifuge. But then what was coming back to the operator was also not what was actually happening. Right. So becomes an integrity issue. And, but I mean, you know, that only, that only works, because you’re maintaining that appearance of the data availability, you’re maintaining the appearance of view and of control and of normal operations, which is still kind of it is that King in just about any ot environments, you need to maintain that availability, and hopefully, in the data you’re getting back is good. And if it isn’t, that’s another problem that you have to address. But if you’re not getting it in the first place, that’s the show.
Tyler Bergman 48:19
Yeah, totally different aspects are, because you want the availability of the data, but the integrity of the data, because it’s also probably driving regulatory decisions out there, too. So like in a admissions monitoring environment, you know, that’s some brick, revelatory components that you’ve got to go out there. So you got to be extremely sure of that data to write that,
Ken Kully 48:40
you know, and that’s, that’s another episode in and of itself is Sam is critical infrastructure. Is Sims, should we be considering Sims critical infrastructure? Because in a lot of the engagements I’ve had over the years, continuous emissions monitoring is, well, it’s it’s a thing. And you know, like, what kind of taking into account but we don’t really like it’s not necessarily under the, you know, the facility under the radar. Exactly. It’s not usually like on the radar for like, oh, well, this is part of our NERC CIP compliance. That’s a critical asset over there, really. So if you lose that, how long can you operate without emissions monitoring? Ag law say?
Tyler Bergman 49:17
So, I mean, it’s under the regulatory purview. Right. So I mean, you’ve got certain permitting limits that you can go up to. So if you lose visibility into what your output is for your permitting limits, well, guess what? I gotta shut down until we figured this out over here. So
Ken Kully 49:32
yeah, but it’s not a critical asset. Right. Yeah. It I
Tyler Bergman 49:36
know, it kind of varies from state to state and all that stuff, too. So, yeah, it probably needs to be under purview because and I just had the same conversation the other day, and it’s just like, well, you know, you guys do have this out here. And you do have this monitoring capability, but it does have impact. And I think that’s where it really comes down to when you’re going through like the risk assessment and things of that nature is you know, what is facilitating impact to your processes. And how does it equate out monetarily? For your systems? Like if I’m not able to produce electricity because my emissions monitoring gone down? Well, you know, it’s pretty impactful, you know? Yep. But yeah,
Ken Kully 50:20
that’s also no signal. That’s another episode. But it’s also you know, that gets back to Bob, the electrician right now you’re expanding his scope yet again. And he’s already kind of tapped out, and he doesn’t understand anything of what you just said, anyways. So what do you do?
Lance Lamont 50:33
I think the summary is go to again, Bob, it’s okay to ask for help.
Ken Kully 50:38
Yes, it is absolutely. okay to ask for help, please. Well, we’re coming up on the hour mark. And that seems like a good opportunity to bring it to a close. But yeah, it just kind of goes to show like, why does this stuff matter? Well, you know, whether it’s power, or food, or wastewater, or paper, or whatever, you know, O T, is hooked up to a lot of stuff that is really, really essential for the smooth and orderly operation of everyday life. In, you know, wherever you happen to be, things get really bad really fast if you don’t have power. And if you don’t have water, and if food production falls off, these things become problems, and very quickly. And it’s a challenge, because, oh, tea is also very fragile, or it can be it’s the devices that exist within these environments, which are often hooked up to critical processes are not necessarily as robust. I mean, maybe environmentally hardened, right? You know, they can, they can stand up to a torrent of, of ash or dust or whatever. But from an electronic standpoint, they’re fragile. They tip over easily, they get the wrong kind of traffic, and they can delete their whole config, disorderly tours, solar flares, regulatory requirements, there’s all manner of things that make the successful operation in anything resembling a secure manner of an OT environments. Very, very challenging. And yeah, maybe that is the best takeaway is that you know what? It’s okay to recognize that you can ask for help. And that you probably need to ask for help. And it’s okay to ask for help. And for budget, because the bottom line looks a lot better when the power plant stays on. When the milk pasteurizer stays on, when the paper machine keeps running, the bottom line looks a lot better when those things happen. And if even something inadvertent causes disruptions, the bottom line looks a lot worse. That was my closing thoughts. So Lance, any other closing thoughts from you?
Lance Lamont 52:47
I think you summarized really well, obviously, as you mentioned, there’s all sorts of crazy failure modes for these devices, once you start connecting to the, to the network, but we need to get them on the networks for the data you get out of him. But so it’s it’s a very unique challenge set.
Ken Kully 53:05
About you, Tyler. Yeah,
Tyler Bergman 53:07
it’s the same thing. You know, it’s all about the monitoring and kind of know what you have out there. And I think that’s kind of the major tenants that we’ve also realistically see across the board is, knowing what you’ve got out there. That’s, you know, across multiple different cybersecurity frameworks, that’s like first steppers is known what you got out there. The other component is, you know, being able to kind of effectively monitor those systems for things and doing backups, things of that nature. So it’s a lot of the same tenants from the IT world that we’re just trying to apply on OT. But you’ve got certain different boundaries that you have to abide by then, when you’re trying to apply those tenants. So it is an interesting venture. And I think there’s plenty of people to help out there, especially at birth. We can definitely help you out with those aspects and get you going there. And
Ken Kully 54:00
Dylan, any last thoughts? Yeah, it
Dylan Stencil 54:02
was an honor and privilege to be on this first episode here and really enjoyed the talk. I feel like there’s a lot of knowledgeable people out there. And there’s a lot of knowledge to be shared. And, you know, I’m just looking forward to seeing what we can share with the world here. And, you know, there’s a lot of things to talk about that don’t get talked about, like a lot of the times you can go into Stuxnet or wanna cry or not Petya or any kind of attack, but like the number one thing is just being able to understand what’s going on, and just starting out with those first baby steps that a lot of places have to get to and I feel like that’s the most important part. You know, people shouldn’t be focusing on step three or four when they aren’t even at step one yet. So I feel like this is a good talk. Brilliant.
Ken Kully 55:00
Well thank you all for taking part in the pilot episode and thank you other than anybody who listens. Until next time this has been ot after hours podcast about industrial cybersecurity because it does matter. And we would like to help you get through those challenges.