Platform Engineering on the Edge: NIS2, ransomware and reality checks

[Pinja] (0:03 - 0:10) Asking these questions for ourselves, how can I monitor this? What can I do about it? Can I identify if something has happened to it?

[Darren] (0:14 - 0:22) Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development.

[Pinja] (0:22 - 0:32) Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna.

[Darren] (0:38 - 0:50) Welcome back to the DevOps Sauna, where we're continuing our short series on platform engineering on the edge, and we have an exciting topic today. But before we get into that, I'd just like to welcome my co-host, Pinja.

[Pinja] (0:50 - 0:52) Hey, how are you doing, Darren?

[Darren] (0:52 - 1:03) I'm doing reasonably well. Pinja's now looking at me because we actually are doing a second take of this. And my answer was much more positive in the first one, but then we ran into a technical issue.

We had to restart.

[Pinja] (1:04 - 1:30) No, but we're jolly people. We can have a second take all we want. But this is the reason why we're also laughing, because this is a topic really close to Darren's heart, I would say.

And we do need to, of course, come up with an abbreviation or something. This is the third installation in a series. We're talking about platform engineering on the edge.

I don't know if we can say P-E-O-T-E, or even platform engineering doesn't have an abbreviation yet, but this is the third installation. We're talking about security today.

[Darren] (1:30 - 1:52) Yes, we are. Security of edge installations. And I think the question that people are going to be asking is how exactly does this differ from security in anything else?

So if we're talking about cloud installations, if we're talking about data centers or server rooms, how does this differ? And I think the problem we have is proximity.

[Pinja] (1:53 - 2:20) Yeah. And in the previous episodes we've been talking about platform engineering on the edge. So in the first episode, we talked about what it is, the 101 of things.

We've been talking about the case stories and the use cases of it. So really, when you say Darren, that proximity is the key here. So it might be actually in your kitchen, it might be your car.

There are so many things that are related to IoT nowadays that it can really get personal to you.

[Darren] (2:20 - 3:43) Yep. And that's entirely the thing that the closer these devices get to you, the more personal it becomes. Because if we look at like your average attack, someone has a server somewhere, perhaps it's hosted on AWS.

They attack it via the network and they attack it via a web interface and no one takes it personally. It happens because it's just the nature of things on the internet. But you bring up cars and cars are a fascinating example because it's where the lines kind of blur.

There's a signal forwarding attack, which is used to open and start cars without the keys, which uses basically two radio devices that pick up and transmit signals to each other. You put those relatively close to the keys and one relatively close to the car, you open car doors, you can start the car. And this is like blurring the lines right up to your front door because it's very common for people to keep their car keys by their front door.

So it's like we have that bit of attack surface, that what we call proximal attack surface, a well-known thing that people aren't discussing. And this is actually the state of what we call OT or operational technology security. It relies on people not understanding or people not being able to figure out attacks because it's uncommon, but they're getting more connected now.

So these attacks are becoming common.

[Pinja] (3:44 - 4:04) And you're now mentioning the car key. So the physicality of this thing is also, I guess, crucial here, but there's a new attack surface. So is it something that you can use?

Because we are always warned about the USBs, right? The USB sticks do not use as if you find one. So is this somehow connected to, I'm not saying USBs per se, but the physicality of things.

[Darren] (4:04 - 5:14) I'm trying to actually see if I have one on my desk. I quite often do. There's this microcontroller called Adafruit M0 Trinket.

It's big enough to fit inside your average USB drive. And what that does, it runs Python as a programming language. So you can have it execute things whenever you plug it into a computer.

And another thing it can do is pretend not to be a USB stick, but to actually be a human interface device. So it can pretend to be a keyboard. It can connect to a computer if someone picks it up and is curious about what's on it.

It can then pretend to be a keyboard and type in like either a curl command to download something or write a backdoor directly into a command line. So you get this interesting attack vector that even bypasses a lot of antivirus and malware protection, because malware protection is rarely applied directly to the keyboard. So yeah, this is exactly the same.

USB sticks are another great example of physicality being abused. And that's essentially what we're talking about here. I think we can come up with some more good examples too.

[Pinja] (5:14 - 5:29) That's true. So if we think about, for example, the network access, and that has now been more open and we have Wi-Fi. So now that's plugged into, for example, in my house, it's not secret information that my TV is connected with my Wi-Fi.

[Darren] (5:29 - 6:13) Yep. Again, it's part of the rapidly expanding attack surface where you have, if we talk about 25 years ago, you had a computer connected to the internet, maybe. And now we have 15 devices in every household connected to the internet.

We have different security metrics on various access points. Often they are insecure and out of date because people just don't understand them. So you have this attack surface rapidly expanding in a way that I think people aren't really attempting to compensate for yet.

They are kind of pushing the responsibility of this security onto people who aren't maybe qualified to understand it. They're pushing it to the end user.

[Pinja] (6:14 - 6:33) And because I'm just thinking about the example here, if I think of myself, for example, now, I'm not a security expert. It's not something that I know the very basic principles of, of course, but at the same time, we're now, as you say, targeting people because it's very personal. It's very closely related to us, which makes it, I guess, more dangerous here.

[Darren] (6:33 - 7:15) And at least people will feel those attacks more. Yes. For example, now we're just talking about what's called IoT, Internet of Things, but we have other examples from OT, which is operational technology, and that's used in industry and infrastructure.

And one of my favorite examples of that are the ATMs, the automated telemachines or transaction machines. I've never actually figured out which, but there's this technique called jackpotting, which is basically that you turn the ATM into a, well, I should say it's a set of techniques because it's never one thing, but you turn the ATM into the output of a slot machine. You just get it to dump all of its money out.

And if that sounds problematic, you're right.

[Pinja] (7:16 - 7:36) Yeah, don't do it. This is not us encouraging you to try this out or anything. Don't get anything into your head about this.

But I guess there are multiple attack points here, right? Because we're thinking about the physicality of this ATM again. I'm using my card, I'm putting it in the slot, there's a card reader, but it's also connected to the network, right?

[Darren] (7:37 - 8:53) Yeah, of course, there's a load of different attack vectors here. And that's why I like bringing up ATMs, because they're an extremely good example of this. So as you mentioned, there's a card reader, but there's also a keypad.

So you have two interactable items, which are connected to a computer. The computer, you will rarely see the operating system behind it. But if you can, you will notice that quite often they still run on things like Windows XP, and they still run on 32-bit versions of Windows XP.

And they are vulnerable to attacks such as Rowhammer, which 64-bits stop being vulnerable to. And then even if you have a very secure front end for this ATM in a very secure enclosure, it's still attached to a €50 network device provided by the supermarket which hosts the ATM. So all kinds of attack vectors for this machine that's been in use for 20 years, that is actually an indication of the problem with security on edge devices as a whole.

This is the one that we can kind of sympathize with because we know it. But imagine this issue multiplied across entire industries like energy, like any kind of infrastructure.

[Pinja] (8:54 - 9:06) And should we talk about space for a moment? We covered space in our previous episode on platform engineering on the edge in the use cases. So we can talk about satellites, and they communicate through radios, don't they?

[Darren] (9:06 - 10:15) Yep, most satellites use some kind of radio communication. I don't know that I can say it's all satellites, but I'm pretty sure it's the majority. For example, last summer when I was bored during my summer holiday, I was using...

There's this device that they're adding called the Software Defined Radio, which basically allows you to plug a USB stick style device into your computer and tune to a frequency using software. And doing that, for example, there are the NOAA satellites which transmit weather data, and you can actually download the photos they're taking because they just send them unencrypted through radio, which is absolutely fine for this use case. That's what they're supposed to do.

But it's made listening on broadcasts possible, and there are a lot of events now spawning about security in space because the idea that transmission is becoming possible is problematic. So in this case, all of these devices have so far relied on this terrible principle called security through obscurity, which is basically that people assume that others won't access and not that they can't.

[Pinja] (10:16 - 10:34) No, exactly. And we're assuming quite a lot here, like that people have the best intentions, that people don't notice about things people couldn't access, as you say. So this is also that we would like to think that physicality is adding a layer of security that doesn't exist outside of IOTs and observation, right?

[Darren] (10:34 - 11:22) Yeah, basically, people, there's still this assumption that if you just leave a service open, it won't get hacked. No one will target you. No one will take a run at you.

No one is interested. And I actually have two demonstrations I always put into client meetings when I hear this. One of them, I just spin up a service with an empty, like a simple SSH root password during the meeting and count her time, how long it takes it to get compromised.

It's usually not that long. And the other is going to ransomware sites and scrolling down the, there's this ransomware live, I've talked about it before, and it just shows a list of people who have been hit by ransomware attacks to show them that it's companies they've never heard of who are being hit most of the time.

[Pinja] (11:23 - 12:14) Yeah. And I can think of an example of us trusting people and assuming the best of intentions of people, go ahead and leave your laptop unlocked in an open office and see what happens. Okay.

I just, it's a very simple example here, but there have been a couple of incidents in the past couple of years, even on IoT security. And it was only last year in February, 2024, when there was a healthcare IoT ransomware attack, and it was targeted at IoT connected medical devices in many U.S. hospitals. So they gained control of, for example, monitoring systems for patients, some infusion pumps and even MRI machines.

So they, the hospitals, had to revert to manual procedures because of this. So we're actually talking about big things here and it's not just somebody leaving their laptop open on a chance.

[Darren] (12:14 - 13:18) Yep. That wasn't even the first instance of medical attacks. They have a history at least since 2016, the WannaCry attacks in UK hospitals.

And there was actually a very interesting paper on embedding attacks into the image that's taken from medical imaging devices to hijack the device itself by manipulating the image it takes. So it's like, it's a fascinating field of study. And a lot of these things are discovered without malice.

They're discovered by people who just find it interesting. But then once they're discovered, they are very often used by people to gain some kind of monetary advantage. So I don't understand where the idea that people can be trusted comes from.

It's kind of weird that I was saying that because like if you drop your wallet somewhere in Finland, I think you have a reasonably good chance of having that wallet returned mostly with all the cash intact. So it's kind of weird. Maybe Finns can be trusted.

[Pinja] (13:18 - 13:46) Could be. But we can also talk about an attack on a smart city. So a distributed denial of service or DDoS attack was targeted at our smart city in Asia.

This was also last year and they crippled the IoT systems that were responsible for transportation, utilities and public safety measures as well. So for example, traffic lights, surveillance cameras and some waste management systems were all taken offline during this attack.

[Darren] (13:47 - 14:17) Yep. And the whole point of DDoS attacks is they are actually strengthened by the IoT. So zombie devices being able to add their traffic is actually kind of the tipping point here.

We actually saw when we were researching this, we saw that security attacks on IoT devices increased by 107 percent in early 2024. So people are starting to understand the value of compromising the devices that are close to you.

[Pinja] (14:17 - 14:37) Yeah. And if we think of, for example, the one of the most used systems like Microsoft devices, they really enhanced security measures, but it's not perhaps the same case with IoT yet. So maybe that's one of the reasons why we see this rise 107 percent in just one year compared to the previous period on the IoT attacks.

[Darren] (14:37 - 15:12) There's also the matter of cost. Like a lot of these embedded devices are designed to be sold as cheaply as possible. If we're thinking like temperature sensors, they don't really have the budget to run security.

But things are actually happening in this space. So there's a NIS-2 obviously in the CRA, the Cyber Resilience Act. I believe the wording that I've mentioned before is every networked device.

So I don't know if that's actually going to come to pass, but the idea that all of these would now be subject to this kind of control is to me kind of good news.

[Pinja] (15:12 - 15:44) Yeah. This is something that many of the EU regulations, for example, they are targeting securing, let's say the common people. So perhaps not the experts, as we say, these are devices that are closer to us.

But at the same time, we don't want to go down this road again in this podcast, that it would be all doom and gloom so that humanity is now destroyed because of this, I guess. But there are a great deal of things that we can deal with, right? So how can we protect ourselves against the cyber attacks in the space of IoT and on the edge?

[Darren] (15:44 - 16:43) I think there are some questions you should be asking yourself. And I've actually outlined these questions in a blog post before that you can find on Eficode's website. But you first need to know how you're handling encryption.

So there are two types of, or let's say two areas of encryption, encryption at rest and encryption in transit. So you have to acknowledge the responsibility that your device is sitting in close proximity to people. And you have to accept the security responsibility that goes hand in hand with that.

And that means that all the data you have should be encrypted. It should be encrypted both at rest and in transit. And that's actually kind of a tipping point, but not a tipping point.

It's more of a, it's a difficulty because if you have the device, cracking things on it is usually not difficult. So having some kind of trusted platform module in there to help with encryption is, in my opinion, a valid, valid defense.

[Pinja] (16:43 - 16:51) Yeah. So when we think about transferring, perhaps there are some very specific things you can think about when the transfer is going?

[Darren] (16:52 - 16:56) Yeah, but it's nothing that hasn't been on the discussion since like 1998.

[Pinja] (16:57 - 16:58) Nothing new though.

[Darren] (16:58 - 17:18) Yeah. SSL gave way to TLS, which is the new standard. But the whole point is everything in transit should be encrypted and it's not difficult to do.

The fact that it's not, by default, the case for a lot of these devices is actually kind of a depressing viewpoint, but it does mean that the steps to remedy it are easy.

[Pinja] (17:18 - 17:31) So are we monitoring enough or distributed IIoT devices? Because I think if we think of the larger weaknesses and the physical distribution around that, can we do something on that front?

[Darren] (17:31 - 18:17) We can. And that's actually where we get this whole platform engineering on the edge idea, because this idea of tying everything into a central system requires that the distribution be done via a central system. Usually we do GitOps in these cases, which means the devices can pull and deploy by themselves, but there's no reason outside of air gapping that telemetry should not be returned.

So having a robust platform that understands its responsibility towards maintaining connectivity to these devices and gathering health data on these devices is critically important, including what we've discussed before, which is physicality. You need to be able to know if someone has physically tampered with your device remotely.

[Pinja] (18:18 - 18:44) IIoT is not perhaps the most traditional infrastructure and it goes on many fronts, but we need to figure out all these safety measures. In a very simple case, as you say, asking these questions for myself, how can I monitor this? What can I do about it?

Can I identify if something has happened to it? But maybe thinking about the attack surface now on IoT. So how about the network restrictions, for example?

[Darren] (18:44 - 19:40) Yeah, very much so. The network is kind of critical, because if you think about traditional networking, we have an enclosed group of servers behind a locked door with network cables running to it. If we think about the move to cloud, that's pushed everything outside so that you only had networks.

And now bringing everything back to the edge, we're talking about devices close by. So we're talking, if we think about, say, a manufacturing plant, you might have sensors on each piece of machinery, which tells you how it's running, how many items it's processed, that kind of thing. Multiply that by 12, because you have 12 different machines on that assembly line.

You then have 12 connected devices to a central hub, for example. So there you have all this spread out information and you both expand the network attack surface and the physical attack surface. So you basically just have to ask yourself how you're running your networks.

[Pinja] (19:40 - 19:59) So coming back to the, can we remove unknown unauthorized devices, I guess, can we figure out who's actually using that network? But if we're wrapping up this discussion today from a security perspective and tying this back to the platform engineering, what can be done from the platform engineering perspective, specifically on the edge?

[Darren] (20:00 - 21:01) Yeah, for platform engineering on the edge, hopefully this is the last time I have to say that particular statement. The whole purpose of platform engineering was abstraction. It was designed to separate developers from all the things they don't need to know.

Maybe in an uncharitable way, you could actually see it as the breaking down of DevOps, where DevOps was, you know, you build it, you run it. And then it became, you build it and you don't know how to run it. So then it's more like you build it and the platform runs it.

And that means the platform has to cover everything. It's very easy to see that the platform is just an IDP, but that's the developer experience of the platform. It's not the platform engineering team's view of the platform.

The platform team then needs to push everything forward into a position of, we are monitoring these devices remotely and have all of this telemetry in a useful way for the operations team. So it's not just serving developers anymore, but it's also serving operations.

[Pinja] (21:02 - 21:22) All right. I think that's all the time we have for this topic today. I want to do something.

If you're interested in the subject, as Darren mentioned, there is a blog by Darren on the Eficode side. There's also a very masterfully written master's thesis on the subject. You might want to guess who has written it, but if you want to go and check that out, please do.

[Darren] (21:22 - 21:38) Yeah. It's a terribly written master's thesis, but it was done through Jyväskylän Ammattikorkeakoulu. So anyone who knows what that is can probably find it.

It's talking more on this subject. But yeah, as Pinja said, that's what we have time for today. So thank you for joining me, Pinja.

[Pinja] (21:39 - 21:40) Thank you, Darren. It was fun.

[Darren] (21:40 - 21:42) And we hope you join us next time. We’ll now tell you a little bit about who we are.

[Pinja] (22:00 - 22:05) I'm Pinja Kujala. I specialize in actual and portfolio management topics at Eficode.

[Darren] (22:05 - 22:08) I'm Darren Richardson, security consultant at Eficode.

[Pinja] (22:08 - 22:10) Thanks for tuning in. We'll catch you next time.

[Darren] (22:10 - 22:16)