Is DevSecOps broken?

[Darren] (0:02 - 0:22) It all comes down to the fact that the ‘move fast culture’ does not support the care required for security. Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development. [Pinja] (0:22 - 0:44) Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna. Welcome back to the DevOps Sauna, where I'm joined once again with Darren. Hey, Darren. [Darren] (0:44 - 0:45) Hey, Pinja. How's it going? [Pinja] (0:45 - 0:50) It's quite all right. We're in September now, so I think officially this is autumn. Can we agree on that? [Darren] (0:50 - 0:58) I think we can. I mean, I'm still wearing shorts, but I'll be wearing shorts until the middle of November at least, so I don't think that's an indicator. [Pinja] (0:58 - 1:05) It was a very, very warm day actually here in Helsinki today, some 20 degrees Celsius, so I think it's okay to still wear shorts. [Darren] (1:05 - 1:09) But that's the thing. You claim it's no longer summer, but the weather disagrees, so... [Pinja] (1:09 - 1:35) I know. I know, but we can talk about climate change another day, but that's an important topic. But today we're talking about one of our favorite DevStarOps topics. A couple of years ago, it was very cool and hype to name everything DevStarOps, but today DevSecOps is our topic. And we have some news from a couple of reports, a little bit of provocative statements here, but can we call it DevSecOps, Darren? [Darren] (1:35 - 1:57) No, and I'm really glad the trend for calling things DevStarOps seems to be over, because they're all the things that should be involved in DevOps. There shouldn't be DevSecOps, because if you're doing DevOps without security, you're doing it wrong. And why do you need to specify that your DevOps has security? Just specify that you're doing DevOps and not doing it badly. [Pinja] (1:57 - 2:23) This was a couple of years ago. There was a whole debate in my team, that is, is product management part of DevOps, or is DevOps part of product management? And we did not agree on this with the product management experts, but at least we can agree, everybody here, that security is an integral part of DevOps as a practice. But there was a report by Snyk saying that DevSecOps is dead, or is it? A little bit of clickbaity, I want to say. [Darren] (2:23 - 2:39) I think actually that was inspired entirely by our clickbait episodes earlier in the year, where we decided to see if various things were dead, like Jenkins and DevSecOps. So I'm pretty sure Snyk decided to call back on that one. We're glad you did, Snyk. We appreciate it. [Pinja] (2:39 - 3:21) Exactly. We already busted those myths. Agile is not dead. No, neither is Jenkins and everything else. But it is true that in the past couple of years, we've heard some alarming statistics about security practices. And it was only in the very recent news episode that we covered that many senior developers actually say that their companies are shipping unsecure code. And we're not sure if this is a trend we can actually get on board with, but there's something going on in the DevSecOps. I'm using this very intentionally, this term, this time, because we want to make sure that we understand we're talking about the security part of DevOps. But is it something that we can't even claim that is already totally beyond repair? [Darren] (3:21 - 3:43) Hmm. That's actually a really good question that I think we should come back to at the end. We mentioned the Snyk report, but we actually dug up three different reports on the state of DevSecOps for this episode. And I think we should go into them because the stories they tell are kind of interesting. [Pinja] (3:44 - 3:52) Let's do that. So we're talking about three reports here. So we mentioned Snyk, then there is one by Checkmarx, and one by Datadog. [Darren] (3:52 - 4:05) Yeah. Well, let's just get started by diving right into Snyk's DevSecOps. Is it dead? Oh, DevSecOps is dead. Or is it? It's a massively clickbaity title, but they kind of had some good points in there. [Pinja] (4:05 - 4:32) They were good points. And it was, they dug down more from the culture reset angle to what is actually bad in the practices of security at the moment. And one of the points that they made was that the shift left was actually misapplied. And the security tools were basically just thrown left, and this disrupted all the pipelines. And does it mean that they were not actually built in, but they were just kind of copy pasted and just added? [Darren] (4:32 - 5:22) It kind of means the idea of shift left was to move security thinking earlier in the development cycle. And what that meant in reality was building security scans into CI/CD pipelines. And the problem is this added a massive cognitive overhead to every developer who had a new control gate. And then they were basically just being hit with these new tests that were failing. And Snyk had this idea that they'd been thrown left. So there was no real integration. There was just a forced application. And that led to inefficiencies and the context switching, which is kind of the death of all development productivity, and this frustration for the developers. So security was never embedded into the natural flow. It was just kind of bolted on. [Pinja] (5:23 - 5:41) And the report from Snyk also highlighted that this also meant that the trust between Dev, Sec, and Ops has been missing. And as you say, Darren, embedding the practices in kind also needs that trust, right? So it's one of the things that kind of doesn't actually support the embedding here. [Darren] (5:42 - 6:20) Agreed. The biggest threat to security is always silos. And I always come back to the, I've actually forgotten his name, the guy from Microsoft who was at the Future of Software in Sweden. But he had the absolute best quote, which was that a silo is an enterprise security device designed to stop information leaking to other departments. And this idea that you basically just break down communication and security is trust. Without trust and without transparency and structure, like what's the point? You're not going to be doing it. It's just going to be in the way. Then it's an obstacle. It's not a partner. [Pinja] (6:20 - 6:40) No. And we kind of keep hearing people complain about the security training, but if we diminish that all the way to, okay, let's just disrupt your flow and your feedback loops with the security tooling and security practices. And in addition, the most visible part of course, is the security training for some people. So it doesn't exactly help in the trust building part and the culture building part. [Darren] (6:40 - 7:17) Yeah. And there's a lot of mistrust in security in general, because something that was actually highlighted in one of the later documents we'll get to, but there's this kind of fear, the FUD, fear, uncertainty, doubt, which has been recently co-opted by the crypto crowd. But it's originally a cybersecurity idea that you have to communicate calmly and clearly to avoid people descending into fear, uncertainty and doubt. So I think there's a lot of work to be done in cybersecurity on a communication basis. And that's what you mentioned at the start, that Snyk was suggesting the idea of a cultural reset. [Pinja] (7:18 - 8:10) So let's move on to the next one. So the report from Checkmarx highlighted something called a maturity map. And there are some statistics on the maturity of organizations. And it said 12 % of them are security focused. So it means that this is a kind of like a, they call it stage one. So they're more reactive, there's adversarial patterns, and this is like, they called it a bolt on security. So then the large majority, over half, 58% are DevX focused. So this is a stage two, and how they named this, they have the security tools integrated, but it's still disruptive to the flow. So kind of what the Snyk report was hinting on is that there is the DevX angle here, and then only 30% are actually mature in their practices. So this is stage three, and with collaborative trust based practices, and they're secure by design. [Darren] (8:10 - 8:26) I do wonder about confirmation bias in this particular statistic, though, because where is stage zero, which is people sat behind their desks with their fingers crossed, hoping the hackers won't find them, because that seems to be a part that's missing in this particular study. [Pinja] (8:26 - 8:52) There is a lot of trust in organizations to have even that stage one level of DevSecOps practices. But the findings from when they did this maturity landscape by Checkmarx is that developers are more engaged with security training and security coding, but it is still so that they still spend a disproportionate amount of time on the security tasks themselves. And again, disrupting flow comes into play with this. [Darren] (8:52 - 9:24) Yeah. The flow is important, but there's also the fact that there's a missing trust between the tools. And they were specifically talking about application security and citing things like false positives. And there's this idea that signal, not noise, and a lot of security tools are still configured to generate noise. They flag everything and there's no subtlety, no real understanding. So the understanding of, say, business cases and human use cases, it's very easy in security to see black and white. [Pinja] (9:25 - 9:36) And there is kind of the trend right now that is the application security becoming too, quote marks, too developer led. Is it okay that it's being led from the developer side and not by the organization itself? [Darren] (9:36 - 10:12) That's a really good question. The thing is you want the teams involved, you need them to take ownership in some way of security because the security functions inside businesses tend not to scale as rapidly as other functions. So it's very easy for security to get out of hand. So having developers like pitching in and being part of the discussion is vital, but to do that, they sometimes need additional training that they may not be receiving or competence that they may not have in their team. So it's kind of a balancing act. [Pinja] (10:12 - 10:41) And then there's what I've seen. This is not just with DevSecOps, but this is the topic of the day. If we think about what is the responsibility of having some kind of alignment in the organization. So there is, I also see the balance in here. Like, do you have some kind of, I don't want to call it central control, but centrally agreed practices then owned by the developers and the developer organizations so that there is that ownership and feeling that this is part of what we do. And that way we can bring in the quality again, but it should not be too mandated, right? [Darren] (10:41 - 11:07) Agreed. And that's actually something that Checkmarx touched upon in their document, because they were talking about the lack of a shared success metric between DevOps and security teams. So like in DevOps, you have failure rates and deployment frequency. They're standardized, but for DevSecOps metrics, they're just inconsistent. And there's again, yeah, as you say, there's no kind of agreed framework. [Pinja] (11:08 - 11:15) So maybe we can also think about the kind of missing metrics part that, in some cases, at least the automation adoption can be very low. Isn't that so? [Darren] (11:16 - 11:41) Yep. So lack of automation adoption is a problem. I think they said 32% of AppSec automation, they'll have significant AppSec automation and automating everything is seen as unrealistic due to lack of shared governance and consistency. So it's quite a, I would like to see everything automated to be honest, as a security person. So that being seen as unrealistic is problematic. [Pinja] (11:41 - 11:50) Well, let's say in the history of time, history of human beings, we've seen many things that were considered impossible become possible over time, but it requires a mindset change as well. [Darren] (11:51 - 11:56) It does. Speaking of a mindset change, should we pivot over to Datadog's state of DevSecOps? [Pinja] (11:57 - 12:18) We should. And their angle was more on the side of data reality. And they augmented the traditional common vulnerability scoring system. And they run it, scoring with it with runtime context. And they said that only 18% of vulnerabilities with a CVSS score actually remain critical once the context has been applied. [Darren] (12:19 - 13:32) Yeah. And in this case context, we're talking about things like if it's in a production environment or exposed publicly, or like even if there is any attack surface to leverage it. So this was so pleasing for me to read because I've been saying this for years that we have this one metric, we have CVSS and it goes, it's a scale zero to 10, 10 being the worst. And I like to cite two examples of two 10 severity vulnerabilities. The first is Log4j, which was affecting a lot of machines. The second is Heartbleed, which was affecting most of the internet. And these were both things of the same severity, the same score, like CVSS is useless. It's not useful from a perspective of understanding the global cybersecurity landscape. It's a lightning strike metric designed for panicking if you get struck by lightning. And Datadog didn't say it. And that's my personal interpretation of how CVSS works, but having data to back it up, saying that a fifth of vulnerabilities remain critical once you apply context was just a thank you Datadog for that. It's nice to know that I'm not completely wrong. [Pinja] (13:33 - 13:41) I don't know if it's okay to say that it's nice to see that huge red flags are being raised, but I guess that the good thing is that they are being raised. [Darren] (13:41 - 13:51) I think so. Yeah. If you think about 82% possible reduction in like patches for non-critical vulnerabilities that are rated too high. [Pinja] (13:51 - 13:54) When you turn it that way, it doesn't sound too bad anymore. [Darren] (13:54 - 13:55) Yeah, it sounds good. [Pinja] (13:55 - 14:05) It does. There were a couple of things that didn't come as a huge surprise for us when we're reading the Datadog report. So if we think of languages, Java, super vulnerable. [Darren] (14:05 - 14:22) Yeah. Java was particularly vulnerable with 44% of Java applications containing at least one known exploited vulnerability. And that was compared to approximately 2% of languages like Go and Python and PHP, which doesn't look good for Java, but. [Pinja] (14:22 - 14:33) No, it does not. And we know how widely used Java still is nowadays. So, yeah. But the other no surprise for us was that, well, supply chain threats are still a big thing. [Darren] (14:33 - 15:20) Yeah. Especially with the prevalence of vibe coding and the idea of slipping dependencies in that people might, like typosquatting people using passport JS to mimic passport and various Python package injections. There's a lot of room for improvement in dependency challenges. There's also one thing I want to raise, which is there have been improvements and all three documents have said there have been improvements, but one in Datadog specifically was a 5% drop in long lived credentials in CI/CD pipelines. So we're starting to see modest improvements across that, which is kind of cool. Nice to see things are improving, even if they always sound a bit doomy. [Pinja] (15:21 - 15:48) Like a couple of years ago, for example, we had the reports for exposed secrets and actually implementing SBOMs. So we do have visibility to these things and slowly organizations, I think, are starting to implement better practices, but it should not be detached from the actual chain of development. But if we summarize the three reports a little bit, basically the Snyk report says that the philosophy is wrong. Is that what we can say about it? [Darren] (15:49 - 15:55) Yeah. Snyk was the philosophy is wrong. And then Checkmarx went on to say that culture is lagging. [Pinja] (15:55 - 16:04) Yeah. And Datadog was basically that the execution was sloppy. And is it okay to say that they're kind of correct in their own way, all of these three reports? [Darren] (16:04 - 16:32) I think they're all fair. We've said for quite a long time that you need buy-in, you need culture, and you need execution. And they've all hit the nail on the head. They've just focused on different things. And because Snyk is providing different kind of tooling to Datadog, Checkmarx is similar to Snyk. So they're all taking their own angle. But yeah, philosophy needs to be the first step to align, bring the culture in line, bring the technology in line. [Pinja] (16:33 - 17:01) And this wouldn't be a 2025 episode of the DevOps Sauna without us bringing in the topic of AI. A surprise to nobody here, I guess. But DevOps in the age of AI is something that hasn't been perhaps discussed so much. But if we think of one example at least, we can talk about the model context protocols or the MCPs, which is now the emerging standard to connect the AI models with external tools. But what is the connection here and what should be taken into consideration? [Darren] (17:01 - 18:28) We actually just yesterday in one of our Slack channels were having a discussion with a couple of podcast guests we've had before. We've had Szilard and Henri Terho discussing this model context protocol. And people keep building in the applications of this that don't really have any security built in at all. There's this habit of AI that AI has to go fast and move fast, break things, including all the rules and good standards about security, which is not as catchy as a tagline for AI. But that's what's happening. So model context protocol is seeing massive potential for attack surface because you can do things like prompt injection, where you just casually have it dump environmental variables, including API keys, because you have a hijacked prompt. So MCP is an example of, if you had examined the problems, they look exactly the same as you would have had in 2015 or 2010. They are just old problems that are not being solved because people don't want to buy in. And we came up with some guardrails for this, and had some discussions on how to actually implement some kinds of security when doing MCP. But it all comes down to the fact that ‘the move fast culture’ does not support the care required for security. [Pinja] (18:28 - 18:57) That's a really interesting point and a really important one to actually remember. If we start to think about the next steps here. So we know that the state of DevSecOps could be better, but what is something that can be done for the time being? In any case of a transformation story, if we go through whatever the topic is with an organization, we usually say that you can't start small with some small steps. Pick one pain point and improve that as a beginning, because even that would be better than what we got right now. [Darren] (18:58 - 19:25) Yeah, especially if that's the position you're in. If you're one developer and you're struggling with security, any small improvement you make will be good. If you can see things from a higher level, like if you are a leader, if you're pushing strategy rather than code, then start looking at how you measure things and how you want to succeed with security in DevOps and start moving towards that. [Pinja] (19:26 - 19:51) There's a saying in our field that says that you get what you measure basically. And I think it applies here as well. You can focus on the measures and focus on the ones that you feel that are most important to you and start from those. But it also comes down to building the trust back to the whole chain of software development, that it is no longer perhaps seen as a DevSecOps, but rather as security seen as an integral part of DevOps instead. [Darren] (19:52 - 20:13) Yeah, that's true. The whole you have what you measure just becomes so much more vivid when you think of security certifications, like no one cares if you have the most secure product in the world if you can't demonstrate that you have the most secure product in the world. So it's an unfortunate state of reality, but it's there. [Pinja] (20:13 - 20:28) That's true. So we started this episode with the question: is DevSecOps dead, really? So maybe a question here. Do we think now in this light that DevSecOps is broken at the moment? Or can we just modify our ways of working and improve here? [Darren] (20:28 - 20:49) I think with the risk of being slightly controversial, it's kind of broken. There's this habit of people to invest minimal security and just think that that's okay, because it seemed like a chore or an afterthought. And like DevSecOps was supposed to shift this and like drag it left, but it hasn't. [Pinja] (20:49 - 21:30) And that's the thing that culture is the hardest thing to change in an organization. And if this is not done right, and the developer's flow and way of working is completely disrupted by adding the security element to it, it's not going to fly and nobody's going to feel any ownership of the security practices here. But there are things that can be done, as we said, small steps. Think of what you need to measure and start building that trust again. I'm trying to end this on a higher note. Not everything is broken. Everything can still be fixed. I think that's all the time we have for this topic today. So Darren, thank you for joining me here. [Darren] (21:30 - 21:32) Thank you, Pinja. It's been a pleasure as always. [Pinja] (21:32 - 21:35) And thank you everybody for joining us in the DevOps Sauna. We will see you next time. [Darren] (21:39 - 21:41) We'll now tell you a little bit about who we are. [Pinja] (21:42 - 21:46) I'm Pinja Kujala. I specialize in agile and portfolio management topics at Eficode. [Darren] (21:46 - 21:49) I'm Darren Richardson, security consultant at Eficode. [Pinja] (21:50 - 21:52) Thanks for tuning in. We'll catch you next time. [Darren] (21:52 - 21:57)