Why is it still so hard to get DevSecOps right in 2025? | Panel discussion

[Stefan:] We're going to talk about why it's still - tough, tough, tough to do DevSecOps in 2025. We all know we should do it. What are we even doing? That's the even bigger question. To go through that, we invited a few people. I'm going to run totally off the order of the screen. We have Mads working for Mjølnir. Then we have Ophelia working for Danske Bank. We have Jonathan from GitLab. I work for Eficode, but you'll listen to - these three people, not me. That's the important bit. Kicking off a bit, we need to figure out what DevSecOps is for you. If we start with you, Mads, what does it mean for you? [Mads:] Everything about AppSec all throughout the SDLC. Maybe even more important is where we draw the line between - what we leave up to developers and what to the platform team? The further I've worked in this field, the more I want on the latter, - especially in large organisations. That tends to be the cheapest place to put it. Hoping you're not going to say, oh, the same as Mads. [Laughs] It's only good if you do. That means we have an agreement on it. [Ophelia:] I would define DevSecOps as - integration of security practices within the lifecycle of app development. And it's more like a style of thinking instead of an actual role. [Stefan:] It's good to hear that. We hear things being a role, - and people forget what it actually means. Last but not least. [Jonathan:] Sure. For me, - it's an organizational discipline around delivering high quality software - as quickly as you can and as securely as possible. Under that, there are various different principles - around people, process and technology. We know there are cultural components that help. We know there are technical capabilities and different methodologies. Ultimately they focus on the goal of secure quality software - as quickly as you can. [Stefan:] Good to hear we don't disagree on everything. That would be - a long, long discussion. We would be here defining it for 30 minutes still. If we jump back to the beginning, we want to start out with DevSecOps. We come from an old organization. We don't have this culture or setup. Where do we start? With the culture, the tech or how do we do it? [Mads:] If I may start, it's all about starting with the management buy-in. You need to convey management this is something you should spend time on. Resources, nonetheless... if you don't have the resources, you can try to push culture, - but you're going to meet managers who will just say, - this is not a feature, so it is not a priority for management. We're not going to spend time on it. [Ophelia:] It's always good to start with management buy-in, - but the danger of starting too much up is that there's a tendency to fall - into totally top-down with no understanding of the bottom's intention. It's like, back in the days when we adopted Scrum - and everybody needed to be agile. It was just, you just need to do it. Use Jira. That's it. [Chuckles] [Jonathan:] I like to think that if you're even considering DevSecOps, - it's because you want to accomplish something. That's the delivery of software to generate value. Every organization has many different value streams. If you consider software engineering or delivery as a value stream, start there. Try to understand how the business and the teams define the value, - so you can come to common ground. Ultimately, you're going to need to build software. I don't know any other way you want to do it - except for having some DevSecOps practices in place. [Stefan:] Ophelia and Mads, you are and have been in highly regulated spaces. I guess things like registrations and legal work can become a kickstarter, - because you spend so much time on all of the legal stuff all of a sudden. [Mads:] I think so, but there's also a chance of - falling into the trap of pushing it out, top down. I totally agree. We should watch out for too much top down. We need to find a way to do development or involvement - from as early on as possible, so we can test - if the things we think will work will actually work, - before we've pushed it to 100 dev teams. They're all saying, we don't know what's going on, so we're going to ignore it. [Stefan:] Sounds like the good old, shift everything left - and don't tell anything about it. [All laugh] we get told that all the time. We need to shift everything left - and suddenly all of the developers have all kinds of issues - and the rest are leaning back and relaxing, having a beer. [Ophelia:] In a sense, - DevSecOps is going to save a lot of time for your developers. As a part of a highly regulated company, - you need to adhere to a lot of documentations. You need to be on top of your security. All of the things you're already doing, - maybe half manual, half automated half in thousands of different ways, - that's actually the opportunity where you can make them consistent. You only do it one way and you do it much cheaper than before. [Stefan:] Yeah. As you mentioned in the beginning, - thinking it as a platform is a good thing. You might not even have a platform yet, you might just have a build pipeline. Start pushing a few things into that would probably be a good start. [Mads:] If it's a shared build pipeline, the audit area, it can be... [Stefan:] Everybody builds their own. The perfect scenario. Nobody knows what's going on. Auditors come around and we have a lot of issues all of a sudden. [Ophelia:] You need to think about what you want to do centrally - and what autonomy you want to give to the teams. From a security perspective, a lot of things are really generic. So it doesn't make sense to let everybody reinvent the wheels. [Jonathan:] I think it always depends on the organisation, - how mature you are, where you are in your journey. If you think about software delivery, that is a process - that you can continuously improve upon. I agree, if you're in an established enterprise - that has proven practices, maybe a platform, - you can bring in a new team and you can get them started somewhere. But sometimes you're not there. I like the idea. We need to deliver software, we have these requirements. We need to put some things in place and then continuously improve, - not just the software you're building, but the process - and the technology you're using, the discipline. [Stefan:] When you start consolidating it into a platform, - I guess it's going to be easier to pitch your platform - to the different teams as well, if they're not using it. Well, if you come here, let's show you all you get. You get vulnerability and dependency scanning, all of these different - technical things you might need to build on your own on the side. I could imagine, I know you've been deep, deep down into the security space. I guess you want to know the full picture of - all of the vulnerabilities in the full organization. [Mads:] Yeah, I think so. Over time I've come to think that - we need to try to move at least the application security - more into the engineering side of things and away from security. So you're the bridge between the rest of the platform and security, - whereas being placed in security. You wrote about the Department of No, right? You can quickly become associated with that. I usually say that if one success is one step forward, - every failure we do is five steps back. It takes time to regain the trust of the developers - if you are Department of No. [Stefan:] It sounds like you have a background in the military. You always have, one plus is good. But when you do something bad, - you're stepping down two steps. So it's hard to catch up. What about success metrics? Are there such for DevSecOps? Is that fewer vulnerabilities? Is it happier people? Faster delivery? Can we actually coin anything in that regard? Is that too complex and we need to - talk to fifferent organizations to figure it out? [Ophelia:] We are actually tracking the time we are spending on - what I call security paperwork. So that would be the matrix I measure on, how much time - we were spending before on the security part, either filling in papers - or actually fixing vulnerabilities, doing things manually. Until the end, after the adoption of whatever central tool - and offering there are, how much time you're actually reducing. Most of the time we can actually see - a reduction of 90 percent of the time you're spending on it. [Stefan:] That's quite impressive. You're actually putting DRC into everything, - making sure it's to some degree automated, as much as you can. [Ophelia:] Well, it has some positive sides. [All chuckle] [Stefan:] Until you get caught in a big pile of paper - and it's hard to get out again. You also have to watch what you measure. We can really quickly end up in a situation where everybody's fixing - third-party dependencies that may or may not have an impact - on your software, because that's the easiest thing to measure. I come from a qualitative research background, - so I think doing structured interviews with developers - can definitely also provide some value in terms of measuring. [Jonathan:] When I'm working with organisations - interested in improving their security posture, - they usually have some major glaring problem already. They could've been exploited by particular - vulnerabilities and they're in the news. It could be that they spent months trying to remediate the latest, - whether it be Log4j or whatever you want to call that's been out there. Or it could be that organisations are struggling with the fact that - they can't release software very quickly, because it's the last step. That's when all the scanning happens. That's when there's a lot of friction. You can anchor on many things to see if you're getting improvements - with a DevSec effort or initiative. Are there fewer vulnerabilities making it to production? Are you remediating vulnerabilities faster? I love that you mentioned happiness, because you can't introduce security, - add more to the developer's plate and then expect them to - continue to be innovating and delivering high quality software. [Stefan:] Then it goes to the good old saying, just run faster. [Jonathan:] Yeah. Do more with less. [All laugh] [Stefan:] We're trying to optimize the organization, - so now you also have these obligations to do. Good for you. Yeah. We need to touch upon AI as well, I guess? How can it even help you? I pitched one of the questions for you. Can AI actually be two of the four eyes in the four eyes principle? I'm throwing you all under the bus now, - because these guys want to know if it actually can be. It can also be only one out of the four. [Jonathan:] I'm happy to jump in there. I want to acknowledge first that AI can help, - but it also is introducing a larger attack vector. You've got now prompt injections to worry about. You've got lots more code being generated, and so forth. You can also have, in what we're doing with GitLab, - you can have a Gentic AI being an assistant, - somebody you're collaborating with as you're going through your process. Think of security analysts now. When you get a bunch of vulnerabilities... hopefully you've got the strong foundation of DevOps - principles and governance architecture. When you have a bunch of vulnerabilities in front of a developer, - if a security analyst, running as an agent, can take a first pass, - there's certainly going to be productivity improvements. [Mads:] I'd like to challenge that in one sense. [Jonathan laughs] I can see a scenario where developers get used to this - and they miss all the important stuff - that the AI agent hallucinates about. [Jonathan:] Sure. [Mads:] At least this is something I'm really aware about. We need to test out and see how this works also in the longterm. If people just get used to auto remediate, auto remediate, - we might actually end up with worse software. [Jonathan:] Yeah. This is where a human in the loop is very, very important. It's a broader AI discussion we don't want to get into. [Mads:] Yeah. [Jonathan:] You cannot just press a button. If a security analyst agent is creating a merge request, - you have to have humans being accountable, reviewing that. If they're going to click that button they are now accountable. I agree with you. There's the broader issue with AI of skills and atrophy. [Mads:] The next generation of software engineers - haven't had to sit and do this stuff by hand. Will their skillset be lower? Will they be worse off making that decision of - whether or not the agent was correct or not? At least that's my fear. [Ophelia:] Going back to the original question... [All laugh] Is it difficult to substitute the two eyes out of four? It can accelerate the four eyes that are involved. Of course, what we have experienced is that - it's better explaining stuff than some of all security experts. And it has more patience towards... [All laugh] Especially for people who are really new. I'm head of development department. My developer doesn't have that much interest in reading up - on all of the new stuff about vulnerabilities that comes in, - that got discovered, how to remediate, what is this, what is that. AI is a really good tool for them to seek up - this information in a relatively cheap way. [Mads:] I'm an AppSec engineer, - so I'm from a different part of the organization in that sense. I don't want to speak ill of AppSec engineers, - but it is a cultural problem that has been there for many years. We're moving in the right direction. As a profession we need to have a different mindset. Curiosity and paying attention to what developers are actually saying - is key to actually making this stuff happen - and DevSecOps in general. [Jonathan:] Yeah. [Ophelia:] That's the eternal question. Back in the day when there was just DevOps, - when the ops came in and married devs, people were saying, - why aren't developers more interested in being more aware of what op is about? How can you be reliable? How do you operate in production - and actually take ownership out there where the fire burns? But the thing is, when you are focusing on one thing, - you give up on something else. You are choosing not to focus on something else. In that case, developers need to focus on solving domain-based problems. Section Ops should be there - as the enabling factor of generating business value. And as I see it, the purpose and mission of platform engineers - is to narrow down the context window. I think that was the word used. That makes it easier to make the proper choice regarding operations. It should be the same for the security offerings. It should be very easy for our developer to do the right thing in security. [Mads:] I think it's all about culture and the small communities - you can create at least within big organizations. In my previous job, I ran a security chairman's program. That's a real opportunity to engage with developers in a very eye to eye level. I think it's about having interdisciplinary forums - where you can discuss these things instead of - having shutters between different professions. [Ophelia:] What happens if that security champion got run over by a boss? [Mads:] The thing is you have many security champions. [Ophelia:] But you usually only will have one security champion per team. So that team is not going to be secure while we find a new one. [Stefan:] Or you have one per area. [Mads:] Yeah, well, - I can talk a lot about that. That depends on how you organize yourself. [Stefan:] You need everybody to be a security champion, - or, wait a minute, you need to have plenty, I guess. [Mads:] You need more than one and spanning several teams. [Stefan:] But do you need security champions? Is that a key to success or can we do without them? Or does it really depend on the maturity level? [Ophelia:] I really agree culture is an important thing. Culture was also an important thing when we adopted DevOps, - because both sides need to link in. It's not just one side, forced marriage. You need to be there, - but it doesn't necessarily mean - that I need to go to my husband's side and learn to write front-end code. There should be an understanding, an appreciation of its importance. I want to understand your work. I'm curious to learn, - but I don't have to be an expert. [Stefan:] Yeah. Someone might remember Joel Spolsky from Stack Overflow. He said, you need to know the level beyond on... you're a master of this. You need to know, - to some degree, the level below it. I'll never become a security expert, but I know - a few things and I know how to tweak and twist things. I'm aware of it, but I'm definitely not an expert. It makes me understand if I see a vulnerability score, - I can actually figure out if I need to rush in - and do something or if I can wait until later. I guess AI can assist, like you said, people joining the industry. How critical is this? Well, this scale is actually 1 to 10 - and you just got an 8, which means it's fairly critical. [Mads:] To answer the question, - it depends on how you define security champions. You do need, especially in larger organizations, - to have some sort of developer loop, a feedback loop. Security champions is a great way of doing that. This way of giving feedback to the developers, but also receiving feedback. If we're rolling out a new feature, nobody knows how to use it. [Mads:] We need that information to flow in a structured manner - so we can catch this early on. [Jonathan:] This conversation around security champions. Are there any organizations here that would trust - just the developers to deliver secure code? [Stefan:] They're all specialists in security software, that's why. [Jonathan:] Oh, all good. This concept of enabling teams, - whether it's one person or a group of people depending on your enterprise. You do need that level of expertise, from my perspective. As a software engineer, I've always wanted to build high-quality software. I've always wanted to be high-performing. I look bad if it's not performing well. At the same time, software engineers should care about delivering secure code. I don't want to put something into production that ends up having - a big exploitation for my organization. You do need the security expertise and you need that team, - but developers do need to care about that. [Ophelia:] Yeah. Developers need to be the owner of security for their own application. It should not be the security champion, - because that person will then become the naysayer. [Jonathan:] I agree with that. Yeah. [Mads:] I see it more like an advocate. This is somebody who doesn't get the responsibility, - but they get to be the first line of support for the development teams. Otherwise our AppSec organization is going to be way too big. We need some semi-experts, what I usually call them. So they know when to escalate to the AppSec department. At least that would be my approach to it. [Ophelia:] But is it really realistic to find at least one security champion? Because it requires very specific, special kind of person. Like coding, understands software, - talks well with developers, also has interest in security. Wants to work in the development team instead of working in security team. [Stefan:] I guess it depends. I've actually met someone in security - that had a background as a commercial diver. That was his background for being in security. He was so used to running with schemes that were set. You need to check all of these valves. If something happened, stay calm, solve it. He had no idea about software before he joined the industry. It was quite impressive to see such a career jump. I guess you don't need to be a specialist - in security or software to succeed with it. Who dares to say a manager can be a security champion? [Laughs] [Mads:] Again, it really depends on how you define it. To me, a security chairman is a developer or at least a software architect. They know what's going on in the team and can understand the problems - that the teams are facing in interacting with - what we as a platform engineering organization are building. [Stefan:] I think you said it is the ambassador. It doesn't have to be a specific skill set, - but it makes sense that it is a developer. It sounds to me this is the same discussion people have around platforms. You need ambassadors for your platforms - to make sure that people actually join the platform. It makes more sense to join forces - when you're doing DevSecOps and platform engineering at the same time. So I guess that's slowly moving things as well. What about the future, if we look ahead? Is there a bright future where everybody does DevSecOps - or is it just something on a hype curve somewhere - that's going to die off and come back in a mid-level somewhere? [Jonathan:] I think we continue to evolve as a community and industry. There are continuously advancements in software delivery in general, - as well as your capabilities from a platform perspective. I don't like the idea that we're just stuck here or we're going to stop. I'll bring it back up and I don't want to derail us... AI is the big elephant in the room. It is going to play a role, for sure. I think we're going to continue to see transformations. We're going to see some of the same problems being amplified. For example, you bring an AI to a low performing team - that's not producing secure code without AI. You give them AI, then they produce lots more in secure code. If you have a team with strong foundations, that... it's a high-performing software delivery organization, doing the right things, - with the right cultural component. AI amplifies success, makes them better. [Ophelia:] Security has never been as relevant as it is right now, - given the political situation and considering how far we are. Maybe 10, 20 years ago, people were not even encrypting things. [Stefan:] That sounds like my first job. Who cares about encryption and security? [Ophelia:] It's definitely here to stay, just like Ops came in to the Dev side. We started to scale because of the globalization. We started using IT and digitalization extensively to solve our problems. Security is more and more relevant, but I still believe - it's a work philosophy instead of something specific. [Stefan mutters] Back in DevOps days, it was... We were also just trying to find the unicorns who know both Dev and Ops - and also can stand there and be in every team. We have failed to find these people and to be able to scale. So, yeah, that's just my opinion. I think it's the same thing... [Stefan:] We're building platforms now. [Ophelia:] Right now it's DevSecOps. You need to have three legs in all the teams and still be there. Therefore every team... we cannot scale in that way if we only depend on the unicorns. [Mads:] I hope we will move to a place - where security is seen more as an integrated part of quality software. If we want to reach the developers, that needs to be the mindset. We're producing quality software and we're also going to make it secure. They need to be tied together if we're going to be successful. [Stefan:] It sounds like you're telling my story being in this industry. In my first job I had no idea about security. Then I've worked at places where we had annual security training. Then we moved into annual security code training, - where we had instructors guiding us with capture the flag sessions, and so on. From what I see in the industry, we are seeing the level going upwards. [Mads:] That triggered something in me because... [Stefan:] Oh, I'm sorry. [Mads:] As an AppSec engineer - I hear so much about capture the flag, we need think like a hacker, - and everybody needs to know how to hack. I'm very much against that. [Stefan:] It looks good on a slide. [Mads:] It does, but then you talk to a dev who says to avoid SQL injections. You ask how they want to mitigate it and they don't know. Nobody taught them. As an industry, we should really watch out for being - too focused on things that look good on the slide. People say, this is so much fun, - but it doesn't provide any value for secure software. I can see there will be some value, but if you don't follow it up with - how we engineer the secure solutions, it's really not much you're getting. [Stefan:] That also reflects the culture a bit. I've worked with people that were running AppSec - who claimed everything was broken, everything is insecure. We need to stop production for two years and fix everything. Well, welcome to a business. We need to make money. We need to keep evolving our products. We need to strike a balance. You need to accept as a culture that you're not going to fix everything. [Ophelia:] That's also why a security strategy is quite important. What is acceptable, what is it tolerable and what can you make wait? The funny thing is it's never been easier to do security than today. There are so many new tools you can adapt. Maybe sometimes when you let the problem wait a little bit, - it will be easier to fix it in the future. [Stefan:] Just make sure you're in the gray mass - where you're not the top one that people are going to hack. [Laughs] [Ophelia:] That's what I'm saying. Sometimes the strategy is important. What is it you need to prioritize right now? What can wait? Waiting does sometimes have some benefit - because the tools and the environment will evolve. Just like the CVEs, there will come a fix, a patch. Don't stress right now. [Mads:] You need a product mindset also from an AppSec point of view, - because you need to, all of the time, look at - how can we provide the most value, right now, with the tools that we have. [Stefan:] Do we need new tools or are we okay at the moment? [Ophelia:] Oh God. [Stefan, laughs:] Do we want to jump down that hole? [Ophelia:] This is just like the DevOps talk. In the beginning, - there were three, four, five, and now it's hundreds on the landscape. From a developer perspective, we really just want fewer tools, - things that work and things that cover us. [Stefan:] Yeah. I guess we need to have other people - than procurement taking care of the tools. Sometimes we see organizations buy a tool, - install it, run it, and they don't look at it. So, great, you increased security, or did you? I've definitely seen that in the industry over the last 10 to 15 years. [Jonathan:] It's not tools that help a team produce secure software. We know that, right? Yeah. If only it did, AI could do that. [Laughs] what do we want to give people today? What should everyone keep in mind for the future? If they're beginning DevSecOps programs, mid-, long-term, - what are your hopes and dreams, what should people look at after this? If anybody dares to say anything. You're going to bring them the silver bullet here. We're going to have three silver bullets after this session. [Mads:] Listen to your developers, involve them whenever - you're trying to do new initiatives and do it in a structured manner. Make sure that you capture the information you want to. [Stefan:] But they're so noisy. [Laughs] Yeah. [Jonathan:] All right. [Stefan laughs] I guess it's a silver bullet. Moving away from having security be a gate or a checkbox. Having it be more an attribute or a quality of what is being produced, - and ultimately for the realization of some sort of value. [Ophelia:] I think... [Stefan:] You're stuck in between now. [Ophelia:] If you really want to be successful with the adoption, - not just checking a mark, you should look at the whole picture. Look at all the people involved, the process and the tools you need. What is your actual requirement? One of the takeaways I had, I don't remember who said it, - start looking at whatever people are already doing. These will be your low hanging fruit to get into the DevOps Cyclops praxis. [Stefan:] I'm happy none of you said, - we need to do a full risk map and threat modeling. [All laugh] do people even do threat modeling these days? [Mads:] They should. That's a good answer. [Mads:] Yeah. [Stefan:] It's been the same answer for the last 20 years. I still recall when I was in school, with threat modeling - everything would be good, all of your risks, but software is complex. [Mads:] To top some of that, threat modeling is one of the places - where we can make teams take ownership of their architectur. Their decisions have impacts. Have you considered what the impact is - of adding this connective to these new services? [Stefan:] I've been in the SRE space. I would love people do - threat modeling so everything doesn't crash and burn all of the time. Anything else you guys want to add? You want to revolutionize the whole business - and create new products or slap some AI on it? [Jonathan:] No. [Stefan laughs] [Jonathan:] I do think AI is really important. Nobody is denying that's going to play a role. But when it comes to tools, it's always people, process and technology, right? You can have the world's best tool, but if you're not operationalizing it - and it's not easy for people to use and make sense of... I'm trying to oversimplify this big problem. [Stefan:] It's so hard to give a specific response to everything. [Mads:] Based on what I hear people say and my experience in industry, - people tend to focus a little too much towards the tools. We need this tool, we need SAST and DAST. When you look at their processes, they're not always in place. Taking that one back is useful for many organizations. [Stefan:] Some people are looking at the tools, because you're going to be - hit by legislation that says you need to have SAST and DAST. Which tool to get, which magical box can solve this for us. Then you talk to people who say, DAST is a bit too complex, - and they don't want to go that way. It's just easier to buy a tool, do SAST, and everybody's happy. Then I guess you can save some time to do the real stuff - like changing the culture, getting people up to speed. I guess developers still need to know a certain degree about security - to write secure code in the end. CtF is not necessarily bad but it's not the solution to everything. [Mads:] It's awareness and... [Ophelia:] A secure developer needs to know something about security. Otherwise when you come with a new tool from the platform department - people say, why the fuck should I adopt it. [All laugh] Of course, they need to be aware it is important, - why it's important for their ownership, why they should care. [Stefan:] That loops back to the Department of No. Because security says so. [Ophelia:] Or because we need to check this mark on the Excel sheet. [Jonathan:] Exactly right. [Stefan:] We get back to the good old checkbox - security that everybody loves. [Jonathan & Mads:] Yeah. Right until you have an incident. Then everything crashes and burns, no matter who you are. Any last words you like, buy our stuff, change your whole industry - or anything you want to give a shout out to at the end? [Mads:] Threat modeling. [Stefan:] All right. On that note, everybody's going to leave the room - and do threat modeling now. Are you going to be running a session in the hallway on it, or not really? [Mads:] Well, like write me a LinkedIn. I'm a consultant. We can figure that out. [All laugh] [Stefan:] It's always like that. Please come to me afterwards. I'll happily do work for you. [Laughter] All right. On that note, thank you to all three of you. They will be around if you have further questions. ChatGPT can make mistakes. OpenAI doesn't use Eficode workspace data to train its models.