Talk: DevSecOps - going back to the roots with Patrick Debois, father of DevOps

Enjoy a replay of The DEVOPS Conference sessions conveniently through the DevOps Sauna podcast. This talk is hosted by Patrick Debois, Father of DevOps and Director of Market Strategy at Snyk

Lauri (00:04):

Every now and then, it makes sense to return to the roots. And now we are returning to the roots. We are having the one and only Patrick Debois who coined the term DevOps, who established the first DevOps days. And he is going to join us to talk, how to overcome security issues through the same mindset and approach that brought us DevOps.

Patrick (00:32):

Good morning, and thanks for having me. Yes, I've been there from the beginning at DevOps. I keep saying it's an accident that I've been promoting DevOps from the beginning, for so many years. And it's in that light that you have to see this presentation where I go back to the roots, where I go from DevOps and people want me to explain the new kid on the block, which is DevSecOps. It's not that new anymore, but people want to hear what we could have learned from DevOps and we can apply to DevSecOps. First of all, also a big thanks to Snyk, who allowed me to spend time on this presentation and help me with putting pieces together and listening in on conversations. I'm there as an advisor, but this is not a Snyk presentation, but I just want to give them a shout-out to their support on me looking and studying the field.

Patrick (01:37):

Coming back to the history, when I first set out to think about DevOps in general, I had a very team-centric view. I was working with Agile teams. I was working together with people. I was not considering global management structures on how to implement or embed DevOps into an organization. I was very much team-centric, and I still believe that is the view that is the most helpful to help the people on the floor. It's a little bit like in the factories, have the people that actually do the work that they sort out the problems they're having. I'm not saying you don't need a more central approach, but today I'm going to more focus on what you can do as a team in DevSecOps and in DevOps and not on the structure in general of managing it.

Patrick (02:41):

The way that I think about DevOps and DevSecOps is about friction points. And what we learned over the past 10 years is the friction point between Dev and Ops and security now, being in different silos. And those friction points were created by putting a command and control structure in between. So if you're looking for what you can do to do a DevOps or a DevSecOps transformation, you look at what are the friction points that got introduced by the silos. A few of the obvious ones are the stack, right? The people running the servers used to be separate from the people coding the actual application. The environment, the production environment was separated from the people doing the coding on their laptop. And it was all with good intentions. And I'm not saying historically, this was probably the aim where we're going for because stability and command and control at that time was seen as the way to go.

Patrick (03:56):

And with it, we also brought separation of prioritization. Should we prioritize a feature? Should we prioritize stability? Where do we spend our budget, who gets to control what goes on? And that goes all the way up to hiring setting incentives. And then on the personal level as well, we kept shutting down people while they only need to know how to write code. They, other group only needs to know about security and the other group only needs to know about the operational side. So what I've seen over the past is that DevOps actually pushed the code toward production. And you could say that's the shift right, the shift towards production, the delivery mindset.

Patrick (04:50):

We've already had the shift from a customer to have the features closer to the team. And that was the movement of Agile. And then the cloud with the whole virtualization and the abstraction layers move operations up the stack. We're not maybe doing our bare model anymore. We're having an abstraction layer for compute and recently with Serverless and functions and all the abstraction layers just go up. And then came around DevSecOps. And they started talking about the idea of shift left and the same ideas were already there in DevOps, we pushed the delivery, but the best way to take care of what gets pushed to delivery is to fix problems early in the value stream.

Patrick (05:42):

And that's my summary of the pressures and the shifts that pushed the central command and control out of the middle circle, pushed the management around the team and made the team the central point to overcome this. You will notice in the next slides, I have on the right lower slide, I have books and each of the books goes accompanied with the slides. Each of the books is a book and a presentation on its own. So this will be an over aging team where you can find more pointers to learn about each of the slides that I'm talking about, but it's important to know that the pressure is building up for the team. And my worldview is that the team is becoming the central part of dealing with the problems in the modern IT organization.

Patrick (06:42):

And if you want to take that a step further, the ultimate goal is to have an autonomous team. And there's a lot of debate around while autonomy, how much you can give autonomy to a team, yes or no. And we'll get to that later, that it's not an instant step. It doesn't make sense just to make a team autonomous and say go figure out everything. There is a logical order, almost like a learning curve that a team needs to become autonomous. But in essence, what I think DevOps and DevSecOps made possible was that the shift is to making that team autonomous. Where you draw the line between how much it is centrally managed and how much it is controlled, that is the different part in each of the organization. And that's what I called the just enough approach.

Patrick (07:42):

And the just enough is different in each of the layers. My mother used to say everybody equal doesn't mean everybody the same, and it will be the same in your organization, how this autonomous notion is perceived on the floor. And to understand how this notion of a team operates within a company. We can look back at the different collaboration cultures. This is a diagram that comes from a book called Reinventing Organizations by Frédéric Laloux. A fellow Belgium person. And it talks about the historical shift of organizations and how they have perceived collaboration culture over the years. Not so long ago, there was a strong command and control. We've seen it in previous slides like the management kind of control, what gets in and what gets out. You could say, this is when we had the castles with people coming in, the people coming out, and we have this notion of controlling everything.

Patrick (08:52):

You sometimes still see this in the metaphor of the early castles insecurity. It's all protecting to do this. Then after a while, this create a kind of order. It wasn't perfect. And we moved on to look at things like the stability. And the stability is when we move from the castles to the factory, we started creating things repeatably. We tried to put things under control and make this a stable system. Again, this wasn't perfect and we learned that just making this repeatable isn't enough. So we went a step further and we introduced scientific ideas and the scientific approach by measuring and putting KPIs. We want to give merit to everybody in the company and we measure what value that bring, and then we bring that into the merit they earned.

Patrick (10:03):

So this is pretty much the scientific we measure everything that somebody does in a company. Next step we took was around collaborative or sometimes seen as the customer-centric. You will see this in initiatives where they not just look at the automation, not just put the measurement in place, but they make it self-servicing. They make it customer-centric. So it's a step up in trying to control the automation, looking at internally, doing the measurement internally, every team is becoming a customer of the transformation. And then the last evolution is that we started making teams autonomously and contrary to the customer-centric, which is still much an offering of what the central team can offer on automation. We flip the charts and we give the team itself autonomy.

Patrick (11:08):

It's not intended that I used the word evolutionary, but the top-level doesn't replace all the lower areas. And when I found this graphic, I understood that we'll always have some way of doing command and control to create stability. We're always going to have to work on automation. We're always going to work on measurement. We're always going to make it self-servicing and we're making it autonomously. So all these layers will keep being embedded in your company. What does it mean for DevSecOps or DevOps? It probably means that by looking at this diagram, you could pretty much figure out how your company culture looks like. What is the most important value that your company thinks they are working on? Are they all about automation, stability? Are they still looking at the control phase only?

Patrick (12:11):

Are they really looking to collaboratively or empowering the team? And the book claims that it is your CEO that sets the tone, and that's probably going to set the tone of your transformation by saying what part or what of these areas is getting more value, but make no mistake. All of these are important in a successful transformation, but it depends on where you are in the spectrum that it will resonate more with you on the floor. You see this level-up of different layers from control to evolutionary in the DevOps team patterns. And you could see their DevSecOps teams patterns as well. In their websites, DevOps topologies, you can see how much of the overlap between Dev and Ops people allow in certain organizations. So some will say, well, there was a little bit of collaboration, others go for the complete team autonomous, and they do everything in a shared way.

Patrick (13:22):

Some would just say, it's a transient thing and we'll go away. And others will keep the liaison officers in between going strongly as the DevOps group. And others just say, it's a technical transformation. We're just going to put some containers and some DevOps, and it's done. In a way, they're all valid patterns. But depending on your culture, there is going to be a better match. And depending on the pattern you'll have different results. Usually people say: having a DevOps team is an anti-pattern. That's true in a way that when you have a separate team that only focuses on their own, then that's a problem. If that's your enablement team, then that's fine. It's like the evangelists going into the different groups and collaborating and making sure you foster the collaboration. So the most important takeaway is that these topologies exist.

Patrick (14:21):

And when you start your transformation, I want you to think about the patterns the teams makes most sense in your organization, in your collaboration culture that your company has. And then make sure you align towards this. The website is interesting in that way that it says, some of the patterns will have more results, but you can't just use one of the patterns that it describes as the most results and not have your culture support it yet. So it's typically a level-up game there. The team has a different way of interacting and in DevOps, it was the operational team or the infrastructural team talking to the development team and aligning those. And we found that there's three ways of interacting. There's the usual collaboration, we talk and we come together.

Patrick (15:13):

There's building everything as a service, which is the automation way. And usually, those go hand in hand, there's some collaboration happening. There's some automation happening and this gives a good feedback loop. But there's a third level which is facilitating. And sometimes that is a platform team, sometimes that's an enabling team. But it's important to know that once you know your company culture, which it is, you look at your topologies, you want to see how are you thinking about interacting with the security team versus the application team, the operational team versus the application team, and look at how that would work. Having them just collaborate and work together is not enough. What I see is a lot of people look at transformations as a way... how they can make things more secure.

Patrick (16:12):

And they're very focused about the principles of technology and making sure no vulnerabilities exist. And that's a good point. And it goes hand in hand with the control and make sure you have everything under control to make sure that happens. But as a lot of teams went from DevOps silos to the DevOps collaboration thing. DevSecOps added this extra layer, and we couldn't put ourselves in between. And that's why the security team actually needs to build up the trust from the developer team. They can't just force teams to developer team. And on the other side, the developer team has to gain the trust of the security team. So it goes in both ways. In The Thin Book of Trust, they use those four areas, the sincerity, the reliability, the competence, and the care.

Patrick (17:13):

And for a lot of people, the competence is clear, yes, developers need to learn about making code secure. And security people need to learn about coding practices and AppSec practices and testing in that area. But competence alone is not enough. You need to be sincere and reliable and care about it. And sincere could mean from the security people, is that they actually understand when things don't get scheduled on the backlog. Or the developer people can understand why security can get high priority than a feature. Reliability is that both of them are there at the correct time. They are there at stand-ups. They are there at making decisions. They are there at incidents and they are available every time it's needed. They are there. But you can be sincere, reliable, and competent, and not even care.

Patrick (18:17):

And that's the hardest part to have people care about security, because face it, if you're just listing out the vulnerabilities, you understand the vulnerabilities, and you're just going to tick the box on vulnerabilities, but you don't really care about fixing and making your applications secure. It's just going to be a drop in the bucket. So you have to work on making this trust trustworthy. And I think that is the most significant dance that is actually happening between the security team and the actual team. That is the dance of building that trust. And we do that by various ways of other interactions. And from what I heard, many transformation and many people I talk to, this is what I settled on. Is this a perfect model? It's not. But it is for me, easy to understand. There's a lot of discussion always going on, like what we are building to make that secure.

Patrick (19:21):

And I call that to secure stack, then how we are delivering the secure stack, which is secure delivery. And then the process to keep the delivery and the people and everything in check is the security governance. And then the fourth area that I want people to focus on in during a transformation is how to empower the teams. Because I think that is an equally important part and not just like a slightly small part of it. So people ask me: how do you get started with the secure stack? And it's probably the easiest, because it's the most tangible, it's technology, but make no mistake. It is not just about making it technically. Behind all these boxes on the screen, there's actually people. Whether you are managing the container, whether you're managing the cloud, whether you're managing external services, that is all people behind it, and your processes will probably get maturity level of dealing with some of that stuff.

Patrick (20:27):

And it will help people building around those parts. This is what most people focus on from my stories that I hear in DevSecOps, which is the AppSec side. But there's also parts around how the code itself can be integrated in security frameworks like user management, authorization and secret key management. And then the operational part, the monitoring the errors in the logging. Because you might say, what has monitoring error exceptional logging have to do with DevSecOps? Well, it's making your application more observable and making it more understandable when anomalies or attacks are happening. So I see this as an integral part of DevSecOps to deal with that as well. And then as a fourth pillar, which I dumbed the business part, there's the data privacy, the licensing, there's everything that is like going around or being produced by the stack. That makes sense.

Patrick (21:29):

So the team itself can go into each of these areas and we'll level up in each of those areas, and that will help to make more secure what they are delivering. So the next phase or area that I want people to look at then, is how are you delivering it? Is your laptop secure? Are your tools being vetted? Is your environment itself secure? Do you verify your repositories and you see the further along you get in the environment cycle or the environments, there's different things we can do. I do not decouple the development from the operational part because I think DevSecOps is that the team is allowed to integrate themselves in each of those areas. Does that mean they have to build all the repositories themselves or build a logging system themselves or build all the controls themselves?

Patrick (22:34):

No, but they are really instrumental in making that useful for the other teams as well. So it goes much further than just keeping it secure in development. I think we should spend the whole cycle there. Up until now, we've been talking about more of the technical side, the stack and the delivery. But a developer or a team can actually engage in a lot of the security practices. For example, if you just have your vulnerability management and you start scanning your vulnerabilities, you can help by setting a policy. And that gives visibility to the security team. The same with the threat management, if you start doing threat modeling on your curb and you start listing the threats, then you help security. I could go over all the lists, but some things people don't immediately associate with DevSecOps, but like the supplier management, why isn't that just the sole thing from the security people?

Patrick (23:47):

Well, if a team is autonomously and they can select their own toolset, their own versions, their own suppliers, those tools need to be vetted for security. So probably your security team is way better at doing those assessments, but the team can already help by pinging the security team and saying, hey, we're going to use a new tool. And how about you verify? Do you have any advice on how we should select these tools? And you start seeing the team engage in what are usually very security-centric processes. Again, the overlap is not complete but it is a third part where the team itself can take a big step for the overall security of the application or the company. And then as the last part, I see empowerment, we can have the stack, we can have the delivery, and the governance. But I would advise you to have a specific call-out that looks at how you build the empowerment of the team.

Patrick (25:00):

From my experience, I've seen this collaboration dance, usually start with the security team, send some of the scans and the words. And then the application team responds back with some insights and some feedback, and you have this phase of awareness and you learn from this. But collaboration by itself is usually not enough. Then you want your team to train and learn about new security areas. And again, that is a maturity level where first, they only ad hoc, look at some of the security. Then they learn about the issues that the security team mentioned them. Then they go deeper in their stack, and then they start proactively learning about new security issues. And as the learning and the collaboration increases, the accountability of the team can increase. And that's because of the competence and the reliability and the sincerity that is built up in the collaboration and the learning culture.

Patrick (26:20):

So that's when you start seeing flipping part of the accountability to the team. Usually, they have to be happening a lot more collaboration than the learning before the accountability can increase. And then at the last part, we want to give that authority to the actual team. Again, there's a spectrum on how much authority you can give that to the people. It is a buildup of all these skills. And I think that looking at those four areas is equally important than looking at your stack, or your delivery, or your governance. The book that I mentioned, Agile Conversations was in a talk previously yesterday. But I think it's a good way of this kind of conversations and having meaningful conversations and learning how to have a good conversation between the two groups is instrumental to making that culture improve.

Patrick (27:21):

And so in the end, when I look at the DevSecOps maturity, I see those four pillars as building up and becoming more mature. So instead of it being a security-owned issue where in the past the operational issue that it becomes like a team embedded thing. And that is what I ultimately see as what DevSecOps is trying to strive for, the empowerment will probably be helped by the governance. And that one will probably be helped by the delivery and the stack be helping the delivery. So there is a not saying cyclic, but there is a causal effect in each of those areas. So if you only are focused on I'm going to scan my code and that's secure, and my stack and I'm done with DevSecOps, it's probably not going to be sufficient. If I'm only going to work on my pipeline and making sure that the delivery process is secure, not enough. If I'm only going to go on the processes, not enough.

Patrick (28:25):

So all of these buildups making it happen is that this levels up, and that's at least what's my initial intention with DevOps as well, like building this, that the team can own the whole stack. But it needs to level up first in various parts and it uses automation, and measurement, and each of the other layers that we discussed before. It really focuses on overcoming those friction points. And if you look closely at your current friction points, that's where I advise you to start. It's the idea of fix your bottleneck and make sure you keep that focus on the ball. Your bottleneck will shift from one area to the other, but looking at the friction points that were created by the silos, is probably a good place to start.

Patrick (29:22):

I call the rest plain engineering. I'm not saying it's easy engineering. And I'm not saying it's not valuable. As I mentioned before, the automation and the technology will influence the handover or the creation of extra time in the transformation. So both go hand in hand. So tools and culture have their use in the DevOps and DevSecOps transformation. And you can't put themselves loose from each other, only do the tools, only do the culture, but it's kind of a collaboration between both. And that brings me to the final part because you could have all the checkboxes in all the four areas. You have leveled up in all of the areas, but you will be narrowed down. And I'm sorry to say this, but there are a few paradoxes at play.

Patrick (30:17):

There's the paradox at the command and control layer. The more you want to control, probably the less you're going to control. The more you automate after a while, the less you still understand the systems beneath, and then you have to start with chaos engineering and make sure you're still trained for when things happen. The same thing is with the measurement. And measurements are good, but if the measurement becomes the goal, then you might get a view that's skewed and you don't measure the right things anymore. And the same thing is with becoming customer-centric, is that it doesn't listen anymore to what the people actually want, because you will have the main way that people want to do things.

Patrick (31:01):

But what about dealing with exceptions? So you still need to get good at exceptions and learn about the exceptions. So a lot of the learning actually shifts from getting things under control to understanding more of the exceptions and what can happen there. I love to hear feedback. If this model resonates with you, we'll be releasing more of the maturity levels in each of the areas, in each of the parts that I mentioned. But this is my presentation. I tried to give you a holistic overview of what to think about in DevSecOps, how it relates to a team, and how people can overcome these first steps by spiraling up all these different areas that I think are important in the transformation. So thank you very much.

Lauri (31:56):

Thank you, Patrick. And truthful to the trend, we have a lot of questions for you. And we have time to go through them. So let me go to the highest level of abstraction on this and ask the question that starts from there, and then we start to move on from there. There was a comment not... Well, there is a question: missing the legal requirements on security, how to manage that?

Patrick (32:20):

In all of the cases I've seen, is that the legal is usually first reciting in the security team. So they have a good understanding of certain implications and they've been in contact with the legal team itself. So the team or the application team might not be aware of all those legal implications. Usually, then it starts with a session of an education where the security team explains what the world is, the team lives in, the domain and those areas. And then they do a joint call on what that could mean in their part of the application or what they need to take care of. So the first part is making it aware. Then usually the step comes about creating a process of when new things get added, or with the existing things to improve that.

Patrick (33:19):

And then the team starts asking the security team for, "Hey, we have this new thing. What about this? Could you have a look?" Then the security team starts creating some guidelines. "Hey, if you have this kind of tool, let's look at that. If you have that kind of tool, look at that." And they start building their list of requirements that teams could give the legal team and the security team to understand better how this fits into the legal framework. And after a while, you see this building up as a very close collaboration, something new comes in, the legal is advised, other people step in, and there's a process that is more and more refined to get that advice. So it's not the ivory tower anymore that goes into the audit and then says, we're not compliant, we're complainant. It is educating the team to make sure they understand in what context they are working in.

Lauri (34:17):

Thank you.

Linda (34:18):

There's a question from Annie on the future of engineering, a small one. So there's a lot of skills that overlap due to the DevOps way of working. How do we see the future? Is everybody going to do everything with no expertise in one or two fields? Or is there another way of working?

Patrick (34:39):

Yeah. It's a question that comes up quite often. It is very tempting to think we're all going to be the full stack and the full engineers. But that means often we will miss depth and it's easy to think that you're able to do this in one person. But in modern things, it's not possible anymore. So usually the best way to do it is to have a network of people you trust that cover other areas. But what really helps is that you have empathy and understanding of the other areas, but you don't need to have all the solutions, but awareness of that things are happening in security. Like we mentioned, the legal vulnerability in other areas, if nobody makes you aware of these things, you can't know them. But it doesn't mean you can keep up to date on all of these areas. So that's why I see the future in the teams collaborating with overlapping skill sets, but not one person or every person needs to have everything. There's usually a specialist to work on each of the fields. And that's, I think is the best collaboration process.

Linda (35:55):

Makes a lot of sense. Thank you.

Lauri (35:57):

So moving on, what if you don't have a security team and the management thinks it's not necessarily not a priority, and it's just a money sinkhole, how would you approach that to make the case for management?

Patrick (36:14):

All of the things that I mentioned in the presentation, it doesn't require a security team to tell you that it's a good thing. And that's one of the points that I try to make is that if you're a team that wants to level up their own game, nobody should be stopping you in a way. Jokingly, I said the same thing about DevOps. Did anybody ever stop you from collaborating between the two teams from the management? Probably not, but we had this in our mindset that it wasn't part of our job description to do this. And I think security is very similar. If you believe this is part of quality of your application, and we should do something about it, then it should get that attention. So the problem usually is around getting the priority on the backlog.

Patrick (37:09):

You can have a million things that are important and security could be one, stability could be one, and features could be one. Your product owner or your boss might say, well, I want to do something else first. And I have experienced myself many times that I would grumble and say, I don't agree with this. I don't agree with this priority. But I learned a lot by running my own business. And sometimes you have to decide that a feature is more important to gain us money than having it secure. And it is important that you have that conversation on the level and not just look at it like, oh, security is important. I want to do this. Oh, a feature is important. And the blended backlog between all of those areas is probably the biggest challenge of teams. I can change your boss. You might convince him if you bring good arguments, but you do have to listen about the perspective of the business as well. Why is important? Do we want to spend time and money on it? And to learn from that as well.

Linda (38:26):

Beautiful. Final question, how can someone who coined DevOps do anything else after that, and maybe more broadly, what sort of career paths do you see for the DevOps engineers and the audience, and what does the future look like for someone invested in this field?

Patrick (38:45):

Yeah. It's an interesting question. Well, it depends a little bit on your equation or definition of DevOps. If you look at DevOps as we're the engineers that build the containers, and now that everybody has containers, right? What's left to do? When I was working with a team that we were using a lot of external APIs. So a lot of the shift is currently going from using containers or services inside, to using external services supplied by other companies. And they might not even be in your own company anymore. So how can you control them? And I struggled a long time between, well, will an API from Google or Amazon replace us, and we're done? The funny part is, it just moved the abstraction layer, but now we are responsible again for dealing with Multicloud to make sure it keeps on working.

Patrick (39:50):

We have to monitor the clouds again, we have to monitor the services again. We have to make those reliable. We have to reverse engineer, how they work on the other sites, just in case things happening. We have to understand the restrictions. So we might not have been talking any more CPU and memory, but the same kind of problems we'll just keep existing. And everybody loves to say like Ops or NoOps, and we're not doing Ops anymore. It's just going to shift and be different. And that's what I see. We're moving to the services economy more and more. And that is what I think we'll be moving towards. And where I see tools like Terraform moving from cloud abstraction to more services abstraction, and deal with those at the higher layer.

Linda (40:42):

Wonderful. Thank you so, so much. There's a lot of discussions going on with the chat. So if you have a moment, go check it out and we'll move on to the next speaker.

Patrick Debois
Director of Market Strategy, Snyk
Twitter: @patrickdebois
Linkedin: www.linkedin.com/in/patrickdebois

Watch all recordings from The DEVOPS Conference 2021 on the event's page:
https://www.thedevopsconference.com/speakers