Interview: 2019 State of DevOps with Johan Abildskov

Heidi (00:04): Hi there, it's Heidi Aho from Eficode and Praqma and welcome to DevOps Sauna, Eficode's podcast about all things DevOps in Europe. It's sweater weather here in Helsinki, which means the leaves are starting to fall from the trees but also that there has been a bunch of interesting states of DevOps reports dropping into inboxes and Slack channels around Europe, which is our topic for today. Heidi (00:31): The first of the reports that came out this autumn was DORA's Accelerate State of DevOps Report. It came out a month ago. DORA stands for DevOps Research and Assessment. It was recently bought by Google Cloud and Cloud featured hugely in this year's report. The second report we're discussing actually only came out yesterday, a bit longer than that for you guys listening, and it comes from Puppet. It's Puppet's State of DevOps Report 2019 and security featured heavily in this report. Heidi (01:05): But that's enough of a preamble. Today I'm joined by Johan Abildskov from Praqma but it gives me great pleasure to say that he's also from Eficode, as Eficode and Praqma joined forces in May this year. Johan is a continuous delivery and DevOps consultant and teacher and last week he spoke at DevOpsDays in Stockholm, which is only one of the cool things he's regularly doing. Heidi (01:31): Hi, Johan. Johan (01:32): Hi, Heidi. Thank you so much for having me. Heidi (01:34): The pleasure is all ours, seriously, I've been looking forward to this for ages. Johan (01:39): Yes, me too. Heidi (01:41): How about you start by introducing yourself to our listeners. Johan (01:44): So my cheesy catchphrase is that I didn't write the book on DevOps but I did write the DevOps song. I opened the DevOpsDays Copenhagen two years ago with a, in Danish lejlighedssang, or song for the occasion. So you can find that on YouTube for much embarrassing of myself. Heidi (02:05): The lyrics are insane for that song. It did do the rounds on Slack when our companies merged. Yeah, fantastic. Johan (02:13): And then my day-to-day job, when I'm not wearing my singer/songwriter cape with the ukulele and fun stuff, I spend my time doing DevOps with customers, upscaling, taking many, many discussions on the how and why and especially perhaps the how and whatnots of doing DevOps. So I see myself as a geek, then I found out in order to play with the cool tools I have to care a lot about humans. So that's a difficult transition. Heidi (02:52): I've also heard you are an Opinionated Git. Johan (02:55): Well, I had some good discussions on a podcast on Git, and as any sane geek would do, I immediately bought a domain name and dumped some content there. So on opinionatedgit.com you can find out how you should be using Git. Heidi (03:13): Yes, and it's a great name, absolutely fantastic. It is true. I would say you are opinionated but Git refers to the tool, not your personality there. Johan (03:26): Thank you. Heidi (03:26): You're welcome. Okay, so we talked about discussing, in our podcast, these state of DevOps reports, which are dropping like autumn leaves and I'm super excited to talk to you about these because you've actually written a blog about DORA's State of DevOps Report and you are quite a big an of DORA's yearly reports. You know a lot about them. Johan (03:51): It's such a valuable resource, both internally and when we, as a consultant, talk about doing DevOps because it's actually the research that enables us to say that it's just not, it's not just that you have to do DevOps because we say so or because we feel it's right, but it's actually the fact that if you're not at least trying to adopt DevOps practices you're knowingly doing worse than you could be doing. So I love the fact that it's a bad ... it's shown to be a bad business decision not to do DevOps and that's where the state of DevOps reports are such a powerful tool. Heidi (04:36): Would you say that they're like the science of DevOps? There's a scientific element to these reports, at least they claim to be surveying all these professionals, et cetera. Johan (04:46): I think it's the, especially the DORA DevOps Report that had 31,000 respondents this year, I think. That's some serious statistical analysis they in there so it's the best science that we have on what works in terms of practices in software delivery and DevOps. Heidi (05:07): Yeah, I don't know if this is nitpicky but I wasn't sure in the report whether they were like, "Okay, we've done this for six years running. Over those six years we've surveyed 31,000." They are super academically grounded and Nicole Forsgren actually spoke at CoDe-Conf, a continuative delivery conference back in 2017, which is pretty cool. Johan (05:29): Yes, so Nicole Forsgren is the elite or chief scientist of DevOps, I think is her job title these days at the DORA and spearheads the awesome team that does this research and it's really a lot of great work, in terms of dissemination, as well. It's not just the findings that's interesting but just being able to convey all that valuable information in a digestible format and in a format that we can actually hand upwards in the organizational hierarchy and still be actionable and useful and understandable. That's a very difficult task from an educator's perspective. Heidi (06:12): Okay, let's dive into the results of DORA's Accelerate State of DevOps Report, 2019. Johan, what were the top line findings about how companies can become elite performers, according to this report? Johan (06:26): I think one of the really nice things is that the report this year reaffirms the findings from previous years. So it's not like we have to just say, "Now, DevOps is in a completely different direction." So that's nice. We are still doing the right thing. Johan (06:47): They have added a bunch about how you should approach DevOps transformations and productivity and there are some good actionable items there. One of the things that I really like is that things like automation is becoming a permission to play capability. It's not something that we can add on or something that we should consider a luxury. Johan (07:15): It's something that is necessary and if we look at the degree of automation that is present, even in the low performing performance profiles, we can see that even ... basic automation is something that is what allows you to be doing software development rather than something that allows you to be high performing. It's just a prerequisite for even getting on the field. Heidi (07:45): Yeah, it's like DevOps, it still is a competitive edge but it's gotten to the point where if you're not doing it, you're definitely losing. Johan (07:55): Yes, exactly. Heidi (07:56): You can't get around it. Yeah, I'm just looking at that automation table that they had in the report and it does say that even the low performers, 64% have had an automated build in place. I mean, the leads like that was 92% of them had it in place but that is the majority of these low performers still have these building blocks or some of the building blocks in place. Johan (08:20): Yes. So I think it's more about at least figuring out what parts of automation we can do and there are some low hanging fruits. For instance, as you mentioned, build automation. That's a bunch of even the low performers, most of them still have it, even though it gets higher as we go towards the leading performers but automation is such a key principle and it's oftentimes an excellent place to start doing some stuff because when we start to automate things like testing it puts requirements or demands, it puts some tension into our architecture, is, "Have we architected for testing, is our code base testable?" Johan (09:09): It puts requirements on the way that we think about our infrastructure. "Are we able to position an environment that we can perform testing in?" It's like if it's sweater weather in Helsinki then the little thread that we can start pulling to unravel the sweater of technical debt, that's the, perhaps test automation. Heidi (09:31): There's this misconception among these low performers that DevOps equals automation, that the DevOps engineer is just someone who automates stuff and even though that isn't the full picture you could be automating a whole lot of process that shouldn't be there, for example. Johan (09:49): Last week in Stockholm I talked about how automation is hard and we're doing it wrong. And one of my points is actually that how we approach automation is hugely important. One of the definitions of DevOps is this CALMS acronym. That DevOps is basically culture. It's automation. It's lean, it's measuring and it's sharing. But that's very, very fluffy or vague. It's not very actionable. So if we look at those from an organization wishing to get started with DevOps, from that perspective then we have these five areas that we can look into. Johan (10:27): Culture, that's obviously way too fluffy. That's difficult. We don't know how to approach that. Lean, that's boring. It's just from the auto industry and production lines and things like that. Who wants to do a lean transformation? Then we can look into measuring and we don't want to measure stuff because then we will be kept accountable for stuff. That sounds dangerous. And then we get to sharing, and sharing, that's not going to happen in a traditional enterprise, right? Johan (10:53): So we're left with automation. Automation, that feels concrete. That feels like something we can do. That feels like somewhere we can start doing stuff. But then automation becomes our first step towards DevOps. So the way we think about automation, the way we work with automation, the mindsets that we build around automation becomes tremendously important because it's our first step towards DevOps. And that's just where there are so many misconceptions that we have. Things like, is automation a luxury? Johan (11:24): I think it was Peter Drucker who said, "There's nothing so wasteful as doing very efficiently that which should not be done at all." And I think that was also one of the things that you alluded to. We shouldn't automate things, parts of processes where the product, the process would be better off being removed. So that's our favorite automation process. Let's stop doing the stuff. Heidi (11:48): Amen to that. It sounds like you're describing automation as this starter drug for DevOps, which I find amusing. Johan (11:55): It's at least, I think, I can understand that from DevOps adoption perspective, that it's the easy starting point or it's the obvious starting point because it feels more concrete than these different areas. Heidi (12:09): Fantastic. We've spoken about automation, the report talks about automation. Let's move on to culture, which is a clinch point for companies wanting to scale DevOps in their organization. DORA's report does tackle this issue in a really interesting way. How do you organize a DevOps transformation with community building in mind? Johan (12:32): So that's a big question. I think that the report goes into different areas of trying to scale DevOps and they describe different approaches like grassroots or big bang transformations or building communities of practice and see what are the generally successful ways of doing DevOps transformations. Heidi (13:02): But they group them into these four categories like community, university, emergent, experimenters. I don't think that's as meaningful to our audience but I felt like I should interrupt you with that anyway. But, please continue, Johan. Johan (13:13): Thank you. At least based on the report they find that the way that they can anchor a transformation or anchor the DevOps knowledge in the organization and not just on silos or islands are by, primarily by building these communities of practice. And basically it's, whether we call it guilds or chapters from the Spotify model or whatever we call it, it's about building a sustainable community about the things that we want to do, whether that is test automation or infrastructure as code. Johan (13:51): We want to have some way that we continuously can have our own experts upscale local expert ... local ordinary users and we can have a solid foundation of knowledge built in the teams and these communities should be continuous things, continuous thing that becomes a part of the way that we work. So, we can become more stable and our organization can become more robust in terms of employee turnover or shifting priorities or having key capabilities on a small amount of persons. Johan (14:34): So building a community of practice, creating a way of working where we can share knowledge in the areas that we have, that is something that is key to scaling DevOps transformation because it's in big organizations that we need to spread a lot of knowledge and change a lot of mindsets. Heidi (14:58): Mm-hmm (affirmative). Johan (14:59): But, the very difficult part about this is that all this community work or building these practices, that's, of course, not value add in a traditional sense because it doesn't go into the product and that can make it very difficult to get and keep the priority. Heidi (15:20): But surely it does go into the product though because by enabling DevOps you increase speed. You increase stability and community is what does that. So surely the end goal is that you get more value to your customers faster. Johan (15:36): Obviously, but it's indirect, right? It's enabling a faster flow. It's enabling a more consistent quality but you might say the same thing for sending your developers on a course and test for development. But I assume that's, or at least in my experience, that can be a very difficult value proposition to make. So I don't only have to pay for my developers to go to a two day course. I also lose two days of productivity. It takes some belief that it will ultimately be enabling more business value to make that investment in both money and time and the same goes for building communities. Heidi (16:21): Yeah, I don't think any of those practices take less than two days to set up. It's a huge organizational change really that these elite performers have undergone through this matrix of practices. Johan (16:36): Yes, so the point is that it can be very difficult and if the transformation fails to get some early wins, it can quickly be deemed as, "Well, DevOps doesn't work here." And that's where we point to the DevOps report and say, "Well, it's probably you that's broken and not DevOps." Heidi (16:55): Yeah, just stick with it and- Johan (16:56): Perhaps in kinder words. And this is where we say that you should book a meeting with your DevOps coach. Heidi (17:02): What caught my attention there was these huge enterprises and in the report among these low performers are companies that are ... consist of more than 5000 people. So for huge enterprises DevOps is very difficult. That's what you were talking about just now. That being said, there were some outliers, as well, and this is the first year where an industry stood out as a high performing industry and that was retail. What did you think about that, that retail is outperforming other industries? Johan (17:43): So first off, I was a bit surprised by the fact that when we had a standout industry it was not a low performing industry. That would have been my intuition, saying that, "Well, either the heavy industries or the finance sector or the health sector, they are not able to become high performing." So I would have expected that we saw an outlier in the other end of the scale, actually, but then it turns out that it is retail. Johan (18:11): So a completely run-of-the-mill, ordinary industry and not just that, they're high performing. It's more likely that a retail company doing DevOps are higher performing than other industries. And that's not very intuitive, at least to me, because I don't immediately connect retail companies to being high tech because it's about stores. It's about shopping. It's about all these things that we don't necessarily see in a every day high tech light. Heidi (18:46): But what about Amazon? Johan (18:47): Yes, but- Heidi (18:48): Retail is increasingly online. Johan (18:51): Obviously, there are companies that stand out and perhaps Amazon is the uninteresting example because that is a very digitally, at least intuitively, a very digitally minded company. But then we get Walgreen's, Nike, whatever we have, or websites, companies like Booking.com or Zalando. But being in industries where the margins are so small and where there are a heavily competitive environment that any edge that you can get is necessary and actually having a culture to where you experiment on being more productive and getting more value out of your investments, that's just both fits naturally within the DevOps mindset of continuous experimentation and be data driven in your decision making. Johan (19:43): And it also has forced them, perhaps, to be early adopters in any area that can give them any business advantage. So they have been in a highly competitive environment for so long that they are ... they're just poised to take, grab any advance. I met an independent consultant at a Git conference whose full time job was making open source Git on behalf of Booking.com and that was not a connection that was obvious in my mind. Heidi (20:26): Okay, so let's go back to Amazon, but only briefly as a bridge to talking about Cloud. So, Johan, could you talk us through the DORA State of DevOps Report mentions these five capabilities of Cloud computing and they say the elite performers have them. They're already using Cloud the right way and the low performers, perhaps need to learn how to use all these capabilities. So could you talk us through those? Johan (20:54): Yes, so let's start a slightly different place, where let's say that I have a traditional IT organization. I have on-premise. I have hot, where I have service and the way that I, as a developer, I get access to resources is by opening a ticket or sending an email and then after some period of time, months are not unheard of, I get an email and I have now access to my new resource and then I can do my work. That's one way of working. Johan (21:23): So we have heard that this Cloud thing, that's where all the huzzahs are, so a person high up in the hierarchy decides, "Well, we should go Cloud first." And what we do now is that we don't have hardware. We have stuff in the Cloud but I, as a developer, I send an email or I create a ticket and then after some time, perhaps three months, I will have access to my new resource. So from a development point of perspective I have gotten no agility. I've gotten no benefit. I've got no control. I have gotten very small value from the enterprise going into the Cloud. Johan (22:04): So that's actually the misconception. Well, we are in the Cloud and the point is that, well you might be using public Cloud resources but you're using them completely as your on-prem and then you're probably not getting a bunch of the value. So the American Standards Institute has created these five, or described these five Cloud characteristics and it's the on-demand self-service. So you don't need to create a ticket. You can provision the resources that you want without the interaction with another human being. Johan (22:42): The resources that you provision needs to be generally available on networks. So they call it broad network access and this basically means that you need to be able to go attach to your resources through ordinary means. Heidi (22:57): As in on your mobile or just ... Johan (22:59): It could be. They actually mention it through the browser, through computers, through the mobiles. So it's not necessarily that the resource needs to be publicly available but it needs to be easily reachable from the developer's point of perspective. Johan (23:16): Then we need to have resource pooling. So we shouldn't care about which resources we get. When we're done with the resources we just throw them back into some common pool of resources. And that's kind of a boring one. Johan (23:28): One of the more interesting one is that we should rapid elasticity. So if we need to scale from the perspective of the engineer who needs more resources there should be an infinite tap of compute and memory and storage that we can just turn the dial and then we get more, quickly. Johan (23:47): And last characteristic is that we should have measured service. So basically, we pay for what we use. And none of these characteristics tell us anything about where our actual hardware is but these services come more or less out of the box on the public Cloud, like Google Cloud, like AWS or Azure, even whether we are on-prem. Johan (24:10): We can, of course, build these capabilities into the service that we provide to our engineers but it will probably be very costly. But it is doable. In the same way we can take our things to the public Cloud. We can get Azure or AWS engineers and still live with the same pains that we did when we had things on site. Heidi (24:34): As I understood it, the organization has processes in place that stop them from making full use of the Cloud, so what can they do about that? Johan (24:43): So as usual tooling is easy and humans are hard and this is ... you actually said it right. They have processes in place that rejects this way of working. So it's about figuring out if your way of working should support using Cloud infrastructure and infrastructure as code. And I think that this again, ties into management or buy-in for management. If you have gone through a DevOps transformation and you haven't changed the way you're working, you have failed. If everything is kind of a trade off. So for instance, the trade off between being on-premise and on Cloud could be something about data security and integrity. Johan (25:27): I think that argument is getting weaker and weaker because Cloud providers get GDPR out of the box and get certified to run the American healthcare system and things like that. So it's getting, at least for me, a weaker and weaker argument and I think it's security by we think we have it rather than necessarily active security. Johan (25:52): So everything is a trade off and I can understand that there is some inertia and saying, "Well, we have this on-prem and it is working and it will be a huge investment to go to the Cloud because it's a migration and we need to teach all of our engineers to do different tasks and things like that." And that's not necessarily untrue but it's a matter of actually framing that decision in terms of data and in terms of facts and hypothesis rather than saying, "Well, it feels nice where we are." Johan (26:28): And things like accepting that we have a ton of hidden costs by running our infrastructure locally because it's from a different box, because we have already paid the engineering salaries so it doesn't matter that they also have to do something OPC on out-of-factory or in Jenkins and things like that. Johan (26:48): Also, the report this year shows, they find that people using the Cloud according to these five characteristics are also much more likely to be able to predict their costs of operating the services. So I think that's a very key capability and I think that makes it clear that one of the benefits of going to Cloud is actually more transparency in costs, which kind of is weird because you would think that it would be easier to figure out what stuff costs when you have it in-house, but that seems to be not the case and it also is making sense to me because there are tons of things figuring out, "Well, you're half assigned to this team and one third to this team so how much is your salary impacting the cost of renting a server from your department? What does power cost? What does networking cost?" Heidi (27:42): So we're coming to the tail end of our discussion about the DORA State of DevOps Report. I'm curious as to whether you'd like to talk to us about change approval process because they do mention that heavyweight change approval processes are a negative effect on DevOps and speed and stability. So have you seen this phenomenon in practice at your European clients? How can companies change these processes that aren't working for them? Johan (28:15): So, I think there are two interesting misconceptions and one is that if we look at speed versus quality or speed versus stability, some organizations feel like they are at contrast or in conflict with each other. So either we are quick or we have quality but it actually shows that's it the other way around. Johan (28:44): Being fast, that enables us to get fast feedback. That enables us to experiment. That enables us to both quickly deliver fixes and bucks to production and having the quality to actually push things through a pipeline, to actually have some reliability that things will work as we expect when we release them or our pipeline will not just randomly fail. Johan (29:11): That enables us to be quick. So, again, DevOps is all about feedback and feedback loops and speed and quality are actually, when we try to drive them a bit, shown to improve each other because they enable each other. Quality enables speed and speed enables quality. Heidi (29:29): You don't need to sacrifice stability for speed and that's one of the main messages of the report, for sure, and it's counterintuitive. Johan (29:37): Yes. Yes. So that was one misconception, in terms of changeable processes. And another from the State of DevOps Report from last year, actually they found a cluster of organizations that they called misguided performers and the point of these were that these were organizations that were very risk averse and very slow at deploying their things and generally succeeding. So having a reasonably low failure rate and arguing that, "This is because we're very careful. This is because we do due diligence. This is because we're very thorough." Johan (30:19): And that's a good thing, one might argue, but what is shown is then that these companies then have a much larger time to recover from failures. So I think on the average, this cluster of organizations, they took between one month to six months time to recover from a failure and that is business destroying down time. So this feeling that we are actually handling risk by slowing down, that can also kind of be a misconception because we're taking much larger risks at a time. Heidi (30:57): That brings to mind one of the standout stats from the report, which was that elite performers, according to DORA this year, recover from incidents 2600 times faster. So that's huge stability, if you're recovering from an incident that quickly. It's like light years. Johan (31:19): Yes. And not to mention what that does for employee satisfaction and burnout. Heidi (31:23): Yeah. It just makes you feel a lot safer at work, that there isn't this constant cliff edge of what will happen if I make a mistake? Johan (31:33): One of the key points in continuous delivery is actually making the release just a business decision. So also another misconception that's kind of tangential is that the goal is release all the time and I don't care about that. We don't necessarily need to release like Amazon and Google does, thousands of time each day. But what's important is that we can release when we want to do it and it's a nonevent. So we want on-demand releases. Heidi (32:00): Yeah, no fear. Johan (32:01): Yeah, yeah. Exactly, it should be a nonevent. It should be boring to release. We do it when we want to and it's never a surprise. Heidi (32:09): Johan, thank you so much. I think that's enough on the DORA State of DevOps Report for now. So the writer of report, Dr. Nicole Forsgren, spoke at Praqma's code conference, which is in the next edition of that, is actually around the corner. So CoDe-Conf takes place on the 28th of October, 2019 in Copenhagen. So do head to code-conf.com to read more about the program there. Heidi (32:37): This is a beautiful dovetail because this year at CoDe-Conf, it's actually Andi Mann whose one of the speakers. He's a chief technology advocate at Splunk and he was actually one of the writers of Puppet's State of DevOps Report, which came out yesterday, which I wanted to conclude our interview with. Heidi (33:01): Johan, you still there? Johan (33:02): Yes. Heidi (33:03): What's your opinion on Puppet's State of DevOps Report? Johan (33:06): Not having read it in nearly as much detail as I have studied the DORA State of DevOps Report, I think that it's very interesting that it's very themed. So it is a heavy security theme and I think that, again, the key takeaway is that just as with change approval or testing and things like that is that we shouldn't tack quality or security, all these non-functional requirements on at the end. We should build them in in our way of working. Johan (33:41): So an audit becomes easy because we have already automatically ran most of the audit as part of our delivery life cycle, our delivery pipeline, and it's just becoming ... there are so many security breaches, right? There's not a week where someone hasn't accidentally leaked a million social security numbers or credit cards or something like that. And that's one thing and the other thing is being, companies more and more are seeing ... have their infrastructures taken down by some sort of hacker attacks and that just means that we need to better improve our way or working. Johan (34:29): And what's very interesting is that companies that actually manage to build in their security guidelines, whatever we call them, their security practices, into their way of life, they don't see a slow down because of ... a slowdown in productivity because of security reviews or security tooling. Heidi (34:52): Yeah. That is surprising because security is seen as this thing where it's not a differentiating factor. It doesn't pay the bills so why should we prioritize this security project over a new feature? And that's why companies are getting stuck where they're not integrating security into their software delivery life cycle. That's the, Puppet makes the case that the high performers, best performing, elite performers, they're all doing this. So you need to get on it, as well. Johan (35:23): I think it's also, again a matter of risk management. So I'm also quite certain that fire prevention and fire fighting capabilities in a production hall, that doesn't go into the product and that isn't a value add but we still prioritize doing fire drills. Johan (35:42): And that might be due to external requirements but I'm not sure that ... the same way that I'm not sure that I have factory hall that I can afford burns down. I'm not sure that the organizations I'm in can afford a high level breach. So what is your insurance for that, right? And that's something that's very difficult to insure, it's something that can potentially be disastrous. So at least how can we mitigate or lower the terms of impact and that's just an investment in robustness or reliability. Heidi (36:24): Yeah. Those who are doing DevOps well are, should, in theory at least, be doing security well because security is another silo and if you've got ... if you're scaling DevOps across your organization then you are breaking down those silos, although, it's so interesting. What I found interesting, I mean this too is semantics but Puppet, in their report, they were like, "Well, we don't call it DevSecOps, we still call it, we just call it DevOps because it is just part of DevOps." And that's an argument that I've heard a lot at Eficode as well, in terms of ... I mean, you can call it DevSecOps but really if you're doing DevOps properly then security is baked into your software delivery lifecycle and that is a shared responsibility. Johan (37:10): Yes, I think that one of the interesting points is that just DevOps is a very bad name for what it is because we say, "Well, let's build all ... Let's destroy all silos but we'll build our own silo over here." Heidi (37:25): Also, it's just two groups of people and it's beyond that by now. Johan (37:29): Yeah, but we know that in software engineering naming is one of the unsolvable problems. So I understand how we get there. Heidi (37:37): Yeah. Thank you so much for joining us, Johan. This was our first ever long distance podcast recording, which is so exciting because we have all these offices all around Europe and we hope to have you on the podcast again. How does that sound, Johan? Johan (37:53): It has been a pleasure to talk to you about DevOps and I'll be happy to keep ranting. Heidi (37:59): Yes, please. Please do keep ranting on Slack and then at our next podcast interview. Absolutely. Have a delightful afternoon in Aarhaus. Johan (38:09): You too. Heidi (38:10): Thanks for tuning in to DevOps Sauna, Eficode and Praqma's podcast about all things DevOps. Do have fun reading those State of DevOps Reports and we shall see you next time. Bye.