Talk: Going beyond DevOps with effective design and NoOps practices with Marko Klemetti, Eficode

Lauri (00:05): Hi there, and welcome to DevOps Sauna podcast. Eficode has organized the DevOps conference where its predecessors for a number of times. Last time in April 2021, over 10,000 people registered to the event. The DevOps Conference is a community event where attendees do not have the pay for the participation, and we are happy to organize it again on March 8th and 9th, 2022. You are all wholeheartedly welcome to join the event and register at thedevopsconference.com. If you are a DevOps Agile or cloud practitioner or a decision maker, I also encourage to review our call for papers. We are introducing a whole new community track to give more stage time for people with their personal experiences and insights. All of the speeches from the previous time are available online to watch, but to make it easier for people to enjoy these great speeches, we are also sharing them through our podcast. I hope you like them as much as I do. Marko (01:08): I made a podcast with Raman Sharma from DigitalOcean just two months ago, and one of the key findings of our podcast was that most developers just want to create and release their applications, without having to worry about DevOps. And I trust that the same thing applies to most of the development organizations as well. So organizations just want to create and really list their applications without too much hassle. Of course, DevOps is part of that, but still that's the key thought. And if we look at the best organizations, the best performing organizations, the link here will point to the DORA 2019 accelerate report, which has been mentioned a few times already during this conference. So if you look at the best performers, they have been able to achieve this, so we're talking about self-organized, being autonomous, being able to release production whenever you're ready. And having those technical advances in place also help the best performers to improve their culture, and help them build a cycle of good culture, good technical advances, doing experimentation and bringing good ideas from there, which you would call continuous improvement. Marko (02:37): However, then if we go and look at the other report, the puppet state of DevOps report last year, which was also mentioned, at least by Mick Kirsten yesterday, you will notice there that most of the organizations still stagnate to be mediocre. So high as 79% of organizations are still mid-level performers in DevOps capabilities. And at the same time as the high performers are getting better, they have created the continuously improvement cycle, they improve their culture, technology, and going forward all the time. The mid-level performers, are stagnating to the so called average level. And how do you know that you will be part of this average level? So you could ask yourself, are you still making releases by calendar or when your features are ready to be delivered? Is rollback in your new releases as hard or even impossible? Are you unable to do horizontal scaling for example? Or does your architecture block using continuous delivery or containers or modern cloud technologies? Marko (03:52): And if you answer any of these questions, yes, then probably you're still in the mid level. And what I'm going to say today is hopefully helping you to get out of there and start enjoying the so-called high performers and better culture. So what if I told you that for your organization and for most of the time, today, there should not be need for DevOps anymore. I honestly believe that technical advances available for all organizations today have made the concept of DevOps closer to that of NoOps. It doesn't mean that DevOps wouldn't exist anymore, but it means that the concept of DevOps has changed into something that regular development teams are consuming, getting the benefits out of, instead of having to build their own DevOps capabilities. So, you'll hear self-service and autonomous teams and all that, it all has to do with what I would call NoOps. Marko (05:00): And of course, if for DevOps, there are multiple definitions, so are there for NoOps as well and I wouldn't use that term too much. But the concept from what we had with Raman from DigitalOcean applies, developers, the development organization, designers, what they want is to get their applications out as smoothly as possible. And DevOps enables that, and at best DevOps would then be NoOps. So today, as we're having this conference, the high performers have DevOps more like a commodity or consumable even that has been put in place to serve the organization, to be able to deliver fast and better? I'm going to go back still, so why do I have the Mars Rover there? So it's perseverance, and also from the QR code, you'll notice that the QR code will point to the Mars mission from NASA. The reason is that if you read carefully, all of the articles that have been published about the perseverance Rover, you'll notice that there was a mention that the first thing perseverance Rover does upon its landing on Mars, is it updates its software. Marko (06:20): So as the travel to Mars was roughly seven months long. Of course there was time to develop the software further, the applications that the Rover uses. So once it landed, a new software was uploaded and then installed, and now it's in use in perseverance Rover. And I have to say that us here on earth have very little excuses to not do DevOps or not do continuous delivery, because we already know that you can do it definitely on Mars as well. So there are three areas that I'm going to be going through today. The culture of trust we have been talking a lot during this conference, I'm very happy about that because it's definitely number one. And if you look at the, for example, ethic codes, DevOps trends for this year, the culture of trust and psychological safety are number one. So it's something that has to do with DevOps, but you can still do it without actually talking about DevOps. And it's very, very integral part of well-functioning organizations and happy culture, enjoying the work, and of course being efficient. Marko (07:34): The second part I'm going to be talking about is understanding what areas need improving. As you've created your continuous delivery pipeline, and you have your tools in place, you have your pipeline up and running. There are still many areas that you can do to understand what areas need improving, and how to improve those. And then last but not least, I'll be talking about leaping over areas left behind. As the high performers, they have also come up with innovations that if you've left behind, can start using immediately and getting the benefits out without having to spend the same amount of R and D and time for creating the DevOps capabilities. From psychological safety, one of my favorite areas is looking back to move forward. So I'm pretty sure that you've bumped into Ron Western, and I think it was also mentioned during this conference, probably will be mentioned after me as well. So he did his research already in 2004 about the organizational culture technologies and how the information flow works in an organization, and what makes a high performing organization. And he found that there were few areas that all of the best organization technologies, the generative have that there was high cooperation, something we'd been talking over the conference, there was the shared risks. Marko (09:11): And then there was failure leading to inquiry which a is very important part. And also Patrick mentioned in his speak about reinventing organizations, that's also something you should be reading about, the organizational structures that work well. But going back, so failure leads to inquiry. One of the things I think haven't been mentioned yet during this conference is blameless post-mortems, and post-mortem for me is a pretty brutal word, so I decided to zone these in the picture. So the QR code will point you to the Google plus door explaining Western and blameless post-mortem. But the point is with blameless post-mortem, every time you have an incident or something happening, you gather around, you find out what happened, why did it happen? How do you make sure it doesn't happen again? And kind of the steps to go forward. And the point of the post-mortem it means to be blameless. Marko (10:17): So one of my startups, Freeed with triple E, it was my mistake, okay? I'll admit it immediately, but we did it without blame. I had misconfigured Kubernetes secrets for file uploads. And immediately when we found that out, we fixed it, it took two and a half minutes basically to fix it, of course, because we knew that it was broken. But still, even while we failed for two and a half minutes, we took a long blameless post-mortem to find out why did it actually happen? And was the ownership correct? And how to in the future, make sure that we don't end up in the same place, do we add automation to make sure that it doesn't happen? How do we make it in the culture that we know that something that like this could happen and such. And at the same time with the blameless post mortem, you get to talk to people in a safe way. And also, you often get to meet people that have been involved in an incident that you don't necessarily talk to everyday. Marko (11:30): One of the things that that Western also mentions is bridging. 2007, I incidentally did go to Brussels, there we had Kip Con organized by Jeff and Paul Julius, and there was also Squirrel. So yesterday you hopefully heard Jeff and Squirrel, and I heard them talk there the first time, and I even had some pieces of the coal or code inserted into cruise control. Which Jeff then merged my pool request back then. But there, I heard the first time, the saying "build bridges instead of ferries". And the point of building bridges is of course building autonomy, and capability of actually having the discussions instead of some media with which you discuss. And usually this leads to making the decisions at lowest levels possible, and it usually enables autonomous or self-organized teams. So you would have the designers and developers, and even maybe some DevOps people in the teams, the same way as Patrick mentioned in the previous session. Marko (12:48): While building bridges, you also have to remember the accountability. That's also something that Patrick [Debois] talked about. When you have autonomous teams, you might end up doing small things very fast, but big things, very slow. So there are two things that you might want to add while building the bridges. The first one is what Western would call claiming the messengers, which means that everybody understands where we're going and how the organization functions together. And then creating, for example, what we in DevOps would call communities of practice. So people interested on the same topic, whether it's a technology or some area of the software or then some cultural topic, for example, would have a community of practice where people meet up and discuss. Bridging takes me to design systems. In my opinion, and as I've been talking for the last few years already, design systems are the so-called turnkey solution for your CD and releasing. Marko (13:57): So design systems for those who don't know what design systems are, usually it's the identity and voice of an organization. And I guess the common description would be a predefined set of visual components that you would use in your user interface. Whether it's a website or mobile application or maybe even an embedded environment, you have predefined sets of components that would present your organization, and that are reusable. And unfortunately the design systems often are left there, but I trust that design systems could also include many parts of DevOps, such as the test templates, so how do you test your components? How do you make end to end testing such as acceptance testing? How do you plug in your application to the code pipeline or continuous delivery pipeline? How do you define your infrastructure needs or the infrastructure that you're going to be using in your end application. Already from the design phase and that, if something, is bridging. So you build a community that's working on one source of truth, which would in this case be for example, the design system, throughout the organization. Marko (15:17): And then also once you kickstart new innovations, you pick the design system and you start using it. The QR code will point you to the podcast we made with Vaisala, a Finnish embedded development organization, they called their design system rock hopper, which is pretty interesting. And I've also written a few blogs on the topic, so if you're interested, I'll share the links afterwards. The second area I'm going to be talking about is understanding what needs improving. So former Praqma, now part of ethic code in Denmark or based originally from Denmark, they created a pipeline game which is a card game to define or build pipelines. It's easy to find, its pipelinegame.eficode.com, I'll send the link afterwards. But it's really, really interesting exercise to go through your pipeline together with your team or organization and define what steps are happening when your application is being developed from the idea all the way to production. And how much time do you each step? And then look into how you could improve that. Marko (16:37): So even without any technical advances, you can just gather together and then go through the pipeline and have the discussion over that, I recommend you to try it out. Unfortunately, since it's from Denmark and Copenhagen, it also brings me memories from there, pre-Corona time. And there's a place called Rev Saloon, which has a brewery called Mikkeller. And when I think about continuous delivery pipelines, I cannot help but think also the breweries and what's happening in a brewery when you make some beer. So if we continue with that theme, you will see that from the puppet state of DevOps report, one of the big things from last year, which we've also noticed that ethical is that 63% of organizations, roughly, have already a DevOps platform in place. And most of the organizations have even been able to make the DevOps platform a self-service place, which means that you can use the resources from your platform individually or independently as a team or part of an organization. Marko (17:56): And that's something that combining the design systems, which would be the key chain, the DevOps platform would be your turn key for delivery. And especially in big organizations, I trust that this is the way to go today. And having a DevOps platform that has self-service in place, has been shown to correlate also with high performance. So once you have the DevOps platform in place or the brewery, one of the important parts is to be able to measure your throughput to of course, better identify or bottlenecks, pun intended. And if we look at, for example, Unity, which stock listed just half a year ago, roughly, and their stock peaked at three times the initial value. We did an assessment for them from Eficode, and they got the best points that we've seen so far. Marko (18:54): And one of the reasons why they actually got so good points was that what they do is, they have their releases connected to their business metrics. Which are, for example, retention and revenue. And when you make a release connected to your metrics, you can actually do a cannery release, or A and B testing in production. So you make a small release to your pilot audience, and then you measure how much does it change, for example, revenue retention, your learning models, whatever you're using as to measure your business metrics? And only after that make a decision, if you're going to release it or not. And another good example would be from Smartly, one of the links I'm going to share is Smartly's culture handbook, they don't have staging environments at all. So once the PR is merged into main or master branch, it goes live. And that's just how it goes, easy to measure because it's live. Marko (20:03): There will be a circle CI talk later today, but I'll have something said from their blog, they've made some metrics you can look into if you want to measure your throughput. The four metrics they have published is throughput a duration of your workflow, MTTR which has been time-to-recovery, and then build success rate. And I think DORA's four metrics are pretty good for starting with measuring your delivery pipeline and understanding where your bottlenecks lie. Then the hot topic of this year is finding KPIs to track your DevOps investment. Yesterday, Mick Kirsten talked about the flow metrics and his book project product. And he talked about the metrics they use, for example velocity, load, efficiency, time, and as fifth, the distribution. Those areas are something that you can start measuring once you know your throughput, and once you've started measuring your customer value, it's time to start also looking into, how do you track the investments you do in your DevOps advances? Marko (21:12): Link will point to you to GitLab, they actually have some of their own KPIs live and publicly published in the internet, which is actually pretty interesting. So they also have customer happiness and pull requests and merges per time and merge amounts as their main KPIs for development, it's an interesting read. The last section for me is leaping over areas left behind. And these, especially for those who are struggling with your DevOps transformations are thinking, how can and you be faster? How can you implement the platform? How can you implement your continuous delivery? Good news, the organizations that are high performers who have invested in DevOps already, and because of the big open source community and the DevOps community your part of today by joining this conference already, have made it possible to jump over many of the areas that have already been left behind. Marko (22:17): So to give you one example, following of the brewery team. So instead of having your own, for example, Jenkins cluster or maintaining your continuous integration clusters and servers, and making sure that not only your bilds pass, but also that the cluster and continuous integration, continuous delivery tools are working as expected. Today, setting up continuous delivery takes a few minutes, so this link goes naturally to GitHub actions, which is one of the easiest ways of starting, for example, your open source continuous integration, and you'll have similar environments from circle CI and you would have code pipeline and AWS and Azure pipeline similar. They all have been built in such a way, you have your code, you have the recipe, and the rest has been taken care of. And it's actually amazing to see how much faster do modern organizations get to what we really did struggle with when DevOps, as the name was first coin, just by the advances that have been made in public cloud and cloud-like environments, and also the way you define your pipelines and how you get everything plugged in. Marko (23:41): So I would say don't do Jenkins anymore, go to use some of the services that provide everything as ready made. You might have Jenkins still though, but then it would be part of your platform and for your developers, it would then be a self-service environment. The second one is keeping creating and going directly to consuming services. This is also something that public cloud has made basically mainstream now, but we see from all the researchers that it's not actually yet true. So there's lots of legacies still not running in cloud environments, and something that still surprises me is that how much organizations spend time on maintaining their own clusters, or set a server environments and everything. Even in the era of, if we look at EU and GDPR and the legislation that impacted also the public cloud, still it's not so easy to move to public cloud or cloud-like environments. Marko (24:43): If we look at the brewery example, the Mikkeller from Copenhagen, they barely do any beer themselves, they actually use mostly the Proef brewery from Belgium who is producing most of their beers. So they make the recipes, they send it over to someone to take care of, well, brewing it, and then of course shipping it. And also that applies to cloud environments today as well. So the QR code you'll see there that points you to the Google Kubernetes autopilot, which was just released a month ago, roughly. Where basically, even if you look at formerly would use Herco, for example, or server-less which you heard yesterday from the Lego and AVS presentation, there's now also the enterprise option, which is provided by basically all of the public cloud providers, Kubernetes. And even in Kubernetes, you can already go autopilot, which means that you just have recipe, you throw it to the cloud, and then it runs to your customers. Marko (25:51): The last area I'll be talking about today is data pipeline being a plugin. To give you an example of providers, Datadog or Datadog, whichever you prefer, they stock listed one and a half years ago, and they've also skyrocketed. What the modern organizations providing data pipelines or data collection and metrics are doing, is that basically it's the same way as you would set up the continuous delivery pipeline, it's just an installation of agent or installation of a configuration. So it could be Datadog or logs IO, which unity is using, or then all of the public cloud providers already provide their own metric solutions. And in modern data pipelines, I don't even see the collection of the information as a difficult thing. The difficult thing comes from what the high performers have done for long time already is finding, making machine learning, which actually fits for data pipelines pretty well, and pattern record ignition and being able to track and modify your cluster pay based on the information that you collect. Marko (27:10): So if we take a look back of the presentation today, the fact is that still most organizations stagnate mediocrity. It's partly because of the whole legacy they're carrying behind and the culture that does not enable the things that are available for high performers. And usually there hasn't been the good cycle of continuous improvement where you will be improving your culture and technological advances at the same time. We are approaching the end of the conference, you've already seen that there are so many areas that you can start already working with to get to what I would then call a concept of NoOps. Thinking of the developers and designers of your organizations as a set of customers that you provide the services for. And building an environment which starts from, for example, the design system, and building a community around the design system that includes DevOps. And providing the self-service environments and healthy culture where whatever fails you go back and look what happened and embed it in your organization in a safe way. Marko (28:28): The three areas that I explained you can start doing, the first one is starting to build a culture of trust, psychological safety. One of clearly the main themes of this conference, you will see that there are many areas like yesterday's Agile conversations where you can do it yourself with a pair, in a team, having the discussions, understanding where the discussions were had. Or you can look at the Westerns organizational typologies, and go into seeing how high performing organization functions, how can you take those in place in your organization? How do you have the bridging in place, how you make sure that the information is flowing through the organization, and there are no stops where you would have to write information to someone else, but rather collaborate or build communities of practice. Marko (29:26): The second part was understanding what needs improving. You can do it either by the card game, such as the pipeline game, or having any design sessions looking at your service design or how your current DevOps environment has been built. And then dropping in metrics or KPIs that you want to track, either to see how you're performing towards your customers, or then how your investments are doing their return on investments on certain metrics that you have to basically, most of them define your own. And as said in this conference, you'll see many places where you can have those metrics picked, and then you'll shuffle your own deck of those. Marko (30:13): And then third and mostly important, now that we're already in 2021, DevOps has been around for soon, 10 years. You'll see that there are many areas that you don't have to touch anymore, but instead you can start just consuming. Whether it's continuous delivery, the cloud environments, where the DevOps has been taken care, either for you, or then provided by your own SRE DevOps teams as self-services for your organization. Third part of leaping over is of course, the data collection and understanding what happens in your customer environment, how your customers behave in your service, how do your servers behave? And of course what's happening in your production pipeline? And how do all of these things connect to each other? Lauri (31:06):