A product-driven approach to platform engineering

[Emma] (0:06 - 0:13) Yeah, I like to ask the question why. A great tip for anyone listening, keep asking why until they're annoyed at you. [Marc] (0:19 - 0:28) Welcome to the DevOps Sauna, the podcast where we dive deep into the world where software, security, and platform engineering converge with the principles of DevOps. [Darren] (0:29 - 0:32) Enjoy some steam and let's forge some new ideas together. [Marc] (0:42 - 1:12) We are back in the sauna and we've got some dear friends in the house today. I'm super excited. This is a pregame for the DevOps Conference Scandinavia, which is coming to Copenhagen, November the 5th and Stockholm, November the 7th. I'm going to do my best to pronounce the names, but we are super happy to have in the sauna, Dan Grøndahl and Emma Dahl-Jeppesen. Hello, Dan and Emma. Hello, Darren and Marc. [Emma] (1:13 - 1:14) Thanks for having us. [Marc] (1:14 - 1:21) Thank you. All right, super cool to have you back. And Darren, nice to see you again. Good afternoon. [Darren] (1:22 - 1:24) Good afternoon. Good afternoon, Marc. [Marc] (1:24 - 1:49) All right, we are talking about platform engineering all the time. We're talking about it with customers, we're reading about it, listening to it in talks and podcasts all over the place. And your guys' topic for the conference is 10 things we hate about platform engineering. I absolutely love it. Let's have a little bit of background. So what are you guys up to in the platform engineering space right now? [Emma] (1:49 - 2:19) Yeah, so right now I'm working as a platform advocate at VELUX. And one of the things I'm really deep diving into is we're doing this developer experience survey. So that's something I'm driving because I, as always, insights are hard. You want to create insights and then prioritize your efforts from them. And that's really what I'm working on, being more data driven in order to, yeah, make the lives better for the engineers. So that's my thing right now. [Dan] (2:19 - 3:02) And I guess for my part, well, I still feel kind of new because I joined VELUX same date as Emma, the 1st of May. So three or four months ago, actually more. But, well, I am the platform owner of our Kubernetes platform in a team that we call CCoE, Cloud Center of Excellence. So in that role, I have the responsibility to figure out where we should go in relation to the greater organization to help our colleagues get the best out of the platform that we are providing to them. And so it's really dear to my heart to do these things here and get the ownership for that, which is awesome. [Marc] (3:02 - 3:18) Cool. So we used to, you know, we still talk about what is DevOps in a certain way. And one of the things I'm saying now is I'm steering all the DevOps conversations towards platform engineering because now we can finally have DevOps as a team, right? So a platform engineering team. What is platform engineering to you guys? [Emma] (3:18 - 4:28) You know, I think for me, I like this platform engineering white paper that's from the CNCF. I look very much into that all the time when I'm confused about what platform engineering is, because I think just like with DevOps, like you said, Marc, people are just getting rebranded to become platform engineers and teams are getting rebranded from a DevOps team to a platform engineering team. But it's more of a concept than it is a team, right? And so for me, platform engineering is having a set of capabilities and thinking about how you can layer with some abstractions that makes it easier for developers. So they don't really necessarily have to know a lot about stuff like Kubernetes, right? Which we know is inherently complex. So for me, that's platform engineering. And I think it's the same with the term DevEx. It's so overused and we hear about DevEx teams, right? But for me, DevEx is something that's driven by stuff like platform engineering and organizational issues and stuff like that. So I think it's, yeah, it's not a team. It's more of a concept and a mindset and a way of working. That's what I'm saying. [Dan] (4:28 - 5:54) I'm not ready to bury DevOps yet because of platform engineering. Sometimes we hear that platform engineering kills DevOps or whatever we say. But if you take like some of the original thoughts in CALMS, you know, culture, automation, lean measurement and sharing, I feel that these mindsets or principles are still valid in a platform engineering team or in an organization as a whole. There's just more to it now. There's also, Emma mentioned the practices, mindset of having product thinking, these kinds of things that comes on top of everything else. So that's kind of what we're at right now. And it's not easy, I would say. It's actually not something that we're used to. If we take what Emma said, that the rebranding of sysadmins, SREs, DevOps, and then we should begin to have a focus on how do we actually provide this value to the rest of the organization in a way that scales, in a way that makes them use our platform, because it hopefully is optional, because we just deliver the best thing out there for them to consume. It takes the discussion in another direction, I would say. It needs to be more pull and not so much push. And I think sometimes that can be hard when organizations try to adopt platforms. There's a gap between what is also referenced in the CNCF maturity paper for platforms, the extrinsic push to the intrinsic pull, that it's hard to jump over, I would say. [Darren] (5:54 - 6:16) There's one thing in there that was quite interesting to me, and that it seems to come up a lot. It seems to be this new concept of the seamless experience, the idea of developers being able to develop. And we had the developer joy talk at the previous DevOps conference, and maybe platform engineering was the basis to push that. [Emma] (6:16 - 6:22) Yeah, that's a good question. Darren, sorry. Dan, maybe you can take that one. [Dan] (6:22 - 6:26) Can you repeat it, Darren? What was the gist of it again? [Darren] (6:26 - 6:35) Basically, to think whether or not the seamless experience is the key to platform engineering instead of DevOps. [Dan] (6:35 - 7:56) Yes, and that is actually super hard to do. Sometimes we talk a lot about abstractions, right? That we want to create the right levels of abstractions such that the users or the developers using our platform does not necessarily need to know all the nitty-gritty details of whatever infrastructure that we're trying to hide from them. And it's actually really, really hard often because it requires us to begin to maybe talk in another language, maybe not to expose certain aspects of, say, Kubernetes and cloud infrastructures and so on. And it's both in the way that we communicate, but it's also in the solutions that we provide such that they can be considered seamless. So I think that is often some of the pitfalls that I think both Emma and I see is that the way to actually go out and discover what would be the level of abstraction that feels relevant to the users but still does not become magical in a way such that we remove like the... that we're actually having things on their APIs and the way that they interact with the platform. Because sometimes when things go wrong, abstractions are leaky. We cannot hide it all the way, I think. But that's maybe a greater discussion. [Marc] (7:56 - 9:04) The thing I like about seamless in this kind of context is that one of the things that we're trying to do is to make it intuitive for the user, in this case, the developer. The developer's experience is that most things just work. And when something doesn't work, it's intuitive enough that I can send a pull request in order to show as code what it is that I'm trying to achieve to platform team, for example, to allow them to then pull that request in and say, okay, this makes sense in the context of a platform that we're trying to do or potentially reject it and say, okay, there's a different way of doing this or this is not something that belongs in the platform. You need to build something else. So that to me is kind of, when it works, it's a very seamless experience. And then the magic, right, is when it doesn't work, how do you make it intuitive so that the person has the discoverability, can have some communication method with the person or the entity that is blocking their work at the time? I think that's what seamless kind of feels really good to me in this context. [Emma] (9:05 - 10:04) Yeah, and I think it requires also for platform team a lot of product thinking, like having those values and that mindset, right? And UX skills, to be honest, like thinking about user experience. And I see that there's a bigger rise in like product and UX roles within platform engineering, within the DevEx space. And I think that's great because it just, once you're laying those abstractions on top and you're trying to make it intuitive and it just requires the, yeah, the skill of doing discovery work, right? And the mindset of that being important. And I think in a lot of cases, a lot of engineers don't wanna do that. They wanna code, they wanna do their job, right? What they're paid for, and they don't need to have all the skills. But then the fact that companies are hiring more people that are into product UX and these things, I think is really key to giving that seamless experience that you're talking about, Marc. [Darren] (10:05 - 10:45) I do think the productization there is probably an extremely valuable point given that in the past, DevOps had this extremely bad habit of kind of blurring lines. And this product-based thinking is very much putting your foot down and saying, this is our product that we're pushing out, even if it's towards internal users. If this is a product, here is where we draw the line. And here is where we push back compared to the old kind of hazy DevOps area of everyone gets kind of dragged in and some people want to, some people don't. It's like, it seems the natural progression from that. [Emma] (10:46 - 11:24) Yeah, definitely. It's some of the things that we're actually covering in our talk at the DevOps conference, because it's like, you need to be very clear on what's your team's and product's vision. And what are the strategies to achieve that? Where are your responsibilities? And also, where do we draw the line? Because I was in a DevOps team at one point that did all the things to unblock all the teams. But in the end, it kind of put a too big cognitive load and at one point they all quit, right? So I think the blurry lines you're referring to, Darren, is exactly the point of being more of a product team, actually knowing what your product is and what is not. [Dan] (11:24 - 12:07) Some of the questions that we actually discussed is, is it best to say yes to every request or is it better to say no to every request? It's not an either or, I guess, but what determines how you can say yes or no to stuff like, is this something that we want to build? Is this something that we want to offer as a service? It requires that we are really clear about our domain and as Emma mentioned, that we have a vision for it. Otherwise it becomes hard and I think that's some of the pitfalls that we still see that we can call ourselves a product team, but the ability to say no is a great ability to have because that means that we know the domain that we're in, where the limits goes, where the premises are, right? [Marc] (12:08 - 12:23) I'm going to pick on you now, Dan, because we got to get to the hate. So I hate it when the conversation has two options, build or offer as a service. What about buy, Dan? What about buy? Yes, that's... I hate not invented here. <laughs> [Dan] (12:23 - 13:39) Yeah, and that's a really, you touched upon a really, really good subject there because I think actually we, in my experience, we're often biased into either... Let me add one, build, customize or buy, right? Or consumer, whatever we call this scale here. So, and I see oftentimes that if we are good at building stuff, this is our go-to mental model that of all the time we think, yeah, let's just build that and not necessarily go into the landscape and see what are others doing here? Why are we building a complete observability platform if we can consume one out there just as an example, right? And the opposite way is also a thing. If we're every time just figure that, let's see where we can build this, then we don't necessarily have anything that is connected to where we see that we can differentiate according to the business that we have on top of us. So it's, yeah, thoughts like this comes from Wardley and stuff like that, right? Where are we on that scale? Is it Genesis? Is it the product or is it something that we just can consume as a commodity? Yeah. [Marc] (13:39 - 14:01) I've noticed that I've started, like, I think on paper a lot. I can't do it in business enough because I need the notes immediately. But when I'm having a conversation with somebody, I think on paper in between us and I've started organizing my notes in Wardley map fashion. So I can't wait to hear him speak at the conference. But how about you, Emma? Hating on platform engineering? [Emma] (14:01 - 15:07) I think once in a while, once I see one of these pitfalls come into play, because I mean, of course they have been in our team and former teams I've been with and I hear them all the time. I don't, I love platform engineering. Let me just say that very clearly. I love it. That's why I'm working in it. But I just think that, again, there happens a lot of, there's a lot of rebranding, right? Of DevOps teams, now platform engineering teams. And I think it's sometimes just some manager who heard some buzzword and they thought, that's cool and that sounds really good and we'll want that, right? But then they're not really working as a platform engineering team. They're not working with platform as a product. They're just doing the same thing, but in new clothing. And I think that's really why we thought of this very click-baity title. We know it's very click-baity, but hopefully it'll be entertaining to hear the things that we've discovered and that these pitfalls we see all the time and yeah, why we do them and what we should do instead. [Dan] (15:07 - 16:37) Maybe just to elaborate a bit on the product thinking. I think some, oftentimes it's, we are engineers, many of us, we are used to thinking solutions. That means if we have this model of the, what is called the double diamond, where we have like the discovery phase, defining the problem, we have a problem. We need to discover and open up what could be the potential way to look at this problem and beginning to define it such that we actually have a problem statement. Then we can begin to open up what are the possible solutions to this that best fits the problem. And then we can again converge into the actual solution. And I think from my point of view, we often forget the first diamond. We often start just with, this could be fun. This is a solution that could be fun to do because my gut feeling tells me that I heard that Jane and Bob in Team Ninja is having a small problem with this thing here. So let's just go do that. And I think we're better at answers than questions in general, that if we begin to talk more in questions, how can we provide a better abstraction level for these things that is annoying to developers, but not necessarily go into solution mode at first. Then I think that is kind of the key to some of the considerations when we are talking about product. [Marc] (16:37 - 16:38) Did you have something there? [Dan] (16:38 - 16:40) Not on that. [Darren] (16:40 - 17:05) I was actually thinking a moment ago about the whole rebranding from platform engineering to DevOps is kind of what happened to spawn DevOps in the first place as a lot of sysadmins kind of got pushed around and all of a sudden the manager heard the word DevOps and thought it was cool. And now you have a couple of rebranded sysadmins. So it's like the wheel turns and nothing is ever new. And now platform engineering is doing the same. [Emma] (17:05 - 17:10) It's like a fashion, right? It circles back and it's just in another form. [Dan] (17:11 - 17:38) And can I be a little bit bold here? Because I said that we were engineers, but if we actually look at what we do, it's YAML and scripting. There's not much software in it. Where is the unit test? Where do we provide these principles that we say to others, do TDD, do these kinds of things. Why do we then call ourselves engineers when everything we do is in batch scripts and in YAML files? I don't know. [Marc] (17:39 - 19:29) It kind of sounds like a daily spread. It's like, what did you do yesterday? Writing YAML. What do you do today? Writing YAML. Is anything blocking you? Writing YAML. That's a good one. I like to think that we're getting better at this as we go along. And it's like, we blew agile because we forgot that people in interactions over processing tools. So then we said, all right, come on, let's just do processing tools then. So then we automated everything we could with DevOps and the management area of the company is still not agile. Okay, fine. So let's forget about that. And then let's say, okay, let's turn this. And this is one of the funny things with this internal IT to product thinking, or is the platform an internal product or not? Most of the products that we deal with on a day basis, yeah, they do go through focus groups and stuff like that. But we all have a love-hate relationship with them. So with platform engineering, I think the greatest thing that we have improved is we've turned to this kind of pull mentality and there's a real customer focus on it. And the thing that you guys were playing with earlier that really annoys me, and I think this is something that platform engineering can be helpful with is when I get to code, I have one technical project. And when I get to code, whatever's in front of me is going to draw me in. And if it happens to be that I can't get this action to run the way I want it to, then I'm gonna spend the rest of the day on that, right? So then if I don't have any support on that, then that's where my time is gonna go. And instead of working on the actual customer problem, I'm trying to figure out why it's just one little part of the build, not the way I want it to be. It works, but it's not what I want, right? So then this platform engineering help developers then to actually focus on the domain-specific value creation that we talk about all the time. We don't use the video in the podcast, but everybody's nodding at me at the moment. But... [Dan] (19:29 - 21:12) Yeah, it's right that... Well, I guess a couple of thoughts on that one. That we have domain teams or business-facing teams or application teams or whatever we call the teams that is consuming or using our platform has like a pure business-oriented focus, can have the pit, the downfall that sometimes it would be nice that maybe a couple of developers within these teams digs a little bit deeper into the stack. Not necessarily the whole way, but it could be awesome that we sometime have someone who was able to express their needs in a way that was more platform-specific. And while I'm saying it, I can also hear that then we are in our domain again. So it's a trade-off, right? So to actually figure out where's this, because I'm not sure that we can keep this abstraction level all the time, all the way up towards the teams using our platform. It's a thing to strive for, but people are different and people have been exposed to various parts of the platform. And if they begin to say namespace, meaning a Kubernetes namespace, and we say, don't use that word, that's our word, say environment or world or anything else that we have put up an alias for, then that's kind of the hard part of it because they have been exposed to a lot of the things that a platform exposes. And it's hard to withdraw that and say, let's use another vocabulary if we have exposed them to it from the start, I guess. [Emma] (21:13 - 22:05) Yeah, I mean, I think there will always be those engineers that are just way more interested in going deep down into the stack. And I think for me, that's okay. Like they can know more if they want to put that cognitive load on themselves, then fine. If you're interested and you want to know, that's no biggie. But yeah, I think it's really important that we try to use language that doesn't, well, leak abstractions, right? And I think having those few engineers that are going to be using our domain language and kind of being more interested in what's underneath these abstractions or interfaces that we've made, I think that's fine as long as the rest of them who are not interested are allowed to, will not have that information leaked and not have that cognitive load put onto them. I think that's the key part. [Dan] (22:06 - 22:50) And especially maybe if they provide solutions that can we have that? And the solution is something that is within our domain, that sometimes you mentioned that, Emma, that instead of saying yes, then we say why? A lot of times, why, why, why, why, why? Such that we get to what is the actual need? Because there might be other opportunities out there than just whatever solution that they had in mind. And we may be able to provide them to more teams than just maybe a team that has one specific pain. But that's also a discipline and a practice that does not necessarily come just because we have created a platform team. [Emma] (22:50 - 23:16) And yeah, I like to ask the question why until the user is almost annoyed because I've asked it one too many. But then I know I'm at the core of the need. And then the solution to the problem that they're having will be in the hands of the platform engineers because they know best how to do that with the technologies and skills they have, right? So I think a great tip for everyone listening, keep asking why until they're annoyed at you. [Darren] (23:16 - 24:00) Annoyed driven development. And that's a cool question. I have one characterization. I wonder if you think it's fair. It feels like we spent quite a lot of time, especially in like very technical areas, sort of breaking down these silos so that everything was visible to everyone and everything was understood by everyone. And then we kind of realized we'd made a mistake and platform engineering is kind of a step back in that way to maybe not rebuilding the silo, but maybe kind of putting together this semi-transparent silo under the guise of abstraction to kind of separate developers from processors without fully closing them off. [Emma] (24:01 - 24:27) I think you're exactly right. I mean, we did leak way too much information across organizations and that's probably what you're trying to remove a little bit, right? Not to say that breaking down silos isn't important. It's still very, very hugely important, but perhaps trying to communicate in a way that doesn't leak the domain language that's not needed to understand something, right? [Dan] (24:28 - 25:32) Darren, I had this epiphany also when I started. I was like, well, didn't we just agreed upon that we should break these silos and now we have them again? It's not any different, but I think it's like with code, right? That you have a public interface, that you have something that is your API and that also goes for teams, I would say. So a team charter, for example, is a way for teams to communicate how are we working with roadmaps? How would we like to communicate and collaborate? What is the interface to us that you can expect? And then there's the private parts that is not necessarily something that they should know or care about. And I guess this is the trade-off between having everything public or having everything private. And to strike this balance that this is something that we can say through a team charter, for example, that this is the team API. This is the interfaces that we provide, whether it's from support, documentation, actual APIs, web polls, the team, et cetera. [Marc] (25:33 - 26:26) So Kelsey Hightower needs a cameo in every one of our podcasts, right? And he said that a silo is anything that doesn't have an API, if I remember right. So I like this team charter as your API. And I like this really much. Leak is a really strong word. And it took me a while to wrap my head around. Increasing the cognitive load by presenting information that's not really necessary to the recipient is leaking abstraction. And I think that's a really interesting concept to kind of consider here. And I guess there's a huge difference between I go select the template and then now I have an application structure that builds and I can create my domain-specific microservice or something like that, compared to having to know how Kubernetes works and what's the ingress for my pods and all of this kind of stuff. So I guess that's what we're really trying to solve. [Dan] (26:26 - 27:08) Yeah. And I guess I heard this quote, I can't recall who it was from that all non-trivial abstractions to some degree are leaky. And I guess it goes all the way from back when we had compilers and stuff, right? We tried to abstract away machine code the way that we, but still maybe sometimes we need to fine tune stuff because we need to optimize performance and so on. So I just like this because it reminds me that the thing that we should strive for is to make abstractions as great as possible, but still accept that they eventually, it's hard to make them non-leaky, maybe impossible for non-trivial things, right? [Marc] (27:09 - 27:51) I think it's been really fantastic to have you guys on. I can't wait to hear your talk. To be honest, I love click-baity stuff. It's kind of like a guilty pleasure, especially when it's in my domain. I don't want to hear another DevOps is dead for the rest of my life. But I think things that you're going to be talking about are actually relevant and valid considerations for how people improve platform engineering for their teams. And Emma and Dan, really looking forward to seeing you at the DevOps Conference Scandinavia on November the 5th in Copenhagen and November the 7th in Stockholm. Thank you very much for coming into the sound. [Emma] (27:52 - 27:56) Thanks for having us. And yeah, excited to see you at the conference. [Dan] (27:57 - 27:59) Thank you so much for having us. [Marc] (28:03 - 28:08) We'll now give our guests an opportunity to introduce himself and tell you a little bit about who we are. [Emma] (28:08 - 28:23) Hi, my name is Emma and I'm the platform advocate at VELUX. And I work mainly with driving discovery efforts and insights for our platform team and for the organization to help bettering the developers lives at VELUX. [Dan] (28:24 - 28:43) Hi, my name is Dan. I'm the platform owner of the CCOE platform in VELUX. I'm working to provide value to the users that is using our platform to get the best experience out of it, to solve actual business problems through the platform that we own. And that's kind of my job. [Marc] (28:43 - 28:52) Hi, I'm Marc Dillon. I am a principal consultant at Eficode specializing in platform engineering and enterprise transformations. [Darren] (28:53 - 28:59) Hey, I'm Darren Richardson, security architect at Eficode. And I work to ensure the security of our managed services offerings. [Marc] (28:59 - 29:06)