When we say we have a DevOps team, usually we say it in terms of you build it, you run it, but what are actually the parts that you're running? To discuss platform engineering and teams, we've invited Johan Abildskov, Software Engineer at Uber, and Eficode Alumni. Johan has been a rockstar in the DevOps scene and the book Practical Git: Confident Git Through Practice, has been influential to many in the DevOps community, including our hosts, Marc and Andy.

Johan (00:12):

When we say we have a DevOps team, usually we say it in terms of you build it, you run it, but what are actually the parts that you're running?

Marc (00:25):

Andy, I have to tell you, we've talked about being like, intimidated or something, but this time it's like, I had like a fan moment to have Johan on our podcast this time. I'm not sure how familiar you are with him, but he's been a rockstar in the DevOps scene and the book Practical Git was something that-- Git Through Practice, was something that was really influential to me.

Andy (00:51):

Yeah, it was definitely cool to talk to him. And it's one of those moments where you know somebody through their work and you just think, oh, they're so smart, they definitely know so much. And then you meet them and, oh, they're a person, but this is so cool.

Marc (01:08):

Yeah, I know. And it was funny that there's this kind of thing that we do where we give each other little elbows or something and it was nice to build a rapport so fast with these guys. And I always find it interesting when we come from different places, but when we come together, we have a lot of common ground and alignment.

Andy (01:32):

Exactly. We've worked on so many different projects and come from different kind of backgrounds, but we're doing so similar. It's easy to mesh together into a group and just align.

Marc (01:46):

Okay. Well, with no further ado, welcome to the DevOps Sauna podcast featuring Johan Abildskov and Dan Glavind platform teams.

[music]

Marc (02:04):

Hello, and welcome to DevOps Sauna. My name is Marc, I'll be your host today. Today, our topic is platform teams and we have a couple of very special guests. I didn't practice pronouncing your name, Johan Abildskov, but your name is something that's been important to me for quite some time. You've been a rockstar in the DevOps arena. And we crossed in the halls. I don't think you're ever an ex of some place that you used to work. You're an alumni, so you are an Eficode alumni as well. And it's super nice to have you on the program today.

Johan (02:43):

I'm so happy to be here and thank you very much for your best effort pronunciation of my name. I think you get the passed grade.

Marc (02:53):

All right. Excellent. Also, in order to back me up against this heavy weight in my corner, I've got as usual Andy Allred.

Andy (03:01):

Hello.

Marc (03:05):

And one of Johan's partners in crime<laughs>, Dan Glavind.

Dan (03:12):

Hello. Yes.

Marc (03:15):

Did I pass your pronunciation as well?

Dan (03:16):

That was excellent. Yes.

Marc (03:21):

Cool. So the topic is platform teams, and in my transformation work and in research and analysis, there's a pretty big trend now towards talking about this thing called platform teams. And my first kind of question is, how did we get here? What has led towards this platform team's trend that's going on right now?

Johan (03:47):

So I can add an opinion to that, which I guess will be my topic for today, adding opinions. In my belief, we have seen tons of focus on how do we create these autonomous self-contained DevOps two-pizza teams in like the DevOps sphere as perhaps some reaction to trying to figure out, we can't really figure out, how to scale Agile in a same way. So we try to decompose. How do we decompose? We do that by having these small autonomous teams. Turns out that's extremely difficult, but then we have work from people like Manuel Pais and Matthew Skelton Team Topologies. We want to have stream aligned teams and tons of different team supporting them. And then we start to saying, well, we also have path solutions, cloud vendors. We want to buy everything in the cloud, but we can figure out how to buy stuff from others, so we'll build it ourselves. So then we have the platform team. It is almost like when we want to do DevOps, we do CALMS, so we start with automation because that is easy and we can't address all the culture stuff and that part, so now we can't figure out how to buy stuff in the cloud. So now we'll just do platform engineering instead.

Marc (05:14):

It's kind of a convenient thing. One of the things that people always say, well, what is DevOps? And everybody has their own opinion. And it's like it's not a guy on a team, it's not a team, it is a way of working that enables teams to constantly deliver to production and realise value. And then we kind of came along with this platform teams and said, oh, hey, wait a minute, maybe there is a team, but we can't call it DevOps because we said that's not allowed, so now we'll call it platform teams.

Dan (05:45):

Yeah. So I would say that some of the promises with what I at least thought of about DevOps is that it's actually like trying to destroy these silos between Dev and Ops, and create the ownership of whatever software to the team that is now this fully capable cross-functional team. So they also had the motto, you build it, you run it. But some of the thoughts that went into my head with the platform team initially was: are we then recreating those silos again, because now we have a non-spoken or non-agreed ways of working together with kind of an infrastructure team or a platform team and these more business-faced teams. That's some of the fears that I initially thought about.

Johan (06:38):

So I think you hit a very excellent point. DevOps is about decomposing silos. At least we can agree on that part. It might be tons of different stuff, but at least it's decomposing silos. And is that consistent with wanting to buy cloud services and wanting to build platforms? But I think one of the interesting parts is also that we have a very clear understanding on what is Dev, but if you ask people about what is Ops, then you'll likely get answers from managing data centers, plugging in power supplies, maintaining recs until the chart into saying, well, I have this Docker container or WAR file that some developers built that I have to carry into production and make sure it starts serving production traffic. So I think from my perspective, it's about when we say we have a DevOps team, usually we say it in terms of you build it, you run it, but what are actually the parts that you're running?

(07:36):

You're doing the Ops work for your application. What is that? So then you might say, what are the application-specific metrics that I'm wondering? We have the five golden signals, like error rates and request ratio, stuff like that saying, well, those are probably very Ops-like for me as an application developer, but whether or not the physical machine I'm on is soon to be decommissioned, that might not be something that allows me to run it or build it because it works on a different abstraction level than I want to do my Ops work on. And then we end up in this usually pure discussion of like, you're not Ops if you can't run Linux command on the host, or like, you're not Ops if you haven't done this or that kind of gatekeeping. I don't think that's really useful. It's about saying, what are the different capabilities we want to be able to do? And I want to be on call for my application, I don't want to be on call for my application’s dependencies.

Dan (08:51):

Great point. And I think it comes down to that maybe sometimes, a pitfall might be that you don't have these kind of responsibilities outspoken and explicit. So for example, cloud providers like AWS always have this kind of responsibility model, this is what we are responsible for, this is the shared responsibility, and this is actually what you're responsible for. It might be some of the same things that an organization that looks into doing platform engineering should really consider how to be really explicit about.

Marc (09:31):

You said the word silo, and I learned what a silo was this year. It's something without an API. We got that from Kelsey Hightower. Ticketing system is not an API, is it?

Andy (09:50):

It's a very, very bad one.

Marc (09:54):

It's a very, very bad one.

Johan (09:55):

What is the SLA on the response time for your API?

Marc (09:58):

Exactly. So what's the real problem that we're trying to solve with platform teams? We talked about decomposing silos and there's some scalability things, is that it? is it enabling more throughput, is it quality, is it security? What is the problem we solve with platform teams?

Johan (10:24):

So the problem that I'm trying to solve with platform teams or I want to solve is that we want to have these autonomous teams and we also want teams that are reasonably small for all sorts of reasons. That will be eight podcasts just to discuss. But let's assume that the general sentiment in the industry is that we want small teams that are able to do things independently. So what are then the capabilities that we need to have on our team in order to do something independently? Do we have to know Kubernetes? Do we have to know AWS? Do we have to know how to write Java code? Do we have to know how Gradle works? Do we have to know all these sorts of different things? Do we have to know scrum or SAFe or extreme programming? Do we have to know how our IDE works? Do we have to know how Jenkins works? Like if we are reasonably few people on our team, we are either going to suck at some of those capabilities, or we have to figure out how we can do without.

(11:40):

And if we accept some of those things as stuff that we must have in order to be effective, then I think the platform team is about enabling the teams to like 'proxy' those capabilities or get those capabilities at an abstraction level where it fits with the context of the work that they're doing. And I believe that will increase the ability for the team to do independent work, autonomous work, take ownership and responsibility of the services that they build and run, and that in the end will lead to better business outcomes.

Dan (12:28):

Yeah. So that is what the literature also like, kind of tries to explain like, is taking away extraneous cognitive load from teams, right? The things that is just either out of scope or not necessary, but isn't like something that we really need to know about from a developers point of view. If I could just magically have that taken away from me so that I can focus on what is really the thing that I'm here to do, which is probably do business facing development, then that would be ideal. That would be the thing to strive for, at least for me.

Johan (13:12):

So we have actually talked about this, I think then in some of our engagement with the client teams, software teams, where like an interesting question to ask a team is: what is something that other people are doing for you that you would like to have them stop doing for you? And what is something that you're doing that you would like someone else to do for you?

Dan (13:30):

Yes.

Johan (13:31):

And if the things that you're trying to solve with your both management structure and hierarchy and your platforms are not aligned with those, there's going to be friction somewhere.

Dan (13:45):

Exactly. And actually, it just adds to complexity that typically a platform team needs to support a wide varirarity of business-facing, our stream-aligned teams or whatever we call these application teams. And in my experience, they are two really, really different because some of them are really-- we want to have hands-on all things. Don't come here and disrupt us in this because we know what we are doing. And then these teams that's just like, yes, please take over everything from us, we would like all the support that we can get. As a platform team, to be able to address both worlds, I think it's really, really a difficult task to solve.

Johan (14:35):

It's also one of the like failings of engineers, 'not embedded here' syndrome. I don't know what the number of employees is that you have to have in your technology organization for it to be guaranteed that you have built your own enterprise message bus. <laughs> But like, I don't think it needs to be very many because it's much easier to build stuff that fits with your own mental model, then taking something that has a different model and then adopt your mental model to fit that architecture at the time. But you lose out on all the best practices and all the knowledge that got into where we're built. Like, why don't you use RabbitMQ rather than building your own pops-up thing. And I think it's the same thing with platform teams. And that also fits with what you're saying then. There are those people that want to do everything because then they understand all the parts and they will have a really hard time letting go. Why are you saying I shouldn't know this technical interesting detail about how Git works when we have 1800 ongoing branches in our repository? Or rather than saying, well, maybe we need to just work in a different way that will reduce less friction so we don't have to know, or someone will have taken care of it for me. And I think like we need to build platforms in a way that support our mental models, but we also need to take charge of the modeling the architecture, the product vision of our platforms, such that we don't just do everything that our users ask for.

(16:15):

I don't remember who said it, there was some game developer who said like 'our players, they're really good at finding problems and horrible at finding solutions'. And I think that fits with how we think about building software. And I think, especially when we build software for experts, software engineering experts. Why are you not something, something? Why are you not doing this? I expect it's a question that you will get a lot building an internal platform. And it's a very unconstructive way of phrasing a question.

Dan (16:57):

Yes. And a reflection that I just got with what you said about, wants to know everything. It might also just be that someone says, well, I have this, 'you build it, you run it' responsibility, so I need to know these things when I'm woke up in the morning because something is down with my application. And again, that comes down to that you don't have perhaps an explicit way of addressing these kind of responsibilities. So that's just one kind of facade of this.

Johan (17:32):

Yeah. 10 years ago it was, I know that our Jenkins pipeline finished with the green build, but I will just poke around a bit to see if it actually works.

Dan (17:43):

Does it work now? It's green? Usually, it's red, right?

Johan (17:50):

Yeah. But like, we need to have trust and comfort in the toolings and the platforms that we were handed to us. And we have had that discussion, at least I know you and I, we have had that done previously. Like, why don't we trust our automation? And it's the same thing. If you don't trust the platform, if you don't feel that you can delegate basically, that's the part, you delegating to the platform.

Marc (18:16):

There's something that you touched on here that I want to interject, which is one of the things that you're talking about as like, you're on call and the thing fails, and you want to be the one that knows the reasons why, or the things that may be in there. But one of the things that we do when we start creating these levels of automation is we start converting stuff from people's head space into code space. And that's, I think one of the challenging things, people want to own the things in their headspace, they don't necessarily always want to let the information out, they don't trust that the automation that even they may be creating themselves can handle whatever corner cases are necessary. But the alternative is, it's the bus factor. The alternative is the only guy that knows that thing leaves and you haven't converted it into code, and now what are you going to do when you get the call at four in the morning?

Johan (19:11):

Yes. And I think that's completely right, and like a corollary to that is we have this joint belief that we are done when it's converted to code. Code should be self-documenting, code should be like, but have you tried reading code? <laughs> Why is there not a read me? Why is it another Wiki page? Oh, but we work so quickly, it will be outdated. Yes, but the alternative is I have nothing to work with other than trying, of course, to interpret what it means that you have decided to use all the newest features in this programming language.

Marc (19:57):

Is there a different side to that whereby like we talk a lot about: if you haven't changed this in a certain period of time, you start to lose the ability to change it. So by maintaining the system that you are working in is the only way to actually preserve that knowledge.

Johan (20:22):

Yes. So, then it becomes a question of where do you get return on investment on time spent? I don't believe that the constraint for a software engineering organization is going to be 'stuff to do'. Like we're not going to run out of ideas. We're not going to run out of Jira tickets in our backlog. And if that is true, then if parts of our software starts becoming stale, then perhaps it's more about encapsulating that piece of code than doing work there for the sake of doing work there. Because we may be in a situation where we will, at some point in the future, have to change something, bump a library, whatever, but it becomes a trade-off between, am I willing to pay a little bit for that every week? Or am I okay paying for that complexity when it comes? And I don't think there's the right answer. I think it depends on what part of the code base, how likely is it to change, all sorts of different tradeoffs that we have to do in the concrete case. But I think that the notion that all codes needs to be kept alive is wrong.

Dan (22:02):

That's it.

Johan (22:03):

I like the distinction between like source code are now artifacts. And as long as we can deploy the artifact, then that might be good enough.

Dan (22:14):

There is an assumption there that we actually know what kind of our code base has changes in it. And I think it's not necessarily that teams actually have this as a metric, what kind of code base is actually active and live and what part of it is kind of dormant or stale. I mean, that would be probably-- and the thing is that if that's true, then we need to change stuff in the stale part of the code base. And it's because we probably have made something wrong or that we need to change these kind of things in there. So the situation we're in is not a pleasant one, perhaps, a feature that we need to do, rather, it might be a bug that we need to quickly address in this part of the code base, I don't know. But there is something around still code bases and where it is that I think for the majority part of teams is kind of invisible for them or in their heads at least.

Marc (23:26):

Let's try something else. We talked a little bit about the ownership of the platform and I think like the investment in the platform. So are there some things here, like, who should be the owner of it, should they have a charter and clear responsibilities and who's gonna pay for this, those kinds of things.

Dan (24:00):

Go ahead, Johan,

Johan (24:02):

I'm just trying to be quiet, so you will start then because I've started the other times. <laughs>

Dan (24:07):

Well, I think typically what I've experienced is that some organization actually doesn't have a product concept of a platform team, right? So, it's just engineers, let's do something for our other engineers rather than actually consider this being a product invested in and consider those that uses the product as customers to it. I mean, we have all the benefits of having customers close to us. And the business-facing team might typically be really much further away from their customers than a platform team does. So, I think a common pitfall might be that we don't necessarily consider it as a product because the other parts of our organization, where we do consider teams and the services as products, I bet a lot of them have actually some kind of feedback loop, some interest in how the users uses it, some metrics that tell something about their product. And I think these kind of things might be something that we just doesn't necessarily do when it's internal or doesn't have the same concepts about. I think that's a shame because we could use the same kind of principles in the platform team that we do with our business-facing teams. So that would probably be my take, at least on that we should consider it being a product and invested it as such.

Johan (25:40):

I think Marc, first off, I want to applaud you. You really tried your best at phrasing that question in an open, unbiased way, and you failed horribly. <laughs> But I noticed that you really tried. I wonder if it's something you should invest in or they should have a charter. <laughs> So I think I agree, yes, we should! But in terms of something that is a bit more value-adding: I'd say like the one thing that might be very nice is that the platform should have a goal or direction of its own. It should have opinions, it should have models, it should have like shape and things like that. And who owns that shape? What is it that the platform is trying to do? Because otherwise we get this like scattershot. So this team wants the platform to do this in this way, and this will have it do it this in this way.

(26:40):

And we'll end up with like tons of different solutions and unmaintainable mess. And like, what is it that we're actually trying to do with the platform? We're trying to accelerate something, but we can't accelerate probably if we are accelerating in 900 different directions that change all the time like we want to have some guard rails, but we can't because all teams anyway want to have let's just say post level access for some arbitrary reason that they can't be convinced out of. Just so many different things where like, you want to have a product, a charter, or something, but like, basically the idea is that someone needs to have a model and a goal and a direction.

Marc (27:31):

I love it. And yeah, maybe the next time I'll ask a question, like what kind of charter? What kind of organization should be owning and paying? But thank you for the feedback. I like it.

[music]

Marc (27:50):

Hi, it's Marc again, hope you've been enjoying the episode thus far. I'd like to refer you to some interesting work that Eficode has done in the area of platform teams as part of our DevOps for executives materials. You can find the link in the show notes. Now, back to the episode.

[music]

Marc (28:17):

But one thing that I wanna put in here is that, we all want to feel good about our work, and we all talk about feedback within DevOps, within product organizations, within life. Feedback helps support the way that we grow. And regardless of your technology preferences as a developer, if you're working in a platform team, you can have a very close connection with your customers and get a really nice tight feedback cycle going. And I love, Johan, that you put on top that the platform needs, it's not just a collection of tickets from the feature teams or cross-functional teams or DevOps teams, the owner should have a vision for what they are really trying to bring to the house, that can be a bit beyond just what the current day-to-day operational needs are.

Johan (29:07):

So during my time as a consultant in Eficode and Praqma before that, one of the things that we often saw is that in reasonably-sized organizations there would be tons of different like ideas and platforms and tools just popping up all over because there is no-- when I talk to people about it I mentioned, you should have a sane default. Everyone uses Jenkins unless they have a good reason not to. Everyone is on Git, unless they have a good reason not to. And many organizations, they forget to have a sane default and they forget to maybe also make it okay to not follow the sane default. So like there are two border lines, like you cannot do anything except for this very horrible friction-filled thing. So we should have same defaults and a way to work around the same defaults when we have more value from working around it than from staying on the same defaults. And that's also kind of what we want to do with the platform, let's build the sane default and take into account that not everyone will be happy with the sane default. And we don't necessarily have to solve 100% of the problem, but we want to create the path of least resistance. We want to make it awesome to be on our platform because then people will want to be on our platform. Then the value they will get from going on the platform will be bigger than the increased friction from like not getting their completely own custom way.

Dan (30:42):

Yes, to elaborate a bit on this is that typically what we have seen is that you have all these kind of custom build stuff, and that ties again back to that, we actually-- I mean, a product, could that be custom built? I mean, in other teams you have these tools and scripts and it just fit their purpose. It doesn't necessarily scale to another team or other teams. But I mean, a platform should at least address some of the common challenges for multitude of teams in an organization.

Johan (31:22):

And here it becomes obvious that about half an hour in, and we haven't really discussed what are platform capabilities. Like what is a platform supposed to do? And I guess that's also kind of because we both have some assumptions around it and because maybe it's basically trying to solve problems for others to enable them. And then the platform may be that we just give them a Jenkins and a Git server and a Jira, or the platform might be that we provide them a cloud environment or a cloud-like environment and all sorts of different frameworks that hook into that to abstract the entire like build-test-run cycle. But like a platform, many different things, how many companies doesn't have a Jenkins server under some developer's desk or cloud server that they just found, or we have so many-- like, I call it the Extensile-Ops. Like you start a small project to help your team, and then suddenly you're running business critical infrastructure on a set of machines on the developer line, and it's not quite monitored the way you would like to.

Marc (32:39):

And the cleaning person's banging it with a vacuum every night.

Johan (32:42):

Yes.

Marc (32:42):

Why do we get these errors at 10:30?

Johan (32:46):

I once slipped and fell into our server. That was not a great day. That is many, many years ago. <laughs>

Dan (32:55):

Chaos engineering, or what did you call it? A chaos monkey. <laughs>

Johan (33:00):

I just call it a really bad day. <laughs>

Marc (33:08):

Okay. And one of the things that we were talking about earlier is this can be different for each company. What does the platform need to provide? But there's also internal enablers, like in banking, you have some of the core banking systems. Is that part of the platform that the platform teams are responsible for or not? What else is the platform supposed to do? So it may or may not include CI/CDs and clouds.

Johan (33:35):

Like, the answer is obviously it depends and you know it.

<laughter>

Marc (33:39):

Yes. Examples are useful when you start with it depends.

Johan (33:44):

I think one thing that some platform teams at least forget is it might be very useful to provide some custom libraries and some frameworks. So let's say a platform could be built on AWS. Because just as many other tools direct access to AWS is likely the wrong abstraction layer for autonomous team, because then we also need to be experts in AWS, substitute any cloud provider or Kubernetes or whatever. But then one thing that would be very useful to have is a logging framework or observability stack. And I think many, this is, of course, just my own preconceived notion, I think many platform teams would forget to actually provide those frameworks and those libraries in the standard programming languages for the given organization team. Okay, so you just have to include this library and then you can put stuff in the docks, put stuff in something out the other. And I think it's things like that that are not obvious, but really helps with adoption and really helps with some concrete value add to the teams.

Dan (35:03):

Also, one of the things that I think about when talking about platform is also to kind of have a developer portal, and it's not necessarily because of the portal, but it is a way to abstract whatever we want a developer or team capable of doing. It might be that we want to have the possibility to quickly scaffold a new service setup with the right way that we want. And how it is done then is actually just up to the platform team. That means that it's possible to change things under the hood, and developers wouldn't necessarily be aware of these changes. So I see developer portal, such as for example, Backstage, that we've seen, Spotify's Backstage, as a means to kind of address this capability. And again, with the promise of actually trying to meet developers and developer teams at their abstraction level.

Johan (36:12):

Yes! And something that we forget sometimes is also that documentation is some form of automation. Just because I don't have a robot that does this tedious thing doesn't mean that I can't just script you then to do it through my excellent documentation. It will still be better than I have to do it myself.

Marc (36:33):

So your readme file is your pipeline.

Dan (36:35):

And your API.

Johan (36:36):

It's a start!

Marc (36:41):

So, we talked a little bit about pitfalls, but I believe there's some others that are interesting from your experience. Have you any anecdotes about common pitfalls?

Johan (36:55):

Yes. So, I guess this precludes to something we discussed in the pre-show banter, where I mentioned that we had bit on an engagement at a large organization where they have started looking to platform engineering and building a platform. One of the platform engineers mentioned to me that I think the developers there are like 40% as productive as before. And I completely was baffled by that, but also sidetracked into not diving into that statement because there's so many different things to unpack in that. Like, there's both the more overarching change management thing. Like, can we do anything new without getting a productivity hit? And do we account for that? Or do we just assume that as we start building platform, it will just be a continuous increase into the maximum productivity heaven? Or like, how do we work around that and how do we as a platform team take some accountability or responsibility for the productivity of our end users?

(38:06):

And I think at least Dan and I, we have discussed platform engineering and how do you start a platform engineering organization, things like that, one of the things we've talked about is this, how about doing a bit more manual work while your platform is maturing? How about doing more stuff for your users so they don't have to do it? Because as engineers, we tend to have this 'hate' towards doing manual, tedious stuff. And we will just do it the right way, rather than doing it for our stakeholders. We would rather be asked 18 times on Slack. Oh, well, now you get this error message, so you have to change this yaml file and this repository, and that will cause this to go away. Oh, now you have to do this rather than just saying: 'let me just do that for you'. Until we have an experience where the developer is better enabled into doing this, either through us getting our stuff together and actually writing the documentation that they need, or setting up the tooling, such that it will become transparent to the developer.

Marc (39:23):

I wanna put one in here. I'm even feeling a little emotional. I've had so many fights over this, which is like, you do it manually until it's automated. And that is usually you're doing something manually in service of something else. Like you will run a smoke test manually twice a day until you've got it automated. And that doesn't mean that in two weeks, you come back and tell me you've automated it and I haven't seen a smoke test. But I think this service part of the platform to customer, platform to develop a relationship is such a huge win. Like, what would you like us to take away from you? Okay, we'll start doing this for you now, and then behind the scenes, we automate it out of existence.

Johan (40:07):

Yes, and there's also like, well, we don't necessarily either design our mental models or able to communicate them very well. So it's like, oh, but in our system, we have modeled it this way, so that's why it doesn't work when you do it that way. And like, we are also always on the side of our system. Our system is our pet, so it can do no wrong. But like the horrible software engineers trying to do stuff that it's not even meant to do, oh, those dastardly <grunting>. Rather than saying, oh, so maybe we should reflect a bit on whether we have modeled it in a way that fits the way that our system is going to be used or whether we can improve the onboarding experience some ways such that our users will have a better intuition on what's going on and like understand the failure modes. I think often that's probably what happens when we forget to abstract stuff to the right level. Pre-show, I ranted about how we built stuff on top of Git and Kubernetes, and we forget to abstract the right stuff. And I think the consequence of that is the users are getting-- I don't have the right tools to understand the failure mode that they're getting into. Because the failure modes are happening so far from the work that they're doing.

Dan (41:36):

I think that the main challenge about-- because I totally buy into this thought about doing this manual service for them until the platform is capable of doing it. The trick here is how do we actually do this where if we consider the platform to be kind of horizontal and we need to have more teams on it, I could easily see an antipatterns as well that says we have too much manual work for our team right now, so we need a service desk for ticket management because we do it manually. And then we end up with the same Dev and Ops ways of working. If we don't have a capability of saying: now we do it manually, but we should also explicitly focus on having the same task automated, or at least the things that we do repeatedly have that automated, because otherwise, we'll end up with another anti-patterns which would be this, let's get a service desk, we can't manage it.

Johan (42:43):

I think this is actually, and, of course, you also want to sell some Jira licenses, so that's also a big plus, but nothing is so bad that it isn't good for something. But I think like, kind of my point is I think that this is part of the rollout strategy. And this is about growing the-- if we end up there where we have the problem that we can't keep up because we're adding too many teams, then we probably have something that's good that we want to invest more in, like growing the platform team to be able to do this. It might also be an ability to say, well, this half, we want to have this many users on it, and we want to have automation degree of 25%. Next half, we want to grow the amount of users by this much and we want to have an automation percent of the platform capabilities of this much. To both have like, everything is incentives. And like, these should be the metrics that you're then reporting to the higher parts of the organization. And this is also like something that you should be able to own as the platform, as the product platform, saying, these are our success metrics, this is what we want to be measured on.

Andy (44:02):

And of course, a lot of developers want this kind of 'Heroku-like' experience. That just I'll point my code there, and then it's running and I don't care about anything else. And then we have some developers who want to get down to every nitty-gritty detail. So we put a platform team in between and the platform team can decide how to abstract at what level and expose the services that the developers want. And do we get that right or not? And what's the cost of it and what's the productivity cost? But the one thing that's really coming out to me as I've been silently listening this whole time, is it keeps coming back to: what do the users need and what's the user experience? So, too often as techies, we sit back and say, well, I'm the platform expert, I'm gonna design a platform team that does the platform the right way. And then I treat it as my pet, as was mentioned before, and like, well, this is the right way to do it. And we too easily forget that what is really the problem that we're trying to solve for the developers who are delivering value to the customers of the company. And as long as we keep that in mind, all of these things we brought up and all these different points and perspectives and tangents are very, very important, but that's the key one.

Marc (45:25):

Excellent summary, Andy. Coming from the heart. Any further short statements before we get into the two questions, Johan?

Johan (45:38):

I think I'm very good at like any kind of-- this is the first principle that feeds all the discussions that we've had. So I think that's the foundation for what we are doing. I think it's so key that we consider: why are we doing, what problem are we solving, do we invest in it? Like, because otherwise, it will just be throwing away good money and good people.

Dan (46:06):

Take time to consider these things. Because I mean, at some point then you just have finally got the possibility to start this platform team and you feel that you owe the company just to show something. I'll at least do that if I got the possibility. But these considerations here, I would think really hard and thorough on these concepts here before jumping into whatever first MVP that you would probably do.

Marc (46:44):

Excellent. Thanks, guys. That was a wonderfully gregarious conversation. We have two follow-up questions that we've started asking all of our guests to kind of get a little bit of the human side. I'm gonna start with Dan. And two questions:  the first one is think back when you were a child, like the first time you can remember that you really wanted to be something when you grow up, what did you wanna be when you grow up if you think back?

Dan (47:12):

I think I wasn't that original. I wanted to be a fighter pilot. That might be a dream for a lot of boys at that age. But I didn't want to go to the military.

<laughter>

Marc (47:23):

Was Top Gun trending in these days?

Dan (47:25):

So there was a conflict there, so, yeah. Yeah, that probably would be that.

Marc (47:30):

That's cool. All right. Excellent. And then was there a point in your life where you either crystallized that you're on the right path or you realized that you need to change your path towards the one that you're on now?

Dan (47:47):

I think paths for me is these kind of where are you in five years is kind of ridiculous in my head because I don't know where I am in five years. So my paths are not that long, but when I see that I should probably jump them and when it comes clear to me. It has happened a lot of times throughout my career, at least. I think I'm really driven by what motivates me, and if I see things that I could work with being motivational, then I would kind of pursue them I would say, yeah.

Marc (48:27):

Do you pick one?

Dan (48:28):

Actually perhaps the jump to my current position, like being a DevOps consultant, I think was one of the really great ones because now I really do what I love to do and helping people and just really nerding about how do we actually work with software and teams and culture and organization, these kind of things. And it's a bit of an honor to do this on a daily basis for me, at least. So, yeah.

Marc (49:04):

Awesome. Thank you, Dan. Johan, what was the first thing you wanted to be when you grow up?

Johan (49:10):

So, the first thing that I remember wanting to be is, I will probably butcher the pronunciation of this, but paleontologist, like a dinosaur researcher.

Marc (49:23):

Excellent. Well done. You pass.

<laughter>

Marc (49:29):

Okay. Paleontologist that's cool. And then did your path crystallize, or did you change paths, and is there a brief anecdote from that moment?

Johan (49:38):

So I know that we have legacy code and things we consider dinosaurs in the tech industry, but I don't feel like I'm working very much in the sphere of dinosaurs.

<laughter>

Johan (49:52):

I kind of come from an untraditional background. I grew up in a commune and I went to a private school very much focused on music. But somehow, because I think we had a very old computer, like 286 or something like that, in the commune that was done to do the accounting. And at some point I could get people to start a game on the computer for me in DOS. And at some point I could do it myself and then I didn't have to ask anymore. And then I was the one who knew the most about the computer. And I think in sixth grade or seventh grade, my mom gave me, she did not know what she had bought me 486 DX with mathematical co-processor CPU for my computer in Christmas present.

(50:43):

So, I've been like since I was a small kid wanting to do something with computers and building stuff. But I also have this like weird very fluffy background. I spent a few years being a kindergarten teacher before starting at the university computer science, I've taught bunch of different courses. So I think it was reasonably natural. I just didn't know it was reasonably natural to become like a DevOps consultant thinking about how we do stuff. Every time you consult, you kind of teach and you're kind of doing social work. So, I was very crisp on wanting to do stuff with computers, but it was like completely non-obvious, and I think that's kind of been my career trajectory is like completely obvious and not making any sense at all.

<laughter>

Marc (51:35):

Love it. It's funny, one of the origins of this question was my one friend who, when asked this question, they say, I wanted to be a dinosaur. It's a small world. You wanted to be a paleontologist, and you said you started doing archeology at code, which is really cool.

<laughter>

Marc (51:55):

So, I'd really like to thank our guests Johan and Dan, and thank you, Andy, as well. This is Marc for the DevOps Sauna Podcast on Platform Teams. Thank you.

Andy (52:07):

Thanks all.

Johan (52:08):

Yes. Hello, I'm Johan Abildskov and I have been working previously at Eficode as a DevOps Consultant for five years, engaging with a ton of different organizations, thinking about how we build software. And I love to discuss how we build and think about making software. And currently, I am a Software Engineer in Aarhus at Uber.

Dan (52:30):

My name is Dan. I have a background as an Embedded Developer prior to working with what I do now. I'm a DevOps Consultant here at Eficode. I have been with Eficode the last four years helping companies excel in doing software.