The end of Software Engineering (as we know it) | Jan Bosch

Good morning, everyone! How are you doing so far? Having a good conference? Excellent. Aren't you happy to talk about the end of software engineering? You know, I couldn't resist and start this talk, actually, - with a picture that I asked ChatGPT to generate. "What does the end of software engineering look like?" According to ChatGPT, it looks like this. And then I politely added 'as we know it', - just for a bit of a feeling. And I work with tonnes and tonnes of companies. So, rather than being at one company, I work for a lot of different ones, - and I'll explain to you later on why that is. What I see is two trends. On the one hand, generative AI. Everyone is looking at how we make better use of AI. I listened to the talks this morning, and this comes up all the time. On the other hand, especially here within Europe, - and I know the UK is no longer part of the European Union, - but I'm pretty sure you have something similar going on, - there is all this regulation going on. We have GDPR, the Data Act, the AI Act, - the Product Liability Act, the Cyber Resilience Act. the Regulation on Cybersecurity and Software Updates, etc. One of the companies that I work with that builds trucks - that they ship around the world, - they are subject to somewhere between 250 and 300 regulations. So, this is a little bit of a challenge for many of these people. But what I wanted to talk to you about - is what is happening in software engineering. This is what you all recognise - as the old way of doing software engineering. It is, you have an intent, - you generate requirements based on that intent. Out of that, you design an architecture or some kind of software design, - and out of that comes code. Out of that code, you put in operation that becomes a system in operation. If you ask me where software engineering is going, - it is going to look like this in the future. We express an intent, - and then some set of agents will take that intent, - turn it into quantified outcomes, - generate a set of requirements out of those outcomes, - generate an architecture and a design out of that system, the code, - and that then leads to a system in operation. The whole notion that we are going to be sitting there coding - is, I think, going away very rapidly. And anyone who is using any of the modern tools - is probably realising that as we speak. And I have lots of examples. Actually, I was in an interview yesterday - with someone from a large German company, - something in common with my last name that I will not mention. But basically, they now have an enterprise architect - and a set of AI agents, - and the enterprise architect basically does not need any developers anymore - because the agents have taken it over for him. So, that's where we're going. How is this going to happen? Well, at the operational level, - we're going to have reinforcement learning, - I think it's one of the big techniques. And at the higher level, it's going to be vertical AI agents. Agents that have a specific job to do a specific task - and generate the things that you need to have. And I think it's important to think about this. The world is changing very aggressively, - and every company that I work with is trying to figure out how to do this. We are no longer building systems. We are going to grow systems. Out of this comes a system, - and then we give it some more input and say, - "We didn't quite mean it like that. We actually meant it more like this". That's where I see the world of software engineering going. And that's why we have the title of the presentation. Of course, we're not there yet. So, what I now see is that many of the companies - are looking for a way to connect those two things. "What do we do offline? What do we let humans do?" "What do we do online and let the system do?" I think those are the questions we're currently struggling with. Then, of course, I had to ask ChatGPT what the fabulous future - of AI-driven software engineering look like. This is what it comes up with. And I showed this to some of my colleagues, - and they said, "Oh, no. We're still in offices looking at computer screens". I mean, that person thought it was pretty boring. But in any case, three takeaways. Because I think that many of you in this room think - that we're doing pretty fabulous from a software engineering perspective. And that it can only get better. Now, I would like to disabuse you of that concept. The state, the effectiveness of software engineering, - as far as I can determine today in the companies that I work with, - is atrocious. It is really bad. Second, in order to address this, we need to shift - from what I call requirements to what I call outcomes. We're going to talk a little bit about that. And we need to adopt AI in the product and in the development process. Those are the three key takeaways I want you to remember from this talk. That's what we're going to do. First, I'm going to briefly tell you who I am and where I come from. I'll talk about the state of software engineering, - about requirements and outcomes, exploiting AI, - and then we're going to conclude. So, very briefly about me. I used to be a young and innocent professor of software engineering. Then I went to Nokia where I got to know Ilari. I don't think he is here right now. And about 13–14 years ago, - I moved back to Sweden and moved back to academia. Sorry. After Nokia, I went to Silicon Valley, worked for Intuit, - for those of you that know the company, and have lived in the US. And since about 10–15 years, I'm now back in Sweden. I have two professorships. I run a software centre, a consultancy. I do some board work and some angel investing, - including Kosli, which is now run by Mike Long. Does anyone here remember the name Mike Long? Praqma? At least one or a few basically remember him. So, that's actually what I do. My main job, however, is being director for a research centre - between 15 companies and five universities, - all concerned with accelerating the digitalisation capability - of the European software intensive systems industry. And these are traditional companies that have built products, - that have mechanical components, that have electronics components, - and that have software components. So, that's basically what we do. And I will not bore you with all the details here, - but we do work on continuous integration, - DevOps, continuous tests, and those kinds of things. Architecture, technical debt management, APIs, - metrics, customer data and ecosystems, and then AI engineering. Those are the main topics that we work on these days. And why should you become part of software centre? We can talk about that when we go offline. But what I basically do - is I identify new ways of working in the pure SaaS world, - and then we translate them to how do you actually do this - in embedded systems companies. And if you look at all the new developments, - if we look at agile practices, at continuous integration and tests, - if we look at DevOps, if we look at A/B testing, - if we look at all kinds of different techniques - that are currently being adopted around the world, - they start off typically in leading companies like this, - and they trickle down to the rest of the industry. So, that's basically what I do. State of software engineering. There is a very big elephant in the room - when it comes to software engineering. Here is the belief of every engineer that I meet. I get a requirements spec from the customer, - I build what it says, I test it, - and I deliver the system, - based on the requirements spec, to the customer. Of course, the customer is happy, right? Wrong. You've all seen this graph, I think, many moons ago by Standish Group - that basically shows that in a typical system, - somewhere between half and two-thirds - of all the requirements in a typical system - are never used or used so seldomly - that the R&D effort that went into building them wasn't worth it. A few years ago, we repeated this study - with a company in the Gothenburg region in Sweden that, - after they saw the data, decided that they wanted to stay anonymous. Funny how that works. But what we see is exactly the same pattern. We have all the features on the x-axis. We have the number of times a feature is used on the y-axis. And we see the same pattern: some features are used all the time, - some features are used occasionally, many features are never used. So, my conclusion is that half - of all the new functionality we built is waste. Because understand me well, this is not a feature - like the airbag feature in my car. I hope that the airbag feature in my car is never activated. Because if it is, I would be in an accident - or one of my family members would be in an accident. These are features where product managers - stood hopping up and down in front of a prioritisation meeting - saying that unless we get this feature, - we're all going to go to hell in a handbasket. And you get it prioritised. It gets built. It gets released. And guess what? Crickets. Nothing. How cool is that? As you probably already figured out by now, - I've been going for nine minutes, - I'm not a very smart person. So, I can only do very simple models. This is a model that we call the three-layer product model, - which says that every product, every platform, every framework - has three layers of functionality. A layer of commodity stuff that no customer cares about. It should just work. It's been in the previous releases. It's already there. Don't do anything with it. A layer of differentiating functionality, - stuff that makes the product special, - that makes your customers pick you over a competitor. And then a layer of innovative and experimental functionality. You know how it works: you start with innovating something, - customers really get traction of it, - it becomes part of your differentiating functionality layer. And then after a while, it becomes a commodity, - and it basically slides down into the commodity layer. So, what I have now asked well over 50 companies, - what percentage of R&D in your organisation - goes into commodity functionality? And what percentage of R&D goes into differentiating innovative - and experimental functionality? You want to know the answer? As far as I am able to determine, - based on self-reported numbers from these companies, - somewhere between 70% and 90% - of the R&D resources in the company, - go to functionality that no customer gets a flying hoot about. It should just work. It was already there. "And why is this?" "Well, there's an update of some external software." There's all kinds of excuses of why this is happening. Often it's also that people inside the company - think it's still differentiating and it isn't anymore for customers. But what we see, and this is my conclusion - about the current state of software engineering industry, - is that seven to nine out of every ten people in R&D - work on stuff no customer cares about. It should just work. The one or two that build new functionality - get it wrong half the time. Hence, half or two thirds of all the new features are never used. And if you translate this into a cost and value game... You can take pictures, but there will be more on this slide. If you have a company... Most companies allocate their R&D budget as a percentage of revenue. So, in this case, I took a company that allocates 5% of its revenue to R&D. You're with me so far, right? You know how 5% works. I'm not getting much of a reaction here. I'm not sure... Is this too complicated or did I hurt your feelings? You know, this is a funny thing. No engineer wants to talk about his feelings, - or her feelings for that matter. So, the question that I then have is, - for every euro that gets invested in R&D, - how much revenue should be generated out of that euro? You want to know the answer? 20X. R&D percentage is 5%. That means that every euro that goes into R&D - has to generate 20 euros of business value. Now, let's take this into a number. I have an eight-person team. I have a three-week sprint, - meaning I spend 24 weeks of effort every sprint - with that eight-person team. If I'm assuming that an FTE costs 200,000 euros, - which of course in London is way too low, right? Okay, not everyone is agreeing with that statement, - but let's not go there right now. Then basically you're burning, in terms of cost... Every sprint, you are burning 100,000 euros. With me so far? How much value do you have to create during that sprint? Well, the answer is very simple: 20 times 100,000, - which is two million euros. So, if you want the economics of R&D to go together, - every agile team, every agile sprint - has to generate two million euros of business value. In a context where seven to nine out of ten work on commodity stuff, - and the one or two that build new stuff got it wrong half the time. Can you see that I'm slightly concerned about the effectiveness of R&D? Are you still here? Is this something you care about? Because I do. I mean, in the end, we are in a for-profit business, right? We have to do what we can to generate maximum value - from the R&D effort that we are creating. I think that's obvious to everyone in the room. You may not like to think about it. You prefer to think maybe about empowered teams and all this. But if we don't deliver business value, no one cares about empowered teams. It's very simple. What's the problem? I think requirements are the problem. If I look at the companies that are the most mature - in how they link the value - created by R&D teams to business outcomes, - they create something that I refer to as a value model, - where basically you have top-level business factors, - the KPIs that you have as a business, - that get translated into product- and system-level factors, - that get translated into team-level factors. If I look at the companies that we worked with, - companies like Microsoft, Booking, Skyscanner, and others, - they tend to work in this way. When I worked with Booking in the 2010s, - every agile team, every agile sprint - knew exactly how much money they made for the company. Because they knew which were the KPIs. And they knew that if in their part of the architecture, - they were able to shift the needle a little bit, - they knew how much money they made for the company. Now, I don't have time to go into this. But the interesting thing is, I have a whole process with reference. And if you want to talk to me afterwards come and talk to me. But what I noticed is that in almost every company that I work with, - teams have very little agreement and understanding - of what the measurable KPIs are - that would constitute a successful realisation - of the functionality that they're building. This is from a Swedish automotive company - that shall remain unnamed. We were discussing the adaptive cruise control feature in the vehicle. You know adaptive cruise control, right? You have a radar, you have lane keeping, - and you basically have adaptive cruise control. That's a feature that has been in the vehicles for well over a decade. It has cost well over a hundred million Swedish SEK to realise, - actually significantly more than that. And now we started, sat down with the team, - these were about 15 people, I think, - to discuss what quantitatively constitutes - a successful adaptive cruise control system. How do you measure success? Guess what? Fifteen people. No one agreed on what a successful adaptive cruise control system is like. Some people talked about how often it's used, - how often it's used at different speeds, - how often a driver interacts with the system. All kinds of different KPIs came up. So, imagine a situation where you have a 15-person team, - and not a single engineer agrees with everyone else - of what success would look like. What do you think the consistency and alignment - of the feature realisation is in that context? So, this is, I think, the first one. You have to translate. We actually have one company now that I, again, cannot mention, - where every time product management comes to the team and says, - "We would like to have this functionality built", - the first question the team asks is, - "What is the outcome that you're looking to accomplish?" "How would we know that we deliver success?" So, that is the next thing. And that's what we see in AI as well. You cannot do a lot of things - if you don't know what you're looking for. You end up in an Alice in Wonderland situation - where she asked the Cheshire Cat, "Where am I going? Where should I go?" And then the cat says, "Where do you want to get to?" Then Alice says, "I don't care". And the cat says, "Then you can take any road you want". Then it doesn't matter, right? You need to know where you want to go. Here, this is the HYPEX model, - where you take a feature out of the backlog, - and the first discussion between you and product management is, - "How will we quantitatively know - that when we build this feature, we have achieved success?" "What is the behavioural change in the system - or the behavioural change in the customer - that justifies us investing this kind of money? And then what you do is you build a minimal viable feature. I first called it, which is now the term to use, - minimal lovable feature. But I've realised that no engineer wants to talk about their feelings, - and they certainly do not want to talk about love is my experience. So, minimal viable feature, you build the first 10–15% of the feature, - you measure what is the impact of this thing, - and then you have four possible outcomes. One is, nothing happens. And what you should do is to ditch the feature, - remove it from the code base, and move on to the next feature. Another one is, you build 30%, you get everything you wanted. Then you don't have to build the rest of the feature. Just move on with your life. You already got what you wanted. Sometimes you get really funky feedback. And when you start to interview users and you ask for qualitative feedback, - what you'll see is that basically they want the functionality, - but not in the way that you built it. So, you have to re-implement it. Or maybe you have a way where you are gradually increasing - the quantitative output that you're looking for. That's one of the ways in which you can work with this. That's what we call the HYPEX model. I assume everyone here knows A/B testing. So, I'm not going to talk about that, - but that's, again, a very effective way where you can figure out - what's the right way of realising functionality, - so that we're actually delivering the value that we're looking for. So, those are the things that we're looking at. What I'm now going to show you in the last ten minutes - is a little bit of progress - that we've made in this area over the last years. And the first is, this is an in-progress maturity model - of how companies work with AI. And this is not R&D, this is the company as a whole. So, the first is, we see, - based on interviews with about, I think, 10–15 companies, - roughly five stages. This model is still under development, but we're still looking for ways - in which we are capturing the maturity of these companies. So, the first step is play-time. Everyone and their brother is using ChatGPT or some other LLM - for trying some personal productivity tasks. Anything from test case generation, email generation, - documentation generation, those kinds of things. The next step is automation 2.0. We have a process. The process has a sequence of steps. One of those steps we have not been able to automate. Now we're going to try to automate that step in the process - using an LLM or something else. That's the second step. The third step is, the local AI first. Then I take a business process, - I re-engineer the entire business process beginning to end - in an AI-first capacity, - but that business process is re-engineered - in an isol