Sneak-peek into Eficode customers’ AI maturity

[Marko] (0:03 - 0:11)

Our customers are moving in new code creation, remember, from AI assisting people into autonomous AI in use.

[Pinja] (0:14 - 1:05)

Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development. Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna.

Hello and welcome back to the DevOps Sauna. We are doing a little sneak peek today to our customer satisfaction survey results, and not just what, of course, our customers are thinking about us, but we did a little extra survey as part of the CSAT. A couple of episodes ago, I talked to our CTO, Marko Klemetti , about the five-minute loop as an aspiration.

So here to join me again is Marko Klemetti , our CTO. Welcome, Marko.

[Marko] (1:05 - 1:07)

Thank you so much.

[Pinja] (1:07 - 1:08)

It's good to have you back. How are you doing today?

[Marko] (1:09 - 1:16)

Oh, fantastic. I brought the sun back from Amsterdam, where I was visiting the Atlassian Team event yesterday.

[Pinja] (1:16 - 1:50)

That's amazing. And aspirations to the five-minute loop is one thing. So we sent an AI maturity questionnaire as part of our CSAT survey to our customers.

And we're now seeing that, yes, the five-minute loop is an aspiration. It is a goal, and we've been already talking about this, why it should be an aspiration. Why does it actually matter so much?

But what did we see with our AI maturity survey with our customers? Because I think we might be seeing AI creating a two-speed organization right now.

[Marko] (1:51 - 3:36)

Yeah, that's an interesting finding. I have to thank our organization for starting the CSAT survey already one and a half years ago, so that the first survey we added these AI maturity questions was actually March -25. And we started asking relatively simple questions like, how much do you use AI for new code creation?

How much do you use it for requirements management? How much do you use AI for legacy refactoring, et cetera? And as we've done it every three months, we are starting to really see a few interesting trends happening in those answers across our customers.

And maybe we'll start before going into the two-fold organization or two-level transformation within organizations. One of the immediate findings that I can bring from here is the new code creation. So GitHub Copilot was launched already, I think, June -21 as a beta.

So it's been around for a while already. Then a bit over a year after that chat, ChatGPT first version started to democratize the AI scene. And now as we, beginning of -25, started asking our customers, which are traditionally more in the regulated industries, the first survey new code creation answers, which essentially you can select from five options, not using AI, experimenting AI, AI is assisting people, agents are assisting in workflows, and then autonomous AI in use.

March -25, half of our customers weren't using AI at all for new code creation.

[Pinja] (3:37 - 3:42)

Yeah. That's considering what we see nowadays. That's a huge leap that we have seen, isn't it?

[Marko] (3:42 - 4:31)

Yeah. Yeah. Yeah.

And that's why I said, that's why this is the most interesting part. And it's good to start with that because so the amount of... So from March -26, which is a year after, half a year ago, and then June this year, the amount of new code creation, the organizations not using AI dropped from 20% to roughly 12-ish percent.

So the change in not using AI, interestingly, is decreasing slower because as we know, the adoption curve for organizations, there are those laggards, the ones that have industries that they simply cannot use AI yet. For example, in the defense sector or public sector or some other very, very regulated sectors, it might be difficult.

[Pinja] (4:32 - 4:36)

We need to look into sovereignty as a requirement for these industries, right? 

[Marko] (4:37 - 5:16)

Correct. So just using cloud or chat GPT simply isn't an option for many organizations. And as we know, in Europe, we have only very few alternatives. And the open models are also still open, which open models should even be used and how we should host and build them for our customers.

But the interesting finding is here. So first of all, the experimenting and investigating has kind of flattened. So organizations move naturally from this not using into experimenting, but AI assisting people has dropped by 10%.

[Pinja] (5:16 - 5:18)

That is an interesting find.

[Marko] (5:19 - 5:52)

It's an interesting finding. And this is interesting also for the listeners. So the reason why AI assisting people has dropped 10% is the software development has jumped into autonomous AI in use.

So the amount of customers, our customers are not moving from not using AI into AI assisting people. Instead, our customers are moving in new code creation, remember, from AI assisting people into autonomous AI in use.

[Pinja] (5:52 - 6:09)

So it might be very hasty to draw any conclusions that, well, nobody's actually using AI assisting people or that we're actually going down in maturity, where instead, we are actually taking that leap. Is there anything else that surprised you with these results when you're looking at it?

[Marko] (6:09 - 7:34)

Yeah, maybe I'll first say that when we look at the five minute loop, the last podcast that we did, and I was talking about the polarization of organizations. So why five minute utopia is so important is because the vulnerability is exposed in the markets. The problem with open source or third party libraries might actually become a problem in a software that customers are developing themselves.

But the problems and vulnerabilities are oftentimes not attached to the software the organizations create themselves, but something that they're consuming from outside. And the speed of being able to react to this change is vital. And that's we're talking about this five minute loop, right?

However, we now see with the survey that we're doing that polarization is happening faster and faster. And this is where if we look at the AI assisting people decreasing and going into autonomous AI in use, we see that the organizations who have been able to adopt these so-called native AI practices embedded into their software development lifecycle, they are becoming better and better all the time. And the ones who are lagging behind, they are kind of falling faster and faster behind.

And that in itself is, I think, a very, very interesting and important finding.

[Pinja] (7:35 - 8:05)

That is one of those very interesting findings as well, because not only we previously talked about how the whole organization should be part of this change in order to enable that five minute utopia, as you say, and now we have these organizations who are falling behind, they might have been laggards previously. So there are even more laggards. So when we have had an implementation maturity curve, basically, do we see that widening?

Like, basically, the early maturity and laggards? Are they getting further away from one another?

[Marko] (8:05 - 9:40)

Yeah. And this is, I think, where the second most interesting finding is connected to. When I look at the numbers for, for example, requirements management, we started on March -25 from 80% of our customers not using AI in any way of defining the requirements.

Now it's already below 40%. And most organizations have moved not only to experimenting with AI in requirements management, but also AI assisting people. And this is natural when you look at, for example, Atlassian Jira, the Rovo, the Claude, ChatGPT, used as part of defining what should be done and getting kind of this more extensive insights.

That too is developing very, how should I say, like linear, but slightly exponential on the best organizations. And then legacy refactoring, security and compliance, test management, which means essentially test automation. All of those are developing in the same manner, which in my hypothesis is proving to be true.

The organizations who have been able to implement these practices in one part of their technical processes, such as software development, are very rapidly implementing the same practices in other parts of the so-called technical delivery, like test automation, AI-Ops, or cloud operations, and security and compliance. However, where we see certain lag is the business and service teams. So essentially the beginning and end of the five-minute loop.

[Pinja] (9:40 - 10:15)

Yeah. And this is also my theory and my hypothesis right now, is that for business and service functions, we're not exactly increasing this agentic way of working. Many organizations are using AI tools and AI assistance, but it's very individual.

It is not exactly scaling and it's not exactly transforming the workflow itself. So it's like in our previous episode, it is that legacy AI. We add a nice sticker on top, business function now comes with AI, instead of actually figuring out how we can do this a new way with AI.

[Marko] (10:15 - 12:43)

Yeah, exactly right. And I think when looking back at why the technical side of organizations are transforming so fast, this is an example I'm borrowing from one of our colleagues. Program languages have been made for us humans to be able to tell stupid computers what they should be doing, right? And now when we start implementing AI from the computer side, programming language for an AI is the easiest place to start because it's been the native language for computers in the beginning already.

And that's one of the reasons why all of these kind of technical aspects of digital services are evolving so fast, because it's easier to write programming language or write structural language in programming languages than it is to aid people with natural language, with all of its complications and all of its kind of slightly biased and colored way of expressing things. So it's not an exact science to use natural language. And here I see that as AI has developed over this telling stupid computers what they should be doing.

We've stepped into an area where essentially it is natural for us to start implementing AI. And we're starting to see somewhat, so some of the findings are valuable, but some are not valuable. And we've kind of stepped into the next phase of AI transformation.

So the first part of that transformation was implementing these tools as fast as possible. Pick a tool, give the license to everybody, see what comes out of it. And it's more of this kind of implementation and AI tooling, AI tooling consumption question.

And now we see that organizations are naturally moving towards what is the value coming out of our AI and how can we improve that value? So the tooling isn't secondary still because the tooling ecosystem develops so fast. But along with the tooling question, we have, for example, both the security and the context questions, which means that how do we make sure that when we consume the AI tools, they're providing us with valuable answers.

They're aiding us in our business as in comparison to the costs that we spend, but also that we can take or let's say leverage these tools in a more efficient fashion.

[Pinja] (12:44 - 13:20)

Who are we having these conversations with? For example, if we take our customers, who's our counterparty? We often talk to your colleagues, the CTOs, we might have a CIO.

Is that creating a little bit of, I'm not saying that we're not talking to the right people, but if we think of a transformation and that transformation should go through the whole organization if we want to implement that five-minute loop and we want to also get the business and the service teams and support teams with us, are we getting a little bit of an investment bias towards the tech side? And do we have an owner for the other part of the organization's AI transformation?

[Marko] (13:20 - 16:08)

Yeah, I think I'll just lift our own tail just slightly. We started doing DevOps before it was called DevOps. We called it Software Production Improvement or SPI back in the day.

Correct. And we started calling software development with its natural language AI in SDLC or AI in Software Development Lifecycle, which is for us, Eficode, one of the two journeys that we have with our customers. So the first one is AI in SDLC and the other one is effective SDLC tooling.

And when we look at the AI in SDLC, it's the DevOps transformation or the automation transformation for those who still don't quite know what DevOps means. It's the same but supercharged. So everything, the same things that happened over a 10 year period in automating the organization and bringing the IT and development organizations together, hence the name DevOps, by the way, the developer and operations, and forcing them to have conversations together.

So building a common responsibility, the common feeling of ownership of whatever is being built. So before developers were creating code, they built a binary, they tossed it over the wall to operations or IT organization, and then they started managing and operating for the customers. And that's a fantastically stupid way of doing software.

It was just the best way we did know how it should be done, that DevOps kind of changed that. And now when you ask which roles we talk to, it's still somewhat divided into two. So when we go to R&D leads and CTOs, of course, we have the conversations of AI in SDLC.

We run our AI survey. We find the pain points for our customers. We identify what needs to happen in both implementing these tools, but also changing the architecture and the processes within the organization, right?

But then when you start looking at which tooling should we put in place, and how do we ensure that the developer platforms or platform engineering spawning from the DevOps world, how do we provide a tooling platform, not only across the organization, but across the so-called five-minute loop, right? How do we ensure that the tooling works and integrates together? This discussion is then with either IT manager or IT directors or CIOs.

So still we're having these similar divided discussions with either the R&D part or the IT part, but since it's supercharged, these parts of the organizations really have started to collaborate faster together.

[Pinja] (16:08 - 16:53)

And that's basically what all of DevOps has been about. It's not, fine it comes from development and operations, but it is the whole organization working seamlessly together, whether it's getting the business into the play, getting marketing into the play, as you say, the very beginning and the very end of the five-minute loop, because we need to figure out from the market discovery side what should be done. And if that is not being supercharged, but we have the middle part that is supercharged, then we don't have the ends of continuous improvement.

And then again, somebody on the product side saying like, this is how we want to improve our current products. And this is the life cycle. If they're still lagging behind, we're not exactly getting the best benefit out of it.

[Marko] (16:53 - 19:55)

Exactly right. And I've been on stage many times saying that if your R&D budget is 100%, most organizations spend 5% of that budget on actual software development. And if you automate the whole software development using AI, you have improved your organization by 5%.

And this is why I'm talking about the five-minute loop so much because it involves, so if you look at the five-minute loops, it starts from the market discovery. So defining what we should be doing, what are the market signals? What are the signals from our customers, either for an existing product or a new innovation, new product?

And then it goes through the kind of requirements documentation definition, the technical part where you have the development, testing, security, and the deployment automation, all the way into what is called service management or customer service, where we start to understand whatever is happening with the customers who are consuming these products or these features that were delivered. And now when we look at the bottom layer of this five-minute loop, which then consists of understanding the market, defining what needs to be done, and then understanding the customers consuming these tools, that's the part where humans have the most insight. And this is where the kind of human insight will be the strongest in the future.

However, kind of the technical part of the five-minute loop, the development, testing, security, and deployment plus operations, like the technical parts of the operations, those will be faster and faster automated over into AI. And now, interestingly, once again, if we go back to the CSAT survey, we can see that all of these technical aspects are being developed, implemented, and employed in organizations faster and faster. But the bottom layer where you would actually hand over the information over to the customers, we call it from idea to value, it still seems to contain lots of information sharing from sometimes, usually from team to team, but sometimes even from department to department.

And these handovers become bottlenecks super fast. And we can see it from the CSAT survey. Similarly, as we'll see the technical parts implementing and automating becoming faster, that there is a clear lag in the actual business and service management culture within organizations.

How do you ensure that you have kind of, how should I say, straight pipe into value and how the same pipe brings you back insights from the customer on how this feature brings value to the customer or the company. And currently, no organization knows that, or cannot say no organization knows that, but the traditional organizations have a very limited amount of information available in making these decisions as part of the whole process.

[Pinja] (19:56 - 20:13)

We understand that AI now, of course, brings this opportunity for us to streamline this interaction, right? So if we're thinking about AI bringing automation, how can we add that now? Where would you start with an organization if we remove that manual process now?

[Marko] (20:13 - 23:07)

This is the same answer I give to everybody. So here you go. The first one is understanding where your organization is.

For that purpose, we do AI surveys for our customers. And the second one is experimentation. So the same way as the development teams and individuals are trying out new tools, they get some level of permission to use.

So at Eficode, we, for example, have an AI policy containing labeling. So you have different labeling for different levels of tools, and then different labels give you the ability to try more extensive and extensive features. And of course, when you have the so-called white labeled products, you can even use them to an extent in most of the customer projects as well.

And then you would have gray labeled, which is something that you can use, trying out, learning something that we look at as an IT organization, do we promote these into white labeling or do we keep them in the gray labeling? So experimentation. And usually experimentation happens in organizations by starting something completely new.

That's naturally the easiest way to start. Most organizations, most enterprise organizations have done this somehow. So I like to tell the Volkswagen example where when the ID vehicles development started, Volkswagen started a new company called CARIAD, which their purpose was to build the operating system and the infotainment system for the new ID vehicles.

And by taking them out of the enterprise environment and starting to build the way that modern organizations would implement these tools and the processes, they had free hands. And as they showed what to do in an enterprise like Volkswagen in a modern world without similar restrictions they had in the enterprise side, they were also able to then bring these findings back into the enterprise. And smaller organizations, especially in regulated industries that I meet, they talk about regulated.

They often talk about really deep sovereign requirements, whereas many of their services are similar SaaS cloud services that any other provider has. And those are excellent places to start experimenting. And then just limiting the findings by the restrictions they have in each of the different areas, instead of just looking at the whole bunch of services that they do with the most extreme labeling, but rather looking at what are the kind of restrictions that we have to apply in each of these levels.

And then if there are little to no restrictions, those are usually the places that you should go and experiment, find a way how in this organization we could do the five minute loop, the utopia, and find what tools are already in place and why, and then learn from that.

[Pinja] (23:07 - 23:25)

If we now consider going to an organization right now, what are the telltale signs for you when they say, well, our AI maturity is quite high? Many organizations are indeed saying this. So what are the telltale signs to you that they are indeed high in AI maturity, apart from them saying that this is what we're doing?

[Marko] (23:25 - 25:45)

So first and foremost, most organizations mix these two concepts, the value of AI and the implementation of AI. So when organizations say that our AI maturity is high, they very often mean our AI tooling implementation level is high. And this is what our survey shows as well.

You can see that our customers are really using these tools. They're really putting automation. They're really putting autonomous AI in place.

However, it tells nothing about the value that they're getting from the tools. And it tells nothing about the actual implementation of these tools. So we tend to divide organizations into two categories.

And if you look at our AI adoption framework, the category first is improving the current work, which means that we have individuals working in a certain process. We implement AI, and it's implemented for each of the individual or team separately, which means that the same bottlenecks and handovers remain in the organization the same way as you had before, which means that you get the so-called 5% improvement in your R&D because you have automated the small parts of the team or individual work. And then there are these organizations who try to be brave in either removing these handovers or bottlenecks.

So that's what agile tried to do in the very beginning. We just didn't have the mechanisms to boost it up more and make sure what we produce can actually be delivered. That's a separate discussion.

But kind of when we go to our customers, it's relatively easy to see with a rudimentary discussion whether they are good at implementing the tools in the organization or if they are really good at finding ways in that organizational context in creating value or improving the actual business processes within the organization. And you can see it, of course, the rudimentary discussion is not only the details on how to do this and this and this, but also it reflects kind of the top management's support and vision into what is being tried to do with the AI implementation.

[Pinja] (25:46 - 26:06)

These are very interesting findings and very interesting results. As I said in the beginning, this is now a small sneak peek of what we found as part of our CSAT with the survey considering AI maturity. I think we need to stop this conversation right now, because as I said, this is a small thing we're now giving out.

But thank you, Marko, so much for joining me here today. 

[Marko] (26:06 - 26:07 ) 

Thank you so much. 

[Pinja] (26:07 - 26:17)

And thank you everybody for tuning in, and we'll see you in the sauna next time.

We'll now tell you a little bit about who we are.

[Marko] (26:17 - 26:46)

My name is Marko. I'm the CTO of Eficode. I've already been there for 20 years, seeing the world change from the traditional automation of software development into the AI-driven world of automated software development lifecycle.

I'm a nerd at heart, so I still do software development myself. And at the same time, I sit on the C-level discussions almost daily with our customers, advising them on how organizations should change.

[Pinja] (26:47 - 26:54)

I'm Pinja Kujala. I specialize in agile and portfolio management topics at Eficode. Thanks for tuning in.

We'll catch you next time.