AI maturity at scale: Increasing the AI ROI from the perspective of private Equity

[Pinja] (0:11 - 1:08)

Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development. Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna.

Hello and welcome back to the DevOps Sauna. Today, I would like to take a little bit more personal aspect on the case of and the topic we're covering here today. For the past six months, I've been working on a customer project myself, and this has been through a private equity company.

They own several portfolio companies, and we've been doing an AI maturity assessment on multiple of those companies. Today, I'm not going to be here talking about this by myself, but I'm joined by our Field CTO, Kalle Sirkesalo. Hey, Kalle.

Hello.

[Kalle] (1:08 - 1:16)

Yeah, I was thinking, should we call it an AI Sauna today? Because we're going to be... It's going to be DevOps, but it's more of an AI today.

[Pinja] (1:17 - 1:52)

It is a lot of AI. There's been lately a lot of AI anyway, because that's one of the topics that is changing how we do DevOps so much in the past couple of years, and it's going to be still changing how we do DevOps in the next couple of years. But the topic for today, yes, we have this case story that we want to talk about, but the red thread here is that we wanted to talk about what have we seen in terms of building this AI maturity with scale and through a private equity company?

So where did we actually start building this case? We were not the only ones from our company to work with this case. We had multiple experts, but we started to look at the AI maturity model.

[Kalle] (1:52 - 3:11)

Yeah, and it's not just us working on this. Every company in the field is trying to do this. You can find a multitude of AI maturity models now in scientific research and so on, but when we started, there were very few for the LLM side, or in the specifics of Eficode, we're on the SDLC side.

So we're not looking at the generic AI, we're looking at the AI that is within the SDLC, within the PDLC of the customers. We're looking at how we actually enable AI for software developers? Not to the software, but to the software developer.

How do we make it visible to the customer software developer? So what we started figuring out is what goes into the software development lifecycle and how can we find it in scale? Because when you work with private equity or at scale like ours, we have multi-thousand customers.

So we want to be able to scale the solutions that we offer. You start figuring out, okay, how can we actually do it? What would be the approach?

So we made a survey and we used a Likert scale to figure out like, okay, this is where you will end up in one to four basically, and tried to figure out where you could position and what kind of questions would be asked? And you, Pinja, made a lot of benchmarking around this based on the other customers. So I think I'm going to throw you about like, how do we actually convert that into something reasonable to the customer?

[Pinja] (3:12 - 4:46)

No, with the Likert scale with one through four, so basically we're asking, does this apply to you or not? So it was a very comprehensive survey, as Kalle said. And then it was not just that we're going to present it like, well, you're doing 1.5 in this area, you're doing 2.5 in that area, but rather actually benchmarking, as you say, towards others who are in the same industry, in the same geographical location, and with the same size, so that you actually get your peer group. You have this context group where we can actually benchmark the customers and these portfolio companies, because then you can actually see what does it mean to be in a regulated industry that is working in Europe that has approximately, let's say, 600 developers or 600 people related to their software development lifecycle, versus somebody who's in a non-regulated industry in the US and has thousands of more people working with that industry.

Of course, the AI maturity is going to be different. It's not comparable, but we wanted to give them the benchmark saying, hey, the best of your class, best of your peers are doing this well, for example, let's say 3.5 out of 4. But at the same time, we need to be extremely careful how we treat that, because at the same time, it was that context that we're comparing to.

So some companies did not compare. Of course, we have a gaming company. It doesn't compare to somebody who's in a regulated industry.

But as I say, we wanted to cover all the parts of the product development lifecycle, say strategy, the building part, delivery itself, observability, but then AI adoption, AI orchestration. But it's not just that - how do you use GitHub Copilot, for example?

That was not the only thing we were looking at.

[Kalle] (4:46 - 5:41)

What we are seeing in the customers is that Copilot usage, yeah, that's high, so to say. Everybody has a Copilot. But when I start talking to the customers, they're like, it doesn't mean that they're using Copilot effectively or widely.

They might be using just a chat box in the corner and asking it questions. And that's very usual, asking questions about how my code base does X. But then you have the whole asymptomatic side of this, or let alone code reviews and automating code reviews, creating quality controls and comparable.

And a lot of the organizations that we actually surveyed didn't have automated AI features in active use when we started. So we had to figure out where you are in the AI scale, like we tried to figure out different use cases and track those. And then on top of this, we had to figure out, okay, how do we actually verify the results?

So we want to actually go and do a little bit more discovery at the customer. So we wanted to workshop with the customer, but in the correct topics.

[Pinja] (5:41 - 6:18)

What is important to understand, when you send a survey to an organization, if we ask, how well do you think that you're doing with product development stuff, for example, product management, let's take that as an example. And it's your self-observed maturity, isn't it? So with these kinds of deep dive sessions, it might be a workshop, it might be an interview.

We get a better understanding of how the organization is actually working with these tools that they say that they're doing well with. And it might actually be that individually you're working very well within these areas, but the whole lifecycle thing is not connected, which then means that your whole AI maturity is not actually on a very high level.

[Kalle] (6:18 - 7:15)

Yeah. And it created this, okay, how do we actually find the use cases that you're developing or not yet developing? What are the possibilities?

Where should you invest? And that created, when you're working with private equity, it's always about the investment. How do we make sure that we are correctly investing the money and the resources so that we can get the maximum change and possibilities?

And that creates what do we do? So we use value stream mapping a lot. We use tool chain mapping a lot.

We used interviews on specific things by trying to figure out how your strategy converts to actual developer decisions? Because we had a few really wonky cases where the vice president was like, yeah, we are fully AI native. And the developer was, yeah, we don't have any AI.

We haven't even seen the AI tools yet. And you're like, guys, how's this going? And they're like, it's so weird when you have these surveys where you see like multitude of layers, like how thinking changes.

[Pinja] (7:15 - 7:39)

It does. And like going into these workshops and having, let's say somebody from development organization say in front of a VP of some other part of the organization that yes, it's not actually working very well and unravel the truth to somebody in that specific organization, you might actually get in some awkward situations when they actually get for the first time to realize what indeed is our maturity when it comes to that.

[Kalle] (7:39 - 8:29)

It was so fun. Like some of the customers were very okay with the results. And some were in total shock about the results that they got.

And it was always interesting to see what happened. And that led to the whole thing. Like what did we actually see in organizations that succeeded and why?

And we're not exactly sure, of course, of why. Because most of the time, I would say that like big thing was the first two boxes, that strategy and how do you actually get the product vision alive. So what is happening from strategy to product.

Because the thing is delivery observability, building software, that's easy to change because it's just developers working on things. But if you don't know what you're going to be changing or how you're going to be changing things, it becomes really difficult to explain to the customer that you need to change. So I think the number one thing for me was how fast can you change as an organization?

[Pinja] (8:30 - 9:05)

Yes. And how fast can you understand where the value creation happens in the age of AI? I recorded another podcast some time ago with our chief product and partner officer, Henkka, and we were talking about exactly this, that when you have AI coding tools, it is so much faster to create that code in software development.

But that's useless unless you have that north star, unless you have product management practice, unless the parts of the organization are working in such a good sync that the strategy is actually flowing from that part of the organization towards the development and into the actual final product. Otherwise it's nonsense.

[Kalle] (9:05 - 9:41)

I liked one of the podcasts that I listened to, reviewing this and saying out loud what 20 times the speed means, because they made it really nice sounding, because they do get sold that you have about 200 days of work in your work year. 20 times the speed means that you remove basically a zero behind it. You do the work in 20 days that you used to do.

No, that's the wrong mathematics. That's still like, that's just 2x. 2x is 100 days.

4x is 50 days. 8x, and it keeps going lower. So you end up in this situation.

So in 20 days during a year it is 5x.

[Pinja] (9:42 - 9:52)

So maybe that gives you, yes, that gives the idea of the speed. So again, but then we can actually compare these two. If you're running really fast, but you're running in the wrong direction, does it matter? Did you win the race? 

[Kalle] (9:53 - 9:55)

And again, can you think a year's worth of work in 20 days? 

[Pinja] (9:55 - 9:56)

No, absolutely not.

[Kalle] (9:56 - 10:16)

So imagine when you, that was 5x, but imagine we have metrics that show that we have organizations like Antropic or GitLab that are capable of running at 20x in specific teams. That's four times the speed of that previous, that's in a day you do a year's work.

[Pinja] (10:16 - 10:32)

Exactly. And like, really, you need to be able to make the changes fast. You need to be able to move quickly, but this really, to make it land, you need to be able to make these hard decisions and make them fast.

Otherwise you're not going to follow up with your peers.

[Kalle] (10:32 - 11:15)

It becomes this problem of the hard decisions usually, what are we not making? Because in history it used to be like, what are we making? Now it's becoming, what do we not do?

Because like everything we do outside of the product vision, scope, customer, waste. And now the waste becomes maintenance waste, upkeeping waste, and it becomes a challenge. Because this became the challenge when we were starting to look at these results, like the organizations that were tied up and couldn't do AI transformation had problems in DevOps safety nets.

They had problems in the architecture of their software. They had problems in their testing. They had problems releasing.

Most of them couldn't have, like if the AI metric to model was low, it meant that their releases used to be very slow. So they couldn't do feedback loops.

[Pinja] (11:15 - 11:40)

Exactly. And when we say releases, and some of the organizations were actually pushing back on this when they said that, oh, but we don't want to release more than twice a year. That's not what we mean.

That is a customer release, which always should be a business decision. You need to, of course, adjust if you're in a regulated industry, your customers are not ready, all that, it is all valid. But in a modern world with the feedback loop actually being that fast, you should be able to do an internal release multiple times a day.

[Kalle] (11:40 - 11:54)

You should get feedback. Yes. From the product.

And you should use that product yourself so you get the feedback. You should have everybody somehow related to that product. Somehow you're getting that feedback.

Because if you don't have a feedback loop, even the internal one, you will fail.

[Pinja] (11:55 - 12:18)

One of the big issues and challenges we saw with these organizations, and we see with others as well, is the communication between the leadership and the development. And as we said, if the communication strategy doesn't work, you're going in the wrong direction. And another one, of course, is scaling from one team to another.

If it doesn't work, you need to be able to fix that real quick, because that is going to be one of the bottlenecks in raising your AI maturity.

[Kalle] (12:18 - 13:18)

I've said it to many places, and I can say it here again, that IT shouldn't be its own box, especially development. It should be under the business, under the profit and loss side, so that it would be close to the people who actually do the business decisions. At the moment, we've always built these R&D businesses and so on, and then they are always separate from the business decisions.

And that leads to this situation where the leadership talks about the business, the business talks about the business, and the IT, as R&D, as wherever the software is being written, talks about how much our uptime is. But at the same time, nobody cares in the business, because they are having a different challenge. They are having a challenge, our competitors are killing us because of X, Y, or Z.

And then the developers hear this, and they're like, well, that's 15 minutes of work now with AI to get those. Some of it is really simple, like, hey, we want the customer to be able to customize their email notifications better. This is one challenge that I have had with one product that I want specific emails from, I don't want all of them, and I want to customize it for myself, not for everybody.

I don't want notification schemas.

[Pinja] (13:18 - 14:12)

No. And this is, especially when we look at these challenges from the perspective of a private equity company, we have somebody who owns multiple other companies, and they need to be able to see, where do we invest next? How do we raise the maturity to a reasonable level?

And that's why we talked about benchmarking. We do not propose that somebody who's in a regulated industry, has a lot of restrictions, has a very specific way of working in a business model, should raise their AI observability to a certain level, as in, for example, I'm taking gaming again, which historically has been in a much higher maturity in DevOps practices, because we also calculated these from a private equity company's point of view, and making a dashboard for the private equity per portfolio company, per these categories for the product development lifecycle, how much would it cost to reach the peers? And does it make sense?

Where do you invest next?

[Kalle] (14:13 - 16:01)

And that created the whole AI business case thinking, like, what are the things that we should be investing in? Like, how do you even calculate the AI business case? And it gets really difficult, because again, it's not revenue generating work that we do in Eficode.

Our job is not to generate revenue for the customer. That's not where we want to be. We want to be fixing things in the customer's organization.

All of these lead to revenue in well-oiled and well-working machines. But the challenge is, you can't say, fixing a DevOps pipeline doesn't make money. That's not where the money comes from.

The money comes from the features that go through that pipeline. So it created this challenge that if we calculated the ROI of a code review, you can easily get a 100x code review number, or you can have a 40x code review number. I think GitHub is 40, and GitHub was like 45, or something like that.

It's the same number on both of them. It's an AI feature that does the same thing. It speeds up your code review.

So if you spend one hour doing code reviews every day, that means that they're trying to make it so that you could do it in 10 minutes. Again, their goal is that you have a multitude of code reviews, you save time. So we can calculate a direct cost per hour on that function and create a code and AI case study.

But then we have the indirect effects that we need to start thinking about. Cost of hiring, cost of training, cost of keeping the business running. We have this cost bucket that we need to figure out.

This is what we could optimize in cost. That's a separate function from the business case that we have. Then you have revenue generating, we can compare revenue per staff.

That's often used. We can take that as a calculation, how much revenue could you be generating by having more time? Theoretically.

We can also take on the other side, how much revenue are we missing? Because it takes time to onboard customers. It takes time to do X, Y, or Z.

How much revenue are we leaving on the table? Because we can't cover X, Y, and Z business cases.

[Pinja] (16:01 - 16:50)

This is where we also were building these roadmaps for the portfolio companies and for the private equities sake as well. Trying to be very reasonable and realistic. How much can you tackle within the next year?

There was this one portfolio company where we were doing a list of initiatives. It was a proposal list. It was the first draft.

I think we ended up with 39 at first. Something like that. Just the ballpark, pretty much.

Is it reasonable for an organization to start tackling 39 different improvement initiatives while running your business and going on business as usual? We chopped that number to, I want to say 13 with the prioritization together with this company. I think one of the numbers that we presented was we would like to have 11 because you have 11 effective months in a year because you need to take the holidays and vacations into account.

[Kalle] (16:50 - 17:52)

We did have one that actually did all of the initiatives in 90 days, if you follow it. They did. They stopped everything.

They made the hard decision. They were like, we're going to AI everything. From the next 90 days, we're just going to AI everything.

They stopped everything. They started doing everything with AI. They just moved everything into an AI driven development workflow.

That was wild to hear. They were really taking it and their numbers are ridiculous. I'm not going to say any of that, but that was really cool stuff to hear.

On the other side, when we are talking about these businesses that are not yet fully on board with AI, they want to look at where we are going, how the other portfolio companies work. We start by looking at, okay, what can we actually do to pull out the correct things? What you want to figure out is how much management focus can we have?

The reality is that 11 is the highest number of initiatives that you can have. Usually, for example, in OKRs, you can only do it so that everybody remembers each OKR in their head. If you have more, it's too many.

It's the same thing in these initiatives. If you can't remember what we were planning, it's too many.

[Pinja] (17:52 - 18:05)

Yeah. This is something from actually improving this organization with the scale, that it's, as we said, it is not just a matter of bringing in the tooling. It is not just a matter of adding better testing, but it's a combination of so many other things.

[Kalle] (18:06 - 19:38)

I think the biggest thing in scaling is how do we actually change humans? Because AI tools haven't been a problem since November 2025. They got good enough at that point.

It's not great. It's not the best. We still have problems with slides and so on.

They put in some better stuff and so on, AI model stuff. I've been talking about that a lot lately. But the thing is, it's good enough to review your code.

It's good enough to give you feedback. It's good enough to make some tests for you. But the challenge becomes, do I know how to make it good enough?

Because if I just tell the AI, generate me tests, it's going to generate me tests, but they're going to be bad. That creates this challenge around the whole thing. How do I actually make sure that the context of my organization and the ideas behind where we are get through to the people?

The next frontier and the biggest challenge that we're seeing in the market is the context management and all of the challenges around it. Everybody who is evolving out of the we are prompting AI in a chat box to a more agentic-driven AI starts to hit this context wall. Where do I actually get the context?

How do I get it? We have MCPs. We can have a bunch of MCPs, but that's just additional integrations.

But the biggest thing is, we hear it from Atlassian, Teamwork Grapht. We heard about it from GitHub. They just released Orbit, their context platform.

These are craft databases. We have had Neo4j for about 20 years now. Getting the data in and working around it is super difficult.

Atlassian says that it works perfectly. It doesn't. It tells me that my old boss is my boss at the moment.

He's not, but again, it tries to guess it. It tries to guess it.

[Pinja] (19:38 - 19:58)

Yeah. There's still a long way to go with creating that context, but that was one of the key messages that Atlassian was promoting in their Team26, for example, a month ago, that context is king. That is what is ruling.

I cannot actually find a disagreeing argument with that. It just needs to work a little bit better, but that's one of the ways to tackle it in the next few years.

[Kalle] (19:58 - 20:23)

And I think the winners are those that can have a company strategy. For us, it's the future of software development. So our job is to make the future of software development at Eficode.

And if we could track that mission against every single line of code our staff makes, that would be cool. And that would be the context. Then we could always follow, are we actually following the strategy when we prompt AI?

Are we actually creating the future of software or are we just doing random code?

[Pinja] (20:23 - 20:40)

That's the maturity, basically. To achieve that maturity is basically all the dimensions, right? You need the context, but in order to have context, you need ownership.

You need a better data quality in place. You need the clarity and strategic intent. So all this needs to be in place before you can actually say that, well, we have the context.

[Kalle] (20:40 - 21:25)

We keep saying data quality and reality is how many of us have seen a Confluence page or a Confluence space that doesn't have an owner? And that means that the data quality can't be in place. And that's number one.

But then the whole thing, how does this whole stack combine together becomes a problem also. And then you need to be able to clarify the strategic intent. How many of us have seen a really nice slide deck of strategy and being like, how does that relate to me at all?

And then you get the OKRs and you're like, these don't actually affect me. How can I affect the revenue of the company? And you're like, how does this affect me as a technological staff member?

There is actually a way to get it to relate to you, but it requires a manager to actually know what your product does. And oftentimes we don't have that context.

[Pinja] (21:25 - 21:56)

No, and this is what we saw with many of these portfolio companies, that there is indeed a gap between these parts of the organization that, well, the engineering or R&D, however you want to call it, is doing their own thing and they're moving fast. And then we have the management, we have the product organization who then thinks that, well, there is this black box called engineering and we have absolutely no idea what they're doing because there is that misunderstanding on what we want to go for and not understanding how this relates to our work at all.

[Kalle] (21:57 - 22:27)

Yeah. I think the big challenge is that technology isn't the broker, it's the people. I think that's the key message that we learned over this.

And the more we work with the customers, the more we learn that adoption strategies need work, center of enablements need work, trust needs work. We still see people are afraid of losing their jobs to AI and that's growing even more with the better AI models. And the reality is we're seeing more people hiring.

Every PE that we worked with or the portfolio company, they were hiring. They were hiring software developers.

[Pinja] (22:28 - 23:15)

That was indeed one of the questions we asked. What is your plan for the next maybe six months? Is that your plan to hire?

Of course, that's going to impact the costs for that portfolio company. And a PE wants to know where the cost structure is heading with those kinds of changes. And having the ability to evaluate where your portfolio of companies is going with and where to invest next is very crucial because there is so much to work with AI at the moment.

And of course, you can start implementing ways of working and tools in a way that somebody who is in a very different peer group than yours or the portfolio company that you manage through a PE is working with. But does it really bring ROI if you're going above and beyond because you can actually manage with less and you might actually hit the ball too with the ROI?

[Kalle] (23:15 - 24:16)

It's been wildly interesting to see this. And we've been doing these assessments more now and it's like the ROI returns even on the direct costs are crazy. I do have to say that when I nowadays listen to AI podcasts that are anti-AI, I'm like, yeah, but there is a lot of ROI in development.

There is so much that you're saying that this doesn't have any value on AI. You clearly haven't actually worked on AI. You're working on some really different stuff than what we are, for example, seeing our customers’.

Because the customers that we are seeing are really, it's so different what they get with AI when it's used correctly. I do have to say it doesn't work on everything. You still need you.

My favorite sentence, which everybody's getting sick and tired of, is who goes to jail? That's the question that I keep asking. And it's never AI.

So it's like asking yourself every time you press a merge request to make you understand that you're going to jail if you just let AI steal all your money.

[Pinja] (24:16 - 25:01)

Yeah. And it is so wildly different. If you implement the same tool and you think you're implementing in the same way for two different companies, you're going to get wildly different outcomes regardless.

So it's important to understand your context. It is important to understand that, yes, you're still needed in the loop. It is still important to understand how you actually benchmark against your peers.

What makes sense to a company in your geographical region or in the industry that you're working with or a size, for example. So to understand that, I think,  this is very, maybe a little bit off the shelf is to say from our perspective, a little bit of a problem, but I think we were able to open up a lot of these people's eyes in these organizations for the management of what is actually happening.

[Kalle] (25:01 - 25:32)

Most of those organizations that we work with said that they will take the initiatives into the roadmap and that the feedback was good and they will use it. Are we continuing in all of them? No.

Are we trying to continue with all of them? Of course. But most of them were happy about the results that we delivered in the form that they can actually take action on them, be it in the internal stuff or external stuff.

That was really cool. And the more we work with customers, the more we hear that they are taking those things into their roadmaps and actually improving the life of a developer.

[Pinja] (25:32 - 26:06)

So this is maybe a couple of words of advice about this conversation here. First of all, just a regular organization, you need to look at the whole picture, not just how we implemented the AI tools, but what is the surrounding product development lifecycle looking like? And if you're working in PE and you're managing a whole bunch of companies, think about how they compare?

Where is the ROI coming from? And what makes sense for each of those companies? Because they're all obviously, they're individual companies.

So that's maybe the advice I'm giving at the end of this. But hey, thank you so much, Kalle, for joining me here today. I think this is all the time we had, but I had a blast again.

[Kalle] (26:07 - 26:09)

Thank you. It was fantastic to join again for a moment.

[Pinja] (26:10 - 26:20)

And thank you everybody for joining us, and we'll see you in the sauna next time. We'll now tell you a little bit about who we are.

[Kalle] (26:20 - 26:46)

Hello, my name is Kalle Sirkesalo. I work as a field CTO in Eficode at the moment. I've been here for over a decade, and I'm at the moment mainly focused on AI-powered tooling, especially in the SDLC pipelines.

I'm building very stable platform engineering platforms and DevSecOps practices. And my biggest job is scaling CI, CD, and SDLC in industrial and regulated industries.

[Pinja] (26:47 - 27:02)

I'm Pinja Kujala. I specialize in agile and portfolio management topics at Eficode. Thanks for tuning in.

We'll catch you next time. And remember, if you like what you hear, please like, rate, and subscribe on your favorite podcast platform. It means the world to us.