The AI mirror: How agents expose your engineering maturity

00:00 - 00:27

All companies want to install AI agents in 2026. And while we think AI agents are great, sometimes they are not, especially when you install them in the wrong engineering system. Because AI is really good at one thing people don't expect, which is exposing your flaws, especially your broken engineering. So in this video, I'm going to show you how that rolled out when we installed it to a company, how AI agents ended up exposing their broken engineering, what bottlenecks came out, and how we ended up solving these bottlenecks.

00:27 - 00:44

Before we jump in, my name is Serop—very similar to maple syrup. I'm a Lead AI Consultant at Eficode. I've been working in software for over a decade now. Originally started with Machine Learning and Data Science background, and slowly transitioned into MLOps and DevOps. And now I'm helping companies and organizations adopt AI effectively.

00:44 - 01:25

Now, if you've been following AI, you probably have seen that it is really, really powerful on greenfield environments. So, let's say you tell AI to build you a dashboard or an app, you can use applications like Lovable, Bolt, or Spark, and it can give you a working prototype within 5 minutes, which is insane. And I bet if the management see this, they will start firing developers—which we don't want. Now, AI works on greenfield for one reason only: because there is no dependency, there's no legacy code, and it's just a clear blank page. And AI is really good at filling out blank pages. Now, if you try to take this concept and install it into your company, you are going to hit the first bottleneck of real companies.

01:25 - 02:14

Which is the following: Real companies are not empty canvases. Real companies have Stefans. Stefans—which is a senior engineer working for 5 years on the project, usually they have a long beard, a ponytail, or both—and they know the ins and outs of every repository, of every code that the company has written. And throughout the years, because they are very heavy involved in every project, they become the real documentation of the project. And the teams are so dependent on Stefans that they cannot trust to deploy things whenever he's off. If they leave the company, the production incidents start to happen. If any bug happened, everybody goes to Stefan and ask them. And now the real issue is that AI cannot reach Stefan's brain yet. And this become the first bottleneck that autonomous AI hits once it starts seeing our real repositories and legacy code.

02:14 - 02:49

Now, I always love to ask this question, especially if you're watching this video. If you have rolled agentic AI—let's say Claude Code, Codex, GitHub Copilot, etc.—in your company, I want you to ask yourself: Have you seen a company-level productivity boost from AI? I know that every single developer has witnessed 5 or 10x productivity improvement because now we are able to generate code using AI. But when we look at the company as a whole, usually we haven't seen this productivity surface up on a company level. And in this video, I'm going to show you why.

02:49 - 03:02

And for this video, I'm going to take a company we worked with with about 100 engineers. They rolled out GitHub Copilot to all their teams, agentic workflows were encouraged, and their goal was to achieve 30% faster time-to-production.

03:02 - 03:54

Now, 6 months later, they looked into their metrics. The coding velocity was up, people were writing code faster, which was not so surprising. Their lead time for changes initially increased, which meant that the time it takes from a developer starting to work on a ticket until that ticket is merged in production is now taking longer. But then, after it increased, it became flat, so it didn't even decrease. When they looked at the deployment frequency, it was initially flat and then it increased, so people were deploying more into production, which was a good sign. However, when they looked into the change failure rate, they noticed that initially it was flat, and then it was up. So we were making more errors in the production. Finally, when they looked into the mean time to recovery, they noticed that it was down, which meant that they were solving issues quicker. So when a bug shows up in production, they were handling it much faster. So these metrics were surprising initially, until we looked under the hood.

03:54 - 04:25

And if you look under the hood, you notice that whenever AI accelerates the coding, the first bottleneck that appears and stops the whole cycle from witnessing this improvement is the code review. This becomes the new bottleneck of the team, and the pull requests start queuing, and that explains the increase in the lead time for changes. So, the team here was a little bit overwhelmed, and they decided to try to solve it. And at some point, they managed to unplug this new bottleneck over here.

04:25 - 05:20

But what ended up happening is that when they did this, more errors in production started to happen, and this is where the change failure rate started to increase. And when we asked the team, "Hey, how did you manage to unplug the pull requests bottleneck that was happening?", and it turned out that they started doing more shallow pull requests. They were just overwhelmed. The sheer number of pull requests was just so large that they started skimming through the pull requests and having less structured ways of doing it. Not only this, but they were using AI to run tests. However, these tests were only covering the code that it wrote, and it wasn't connected to some sort of a business use case or edge cases that was predefined previously. And when you mix very shallow coverage testing with not thoroughly done code reviews with a code that a developer didn't write themselves, but the AI wrote, this has led to situation where production errors started to increase.

05:20 - 05:27

And this gave me the analogy where we're not going faster, but we're creating and fixing at the same speed—just like a dog following its tail.

05:27 - 06:44

So, when we look into the gaps to understand what was the issue behind those bottlenecks and problems, we noticed that the first issue was the AI itself. The AI was not connected to internal docs, it was working without context, and of course, just as any AI model, when it was given too much context, it started hallucinating. So there is some limitation from the AI models themselves. The other issue is generated from the requirements. The agents were generating fast, but when humans were writing vague or incomplete requirements, the AI agents are now shipping wrong things much, much faster. The same thing from the testing. AI is really good at testing its own code, but usually it writes unit tests that make sure that the code generated is covered well, but if it's not connected to higher context or understanding the business use cases, it will not write tests for some edge cases. And in many companies we worked with, there was always these flaky tests that usually fail without any reason. So, whenever these tests were failing, the developers thought that it was just one of these flaky tests and it's not a big issue. And if you have this problem, eventually some of real problems and real issues end up leaking into production. And finally, the code review became the new bottleneck of the company, and senior developers became a full-time AI babysitters, and they were very unhappy with that.

06:44 - 08:24

And I notice people in general complain about many different things, and also when AI start to fail us at our work, we start to complain and blame AI about different things. And as I showed you, AI had some blame in the whole process. However, sometimes it was giving the wrong blame. For example, when we blame AI that it was building the wrong feature, what was actually broken in the company was that the tickets were already vague. And when we interviewed the developers, some of them were guessing as well, and they told me that they had to rewrite the requirements multiple times even without AI interference, and that was way before they implemented AI. When the company was complaining to us that the AI-generated code was breaking into production, what we noticed that the AI was writing tests, but it was not measuring business-critical edge cases. All of these tests were not mentioned in the requirements. When the company complained about pull requests are now piling up and reviewers couldn't keep up, we noticed that the AI was not writing well-tested code, otherwise they would have let the code pass once everything is green. So the humans had to check for every single pull request. Also, the company, after they implemented AI, stopped doing proper prioritization. They thought AI is the one solution to all their problems, and automating and working on all the features that they can, just because they can—not because they should. And finally, when people were saying the AI doesn't understand our system, the real problem is that the architecture is living in Stefan's head, and not in a documentation an agent can read, for example on the repository. And don't let me get started with how much outdated data is there—for this company it was around 40%. So even if we connect the AI to their Confluence, most of this data was outdated anyway.

08:24 - 08:31

So, for me, all of these problems are not an AI problem, but an engineering problem. And the fix is quite easy.

08:31 - 09:29

So the first fix that you can do is to have better inputs. And by that, I mean you work on the requirements to make them better. And you can also get the agent involved in the requirements writing criteria, add the edge cases, add the testing directly in the requirements, and then let the autonomous AI complete the build. And as a rule of thumb, if your engineers or whoever's working on implementing the tickets are already complaining about the quality of the requirements, then that's already a signal that you're not ready to add AI just yet. Try to fix these first, and then add AI on top. The second thing that you can fix, which is adding a feature break. If you notice that your rollback is up by 20%, add a new gate for new features and consider only customer feedback on what you should develop. The goal here is to ship less, remove some of the work from your developers, but ship right and add features that really matters. Just because you can use AI to implement things faster doesn't mean that you should.

09:29 - 10:52

The second fix is more focus on the AI agents themselves. You can improve the quality of the AI agent output by wiring in the internal knowledge, so you write down inside the repository what dependencies this repository has, how many connections, where does it get the API keys, etc. So all these informations that usually end up making the AI make the wrong decisions or hallucinating, etc. If you notice the AI is writing code that is not similar to your way of working, you can add instructions in the repositories, make them readable to the AI to guardrail the way it works, so it produce good quality that it is very similar to your engineers. The third fix that you can do is to close the feedback loop. AI is autonomously good at changing one line in one file, and then forgetting that this file is being called somewhere else. So make sure to give your agents testing tools—like simple syntax testing, etc.—to make sure that before it even commits any code, it can check for errors that it just generated. And of course, make sure your CI/CD is including all the testing, so people spend less time on the pull request review and they can trust that whatever is written is actually correct. And as a cherry on top, you can break the silos by adding shared playbooks, weekly demos, and one channel for mentioning what has worked for this particular team.

10:52 - 11:39

So, if you start implementing these fixes, you notice that maybe AI start to work better now. So, if you want to see if AI is working on a company level and helping you increase the productivity of your developers, here are some metrics that you can follow. Most people look at metrics like line of code generated by AI, pull requests opened by AI, and the percentage of developers using Copilot, etc. And I believe these metrics only show you one part of the picture. If you want to see the full picture, you should also look at the lines that were shipped without rollback, how many pull requests were merged without human rework, and how the change failure rate progressed after you installed AI into the company. This would give you an understanding of the quality that the AI agents and the autonomous AI has introduced to your company.

11:39 - 12:10

So, if you would take one key takeaway from this video is that AI is a mirror. If you do not like what you see, you need to fix the engineering behind it. Now, I would love to hear it from you: What is the most common challenge that you've been having in your company? Make sure to write it in the comments. And if you would like us to have a look into your software development lifecycle and see what are the common bottlenecks that you have in your company, make sure to book free discovery call. You can find the link in the description, and we'll happily help you. Thank you so much for watching, and we see you in the next video.