Who's really in control? The human-in-the-loop problem

[Dan] (0:03 - 0:08)

If you keep it going, you're just putting more and more defects out into the world that you're going to have to pay for.

[Pinja] (0:14 - 1:20)

Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development. Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna.

Hello and welcome back to the DevOps Sauna. In many of our recent episodes, we have talked about AI. And anyone who has heard about AI cannot escape the term human in the loop.

But if we actually think about what it means to be a human in the loop? What does it mean to add that? What kind of structures does it require from an organization to do this?

This is what we're going to be talking about today. Let's set this up a little bit. We're not going to be talking about AI technically so much, but this is now the backdrop.

Because AI is amplifying a lot of what we're doing today. So if our governance instructions are not in place, AI is going to amplify that again. I'm not here by myself talking about a monologue here, but I'm joined by Ithaca's lead consultant, Dan Khurram.

Hey, Dan, how are you doing?

[Dan] (1:20 - 1:33)

Hello, Pinja. I'm good. It's a very nice morning.

Thank you for having me back. It's been like a year. It's been a year, yes.

Yeah, it's hard work to get an invite to come back here. I have to do a bit better so it's not another year. I have to maybe close that gap.

[Pinja] (1:34 - 2:07)

Let's try that. But we have a very interesting topic today. And Dan, you've been working a lot with our customers lately, especially around AI governance.

But you've also been assessing our customers' full software development lifecycle. So basically from ideation to all the way to that value that they're providing to their customers. And we have seen different ways of organizing their AI governance and how quality is being perceived, how that is being controlled.

So is there anything that has changed in the past couple of years that you've seen now with AI as a backdrop here?

[Dan] (2:08 - 3:30)

Yeah, well, I mean, I think we've all felt something is weird, right? Something not weird, something has changed. Like our whole world seems to have kind of gone into hyperdrive since I guess chat GPT came out.

From what I've seen, it's kind of a big paradox. As you said, I go in, I look at different companies and software organizations, and we think, okay, what is going to be the best structure, governance or non-governance to get what we want out of this software organization, which is usually good software, nice products that people like to use. And the big paradox is there's more people than ever looking at what's coming out of this software and what's going wrong with this software.

And when I say more people than ever, it's essentially the humans in the loop of governing how the organization works, if something goes wrong, who manages an incident, who approves, go or no go to production, and all this stuff. And before AI and all this stuff, we already had that, and we had processes on how to get good software out. And sometimes the software was bad.

We had a bug, we failed when we went to production, but it was a thing. We had processes to stop it, and we kind of accepted it. Now there's just more people than ever, but I don't know if it's effective.

We still have a lot of bugs. We still have security failures and so forth. And it almost feels like AI just amplified all the problems we previously had, and our solution was just throw some more people at it and hope we can keep it under control.

[Pinja] (3:31 - 4:01)

So that's the guardrail for many organizations to add that. So it's kind of a paradox here, because we say that in many organizations, where many organizations are laying off people because of AI is making it more effective and more efficient to produce, especially software. But at the same time, many of the people who are staying, their jobs are actually changing to controlling what is coming out of that.

And if we actually ask in an organization, who owns our AI governance? What do we get as an answer?

[Dan] (4:02 - 4:04)

More often than not, there's a committee.

[Pinja] (4:04 - 4:04)

Oh, nice.

[Dan] (4:07 - 6:14)

It's a committee, and this is the paradox. We used to have owners and accountability when things went wrong, and people would put their hands up because it's not a bad thing to be an owner and accountable. This is, I let this go through into production.

We know there was an incident. Here's why it went through. There was business pressure.

There was customer demand. We just failed a test, and it happens, and we improve off the back of that. And now, what I see when we go, I'm like, hey, just to use the common term in the media, if there was a rogue agent, got access to customer data in a test environment, and something got into production or into training data, who's responsible for that?

Because that's a GDPR breach or some other regulation. And more often than not, people are like, CISO? Maybe.

I don't know. I should ask them. Or I think we'd call the AI board.

And I'm like, who's on that one? Ah, head of legal, head of CTO, CISO. And I'm like, what would they do about it?

They're like, they tell us when it happens. We haven't had a case yet, but we're kind of worried. And I'm like, this doesn't sound like there's so much of a process.

But we just move so fast. We were like, OK, we need to do this. And then let's have a guardrail or a backstop.

More people will just check it. And I'm sure it'll be fine. And thankfully, not too many things have gone wrong in most organizations.

But then I also look at some of the data coming out of respected research. I think there was an IBM study that asked many companies and many boards, have you got a governance process in place? And I guess 90% of them said, yes, we've got a governance process in place.

Then they asked those 90% who's responsible and accountable. And I think 15% could name an owner. So to me, I took that away as, I think what you've done is put in a committee, but there's no single accountable person.

And I don't know how effective that is. I think we've seen from decades of scientific research, when you put groups in charge of making a singular decision, it doesn't turn out well. There's the bystander effect.

There's diffusion of responsibility. When there's so many people looking at it, you're like, ah, someone else will take care of it. And then you might have an incident.

So yeah, it's really paradoxical and slightly concerning.

[Pinja] (6:15 - 7:11)

This is, I'm reminded of this concept of a design by committee that has been one of the so-called jokes in the product development world in the past couple of decades. When you actually assign a committee, there needs to be some kind of consensus. Who's actually the deciding person or party in this matter.

But it is true that since we're moving at a faster pace, it is paradoxical that we need to slow down because of humans. This has been accepted in the past couple of years, obviously. But if we add this as a safety mechanism, this might be actually slowing us down.

So this is basically, if we think about the loops, when we say human in the loop, can we actually call it a loop? Is it coming back as some kind of feedback? In lean and in agile development, we've always said that we need the feedback loop.

We need to be able to do the plan, do, check, adjust the loop in that way. So is this actually now bringing a loop into the play or is it something else?

[Dan] (7:11 - 8:44)

Oh my God, loops, loops, humans and loops. I'm going to try not to go off on too much of a tangent, but to have an effective feedback loop in this process, it means you have to make a decision and measure against it. And then if the measurement shows that something succeeded or failed, it means you changed something.

And all those points I start to see are not happening. First, we have the human side of things to make the decision. A committee is not going to make a decision and it's not going to make a fast one, at least.

An accountable, responsible person whose incentives are aligned to that decision-making process will make a decision. And then once I've made the decision, how do I know it was right or wrong? Can I reverse that decision?

And then once that's gone through, then I'll manage the feedback to that and say, oh, maybe I should change my decision. I should change the process or the data that took to get the decision. And to me, we've misunderstood what the loop is in general.

And even the term human in the loop, for me, I don't think it was supposed to be this way. I don't think we're using it, you know, the old meme, I don't think it means what you think it means, kind of thing. Humans in the loop came from the AI companies, you know, and not throwing shade at any of the companies.

We use the products every day. We know they're useful, but there's a danger in taking what I see as a product improvement loop, a product loop, and using that as the control loop for a giant organization that has SLAs and customers and sometimes regulations. I think we've mixed it up a little bit.

[Pinja] (8:44 - 8:48)

Is it bringing some kind of false sense of security in a way to an organization?

[Dan] (8:49 - 9:41)

Absolutely, right? It's a sense of security. I don't know how it's false.

I think we all feel safer when there is someone responsible. We think, paradoxically, make more people responsible and make us more secure, but then the whole paradigm has shifted because security is being affected by machines and more responsibility has gone to humans. I can't oversee all these machines and keep everyone safe.

We've put in a sense of security for the people, and we've put in actual security. That, to me, is kind of weird. I use this term, would you use a customer satisfaction survey as a fire alarm?

You wouldn't. And that's essentially what we've done, right? We've taken people's opinions as a way to signal something serious is happening because we cannot keep up with the amount of data that's coming in.

[Pinja] (9:41 - 10:19)

That has been, even if we go to the times before we had AI tools, like we do nowadays, and customer satisfaction surveys and how we interview our customers and the users of our products, many organizations, obviously, have been using this data and as a signal, hey, this is where we need to go in the market. But is it always the best way of shifting as soon as you actually figure out there's one person in the CSAT who says, well, I don't like this. Do you always divert your whole product strategy to something else in the same way?

Do you use a CSAT as a fire alarm?

[Dan] (10:20 - 11:41)

Yeah, you wouldn't use a CSAT as a fire alarm. You wouldn't use the CSAT to fix SLA and bugs issues, right? We're completely measuring the wrong thing.

And I use CSAT as an example because that is opinion-based, right? And the more opinions you throw into a machine, it's going to make it challenging. And I'm old enough to have worked in places that didn't use software, everything, in factories, right?

And I always kind of like to use this example. If your factory is producing a chocolate bar and you're producing 100 chocolate bars a minute, people can't check every single chocolate bar, right? You have a machine to check what the machine's doing.

And then you have people who are, I don't know, your taste testers of your chocolate bar, tasting the person who'd had the recipe for the chocolate bar. Now I'm getting hungry, right? So you use the appropriate mechanism to check the appropriate system process and so forth.

And kind of bringing this back to AI, we've kind of gone wrong to me because I don't see humans in the loop being an effective and appropriate control mechanism to govern what we're outputting. And I'd almost like us to roll back a little bit to how we used to govern software development life cycles or any kind of software or products that we're shipping out there. Get the humans to decide on what the humans can do and check and get the machines to decide what they can check to the kind of precision of accuracy that they can do it.

[Pinja] (11:41 - 12:25)

And I think if we think about what companies are now required to do, so there is, you mentioned briefly in the beginning, let's say EU legislation, which is one of the things, we're recording this in Finland and our company is based in Europe. We have some operations also in the US, so we cater to customers all around the world. But this is one big piece of legislation if we talk about the EU AI Act and how that's been phrased when it comes to high-risk AI systems.

To oversee those high-risk AI systems, the EU AI Act says that we need to have humans to be able to oversee these systems. I'm not going to start dissecting the EU AI Act in detail. I'm not in no position to do that.

But at the same time, is that phrasing, because that is so common what we say in how we regulate AI nowadays.

[Dan] (12:26 - 13:44)

Isn't that not lawyer speak for humans in the loop? Exactly, exactly. And this is where, you know, when we work with people, we work with developers, engineers, managers, CTOs, CISOs.

I'm not there to give legal advice. I'm not there to give financial advice. We're there to say, how can you be effective as an organization to what you're trying to achieve?

And to me, let's sidetrack. Most companies don't have that kind of product that that particular clause of the EU AI is talking about. It's not a high-risk system.

And I get the point that for really high-risk regulated systems, you do need accountability. That particular clause is about the producers of the AI mechanism needing to build into the mechanism and oversight for that. And as I said, I'm not giving legal advice.

That's for high-risk systems. But what can we take from that? I take it that it's kind of vague, but even the regulations have said, well, put a human in the loop and you should be fine, right?

So it's okay for the people who create AI to just design it in a way that they say a human can oversee this effectively. Can they do it well? Can they make a decision out of it?

I don't know. No one's measured it. No one's asked to measure it.

Because in my opinion, what the AI Act has kind of taken inspiration from is from the AI producers, the people who make it and say, yeah, just chuck a human in there. You'll be fine. It'll be overseeable.

[Pinja] (13:45 - 13:52)

But it's not defined. What does it mean to actually put the human in the loop? Does it mean it doesn't actually say that you need a committee?

[Dan] (13:52 - 14:23)

No, I mean, if we do a pre-mortem on what could go wrong, well, the law said I need to design my AI agent to have someone to oversee it. The law said it. So that's how I created my tool.

And here's a nice dashboard for someone who could oversee it and regulators are fine. And then we suddenly have a catastrophic fridge hacking incident or something like that. Now, in Europe, the shop down the street has no frozen pizzas, which I'm finding pretty catastrophic.

[Pinja] (14:23 - 14:24)

Devastating.

[Dan] (14:24 - 15:27)

No one's going to get a fine. The regulator can't govern that. No one's actually going to get a fine.

The people who suffer are me without my frozen pizzas, right? And to me, this is not an effective control mechanism. Seriously, but no one's going to be happy in that case.

The actual retailer is going to have to throw away stuff. They're going to get fines for something. Someone's going to go to jail.

And to me, the governance, if I replace the AI Act with just you as a CTO or CSO, you're supposed to oversee something well. You're not just supposed to oversee, you're supposed to do it effectively. And if you can't do it effectively, someone's failed in that.

So whilst we have to adhere to the regulations, the AI Act's a big one. Not every company making software has that relevant, unless it's about human observation and kind of HR telemetry. Cybersecurity Act is maybe more relevant to most companies when they're shipping features maybe coded by someone who's not a human.

And GDPR, right? We have to adhere to those. But again, the question I would say is, is it effective?

And are we looking in the wrong place for inspiration on how to control them effectively?

[Pinja] (15:27 - 15:44)

And if we move on to the fix of this, we're not giving legal advice here. But what we have seen works in terms of how to actually fix this, how to make it more effective. And I guess based on what I know about the conversations with you, Dan, we're not proposing to have more people in the loop.

[Dan] (15:44 - 15:45)

No, no.

[Pinja] (15:45 - 15:46)

That wouldn't hold up.

[Dan] (15:46 - 16:46)

Yeah, I'm sounding very anti-human, unfortunately. I'm not saying more humans. I'm not saying less humans.

I'm saying effective, accountable people. And before I tell you all what I think is effective, I'm going to tell you what I usually see as what people change, right? Usually if something has gone wrong, we again, we roll back to, well, whose advice and playbooks can we look at?

We look at the regulators. We look at the AI companies, Anthropic. I use Anthropic tools, so I'm not throwing shade at them.

I'm just saying they've got a playbook. And what they recommend in the playbook is humans in the loop, right? Because it is a good incentive for them to have more humans in the loop, because more humans mean more training for their models and more usage of their models.

And it makes sense. That's how they are governed. So that's what they would do in their playbook.

But, you know, what do we say? Okay, if there's more humans in the loop, take that thought further. What's the second order of that?

Ah, you need awareness training. Train the humans and that'll fix it, right? There's no evidence.

[Pinja] (16:46 - 16:51)

Can it be done in isolation? If we just do that, would that fix this whole problem?

[Dan] (16:51 - 16:52)

It will not.

[Pinja] (16:52 - 16:53)

Oh, too bad, too bad.

[Dan] (16:54 - 17:43)

We know it won't, because like, almost like training the human more, what are you training them on? It's the old paradigm of just like, I'm being trained on how to do something in this loop. Or then we say, okay, well, now the people have been trained.

Let's use AI to write summaries for them to review. So now we're already at a scale problem, because now we know humans, even with their training, can't review everything. So now I need a tool to do it.

We put committees in place. We have declarations, you know, we have technical guardrails. Trailers on submitted code saying this was AI and not AI.

And to me, I have not seen any of these be effective. Have they made the product better? Have they stopped bugs?

Have they stopped incidents? To me, they haven't. And that's, again, because we're kind of stuck in this paradigm of humans in the loop will fix everything.

And to me, it won't. So I'll actually answer your question now.

[Pinja] (17:46 - 17:57)

So it won't work. Okay, we've now discovered what doesn't work. We've covered that.

So if we actually think about what we should be doing? And I think my guess is that we need to go back to essentials.

[Dan] (17:57 - 18:26)

We need to, yeah, just fall back to the last stage that it did work. You know, go back to the basics, roll back. And I'm actually segueing into one of my, a few of my suggestions is, actually, number one is to give accountable humans.

If we consider, if you're going to put a human in there, they need to be accountable. And I don't know, what's the word? They need to be, what's the word when you give someone power?

Made responsible.

[Pinja] (18:26 - 18:27)

Not empowered?

[Dan] (18:27 - 18:44)

Empowered. Now we're really lifestyle gurus, right? You need people who are empowered in your process.

Call it a loop if you want. And my test for that is, can this person switch everything off? Are they allowed to do that?

Because if they can't, they're not empowered.

[Pinja] (18:45 - 19:04)

No, they don't. They don't have the ability to impact anything on paper. Because I've seen this in many, many different cases.

It doesn't even need to be the actual human in the loop, the AI accountable person. But even before, if you put somebody in charge, and they don't have the ability to act. But on paper, we have somebody, right?

So it's not, it doesn't work.

[Dan] (19:05 - 19:47)

We have it on paper. And it won't work unless you have the ability to act. And again, many, many moons ago, before my life in software, and you look at some of the methods of the lean production system in factories, people were empowered to either press the stop button, put up a red flag and say, Hey, I've got a defect, stop everything.

If you keep it going, you're just putting more and more defects out into the world that you're going to have to pay for eventually and fix later. Take that back to software. Like, can everyone be responsible for that?

You can't, because we've literally built it into our software development lifecycle processes. You can't get things into production without a check. And imagine the poor junior developer who says, I'm going to switch production off.

[Pinja] (19:47 - 20:01)

No, but you need to have an escalation process for this. You need to have a procedure for that junior to be able to say, Hold up, there's something, I think this is critical. We should not proceed, but it should not be on them to make that decision.

[Dan] (20:01 - 20:45)

They're not accountable, but they should be empowered in a way, right? Yeah. And you have to ask yourself, would you rather stop production and annoy some people and have to put an error page?

I'm talking about a website now. And then say, Hey, sorry, we're unavailable. We're fixing it.

Or would you rather have maybe dangerous code exposed to the internet that anyone can access and you leave it out there while you think of how to escalate an incident process? And, you know, now we're getting to the stage, if you leave something exposed to 60 minutes on the internet, someone's going to find it and you're going to be in security trouble, right? So, or in all those ways.

So kind of wrapping that back, accountability, empowerment, have as many humans as you want, really. I'm not against them. Make sure they're doing something, right?

They're not just sitting on a committee.

[Pinja] (20:45 - 21:07)

Basically, this is the same old as we have discussed, not like for decades now, like give people the power to do that. If we think about the PDLC loop in a way, we need to always plan, do and check. Can we actually have that somehow in this human in the loop way that we have that from check to adjust somehow covered?

[Dan] (21:07 - 23:38)

We used to have it. Again, this is going pre-AI. We had it, right?

Plan, do. Check. That's kind of in your PDLC loop.

The check was, we used to have metrics and we used to have kind of staged launches, canary launches. We used to roll soft rollouts and stuff. So we had that, but I think maybe, you know, you need to do that at machine AI scale now.

So that's why I would say some checks, let the machine do and trust the machine. If you have the accountable person behind that, it's fine. Have the check.

I don't think the check is the problem. I think the check's only the problem if you have too many humans and not enough machines. And once you find the equilibrium there, it's fine.

You can have mutation testing in your software. You can have really smart ways to do test quality, test pen testing and so forth to make everything check secure at high quality. I think the problem is the act part of that plan, do, check, act or adjust the loop that you had.

Before we used to have, it used to be quite cheap to undo a mistake. You could roll back, right? You could roll back and you could be like, what went wrong and so forth.

And you just say, sorry, you couldn't say. So there were ways to act, but now things move so fast. You can't roll back.

You have so many branches. You have so many things constantly going to be committed. And I know we're going in between the analogies of software and not.

So I thank the audience for keeping up with this DevOps. So just when we think of the act, find ways to act before you have to. Have ways to roll back, have ways to verify, ways to complete the loop of feedback.

And really what we're saying is we've gone back to the basics and this is probably the best way to govern at the source of the code. And then you can, once you can govern that and prove that when you have to do real governance, which is audits, approvals, compliance, and risk, you just have fewer mistakes to have to report. And we're trying to solve the problem at the source, right?

And this is where I'll take one more second to say, if we took humans in the loop as inspiration from AI, but I think we took the wrong inspiration. We should have taken how the AI, most of the models work, or have worked with the reinforcement learning through human feedback, right? We know that term.

We know if you make the model react and improve every time you talk to it, it gets better for you as a product. Why don't we take reinforcement learning through human feedback for our processes and actually apply them? Because people know.

People know what can make it better, but now we've got too many people to make the decisions and we can't actually apply that learning.

[Pinja] (23:38 - 24:27)

No, and this is basically software development and DevOps 101 to what we should have in place to reduce that friction and to be able to actually undo those mistakes easily, fast. And for example, you mentioned if we use feature branches, if this is, again, going back to the very basics, branching strategy. And how do you actually use that feature branch that is not open until the end of times, but you commit as fast as you can, whenever that makes sense.

Once you've actually hit those same goals that we've had before, quality checks out, you commit, and then you do the next one. Because then we can actually roll back to previous versions and we can undo the changes, even now with AI. Because now, as you say, we're getting to a faster pace, but the basics need to be 100% proof they're in the background.

[Dan] (24:28 - 25:09)

Yeah. Amplifying good basics is a great thing to take away. And I think everyone will be so much happier.

And whenever we go into people and we get asked, can you come review our AI maturity across our SDLC? I know I'm not giving you exactly, but I would say like nine out of 10 times, we go in and say, fix the basics, please. You moved a bit too fast on AI adoption, use AI to fix your basics.

You can use it to go back and do the things like you said, what's your branching strategy, right? Let's take a moment to breathe, fix the basics, and then go forwards. Governance is so much better.

Compliance is so much better. Everyone's happier, fewer incidents, and we're effective. So that's the big takeaway.

[Pinja] (25:09 - 25:20)

One thing we want to mention about it. So you mentioned that let's use humans in a more effective way. And if we fix the ratio, how do we treat that human seat?

And this loop, if you want to still call it a loop?

[Dan] (25:21 - 25:22)

Do we want to call it a loop?

[Pinja] (25:24 - 25:32)

I don't know. After a debate, it is a known term in the industries. Let's keep to that for now.

[Dan] (25:32 - 26:38)

Um, it's one of many instruments, right? The human is there as one instrument with judgment and appreciation for the experience that everyone is going through. So scope it to that.

The human can check what the other human meant. Intent is a word. Maybe we talked about if you make everything reversible, the human is there for the irreversible things.

Like I know we can never roll back from this. I'll make that decision and go forward. And that's, that's kind of really in the world that we live in.

I think if we said I'm responsible and I'll be accountable for this, like people appreciate that organizations, you know, this is about not just SDLC and software pipeline and CI/CD design. This is about people in organization design. And most people would much rather work with or for someone or have someone on their team who says this was irreversible.

I made the decision. Here's why, let's go on and let's complete the loop or, you know, get the feedback from it and build from where we are.

[Pinja] (26:38 - 27:07)

Yeah. And I think if we summarize this whole conversation to one, like just going back to the basics and yes, as I said in the beginning, AI is amplifying, it is amplifying the good, but it's also amplifying the bad. So there's both sides of this.

Like if the basics are not there, if they're not in place, AI is going to take you faster, but to the wrong direction and completely wrong direction from where it is really expensive and really slow to get back on the track to the last working version.

[Dan] (27:07 - 27:34)

Yeah. Go back to basics and apply it to your situation. You know, the regulations are vague for a reason.

They need to be all-encompassing. The playbooks are all biased to vendors. That's fair.

We do it. It's the same. So go back to basics, see how it would apply.

I don't think it's as hard as it might seem and just go from there and yeah, give people power again. I guess that's maybe a big takeaway, people power.

[Pinja] (27:36 - 27:41)

I think on that note, I think we're going to end this conversation today. Dan, thank you so much for turning into this.

[Dan] (27:41 - 27:42)

Thanks for having me. See you.

[Pinja] (27:43 - 27:46)

And thank you everybody for tuning in and we'll see you in the sauna next time.

[Dan] (27:50 - 28:03)

Hey, I'm Dan Khurram. I'm a lead consultant here at Eficode. I help our customers assess and mature their software development lifecycle processes and I've been working a lot with the AI governance theme.

[Pinja] (28:03 - 28:08)

I'm Pinja Kujala. I specialize in agile and portfolio management topics at Eficode.