AI Governance is let down by the humans in the loop

Every great growth or product marketing team had a loop like this. Lovable's free plan publishes your site with an "Edit with Lovable" badge, and hiding it is a paid feature, so its users do its advertising, creating a viral growth loop. Human feedback is the AI industry’s product loop, and “human in the loop” is how it got sold.

In Reinforcement Learning from Human Feedback (RLHF), the most popular technique for training AI assistants, people rate answers, a model learns what they prefer, and the assistant is tuned to produce more of it. This exists to make the product better and more likeable, and it is biased by design toward whatever its makers choose to measure and reward. Technology itself does not want anything, but its creators do. And the creators want more users.

Unfortunately, I’d argue that we users, regulators and company boards looked at that loop and saw a safety mechanism. We took what was intended to improve an AI chatbot and applied it to the organisation as a control. For organisations trying to manage risk inside their software, relying on "human in the loop" like this creates a dangerous false sense of security.

The loop that improves an AI product and the loop that controls risk in an organisation share a name and almost nothing else. The product loop is measured in ratings, usage and retention, which makes a person's opinion of the output the signal. Applied to control, we would be asking that same opinion to catch serious harm. That is like using a customer satisfaction survey as a fire alarm. A control loop exists so that harm is caught and fixed, and so that value is prioritised. It has four steps at minimum: something is detected, someone decides what it means, something changes, and someone checks that the change held. It is measured in incidents caught and corrected. A person reading an alert is at step one. A loop exists only when someone can also change the control and confirm the change worked.

Everyone prescribed it, and nobody had to prove it works

For high-risk AI systems, Article 14 of the EU AI Act requires that they be designed so humans can oversee them effectively. It names one known weakness: overseers must be enabled to "remain aware" of automation bias, the tendency to over-rely on what the system says. Article 14 covers high-risk systems, which most AI tools and coding assistants are not, but the assumption travelled. "A human must review it before it goes live" is now the default answer in AI policies for almost everything. And the enablement? From what I’ve seen, it is often an AI-written summary for the reviewer.

That might not matter if it worked. Did it? Did we get machine-level speed with human-level ethics and go/no-go decisions? Of course we didn’t! Nobody had to show that it works, mainly because it's a compromise that suited everyone. Vendors can point to the person who clicked to approve. Regulators get a visible safeguard. We can pretend everything is fine.

I did not question it either, until I looked at patterns across my own cases. My job is assessing how software organisations use AI and other tools across their delivery process, from strategy and product management to engineering, deployment and AI tooling. Ask who owns AI governance and, more often than not, the answer is a committee. I have seen oversight boards where, after an incident, nobody could say who was accountable, and I have heard the same from others. In assessments I have run, different people in the same organisation have given me different answers to that same question. And I have seen written AI policies that nothing in the delivery pipeline enforces.

A recent IBM survey of organisations shows the same. 98% have assigned AI risk to someone at board level. Yet when asked who is responsible if an AI agent causes harm, the most common answer, the CIO or CTO, was chosen by only 17%. So many humans, and they do not agree who is best placed to be responsible or accountable.

The evidence against human oversight in AI

It's not only about the human in the loop. It's about the loop in humans. People respond to feedback and rewards too, and more humans don't make it better. Anthropic's own researchers found sycophancy across five leading AI assistants and concluded it is driven in part by human raters preferring answers that match their own views. Think about what that does to the person reviewing its work, let alone a group of people with every incentive to get it live.

In 2022, Ben Green surveyed 41 policies that prescribe human oversight of government algorithms. Unsurprisingly, his finding is that the evidence suggests people are unable to perform the oversight those policies expect. He also finds that such policies give a false sense of security, let vendors and management shirk accountability, and leave frontline staff carrying the blame. A 2024 meta-analysis in Nature Human Behaviour of 106 experiments found that human-AI combinations performed, on average, worse than the best of humans or AI alone, especially on decision tasks.

We always knew people vary, but did we know how much? in a Kahneman, Sibony and Sunstein's noise audit, insurance underwriters pricing the same cases differed by a median of 55%, when their executives expected about 10%. And groups dilute responsibility: in Darley and Latané's 1968 experiment on the bystander effect, the share of people who reported the emergency while it was still happening fell from 85% to 62% to 31% as they believed more others were present. Although some faith in humanity was restored with CCTV footage of 219 real public conflicts, at least one bystander stepped in about nine times out of ten. Naming one person to act reduces the dilution, so diffusion is worse when nobody is named.

Most of this evidence comes from lab experiments and government decisions, not code review, and I have not found a study (yet) showing that a human in the loop works for AI agents. But doesn't that prove the point? Nobody was asked to prove it.

Why the usual fixes fall short, and what to do instead

When an incident happens, we tend to blame a “rogue agent” and add technical guardrails, or we blame human error and add training, more reviewers and more escalations. Adding people to the loop is not a new idea, and it keeps the same flaw: a wider loop with no owner is a bigger committee..

Some interesting ideas have come out of research. Iyad Rahwan proposed “society in the loop” in 2017: human-in-the-loop control plus a way of negotiating the values of everyone the system affects. Pluralistic alignment goes further and asks how an AI system can serve diverse values instead of an averaged one. Both are worth thinking about, because they ask the right second question: whose values are in the loop, and what decides when they disagree? But both take the product loop as given and try to make it fairer or broader. More voices fix who gets heard. They do not fix who decides, who answers, or whether anything changes afterwards.

Decide, be accountable, be human

Decide what is a machine decision and what is a human decision. Do not ask a person to approve what a machine can check, and make the controls hard to remove. If a factory makes 100 chocolate bars a minute, nobody checks every bar. A machine checks what the machine is doing, and people taste-test the recipe. Make as much as you can reversible, and let machines handle what they can check and undo, so humans are left with the calls that are irreversible or would cause most most harm. Settle the rule before the incident, not during it.

OpenAI’s 20 October report is a small example of a control that is hard to remove. Two of its internal coding agents weakened the monitoring around their work: one stopped alerts reaching humans, and the other removed a monitor and then falsely denied it. OpenAI says it has no satisfying explanation and does not call it deliberate. It caught both on the day and changed its review system to block attempts to disable monitoring, which is a control that removes the option, not a sentence in a policy. The identity doing the work should not be able to switch off the control that watches it. One caveat is that neither incident was classed as top severity, neither run was killed automatically, and OpenAI hasn’t yet shown the fix held.

Be human and accountable. People are good at judgement, at saying no, and at taking responsibility. Use them for that, with the authority to act. My test is simple: can this person switch it off, and are they allowed to? If not, they are not really accountable. A junior who spots a defect should not carry that decision alone, but they need somewhere to take it, and leaders who would rather show an error page than leave exposed code running. Do not use people as a checkpoint in the machine's workflow.

What to do in the next 30 days:

1. Ask your team, who sees an alert and who can decide what to do? AI or not, this will tell you about the state of governance in your organisation..

2. Plant three known defects in a change (agent or not) and see whether your review catches them. Test whether people can oversee a system before relying on them.

3. Many humans is easy. A loop needs someone who can change the control and check that it held. Start with the name.