AI's 'human in the loop': when oversight is only for show

Whenever doubts arise about the risks of AI systems, organizations' standard response is to invoke the concept of the "human in the loop," a governance phrase so often repeated that it has become almost automatic. Yet several AI security and governance experts warn that many of these supposed controls are little more than an empty stamp of approval: they create the feeling of oversight without offering any real ability to intervene.

Doug Shepherd, head of offensive security at internet services provider Cloudflare, explains that for a human-in-the-loop system to genuinely work, the person supervising the AI needs enough knowledge and context to act in the machine's place, plus the authority to override its decision. Shepherd, who spoke at the recent CIO 100 Awards and Conference held in Frisco, Texas, said many such promises give IT leaders a false sense of security, because the underlying process does not work as advertised. In his words, "if your human in the loop can flag something but can't actually stop it, that's not a human in the loop, that's a human adjacent to the loop" — a form of "performative governance" that helps win political approval for a project but does not address the real risk.

Shepherd encouraged attendees to embrace AI and focus on projects that drive its adoption and impact, while warning that adopting it blindly, without attention to meaningful outcomes and appropriate safeguards, can cause major setbacks. In his view, many organizations lean on the human in the loop as an important control, but no one subjects it to real stress testing.

Darren Kimura, CEO and president of AI integration platform AISquared, agrees that many companies are deceiving themselves with these systems. "Most companies that say they have a human in the loop actually have a human watching the loop," he says. That person can see the decision and flag a concern, but cannot stop it, change it, reject it or escalate it. Kimura suggests IT leaders ask themselves specific questions: can reviewers halt actions before they take effect? Can they modify the outcome? Are their overrides logged and actually enforced downstream? If the answer to any of them is no, the human is merely monitoring the AI, not controlling it.

Another problem flagged is decision fatigue: when employees are asked to review too many AI decisions, they end up approving mechanically rather than reflecting on the consequences. Eric Billingsley, director of operations and technology at AI assurance company TrustScale, stresses that the reviewer needs expertise, context, sufficient time and also the authority and technical ability to reject or reverse a decision. But even a qualified reviewer with authority can gradually lose independent judgment when the AI is consistently right: "if the system is right 95% of the time, the person's job becomes waiting for the rare case where it's wrong," he explains. Humans are not especially good at maintaining sustained vigilance over a highly reliable automated system, and over time review turns into mere confirmation. A good AI system, Billingsley notes, can end up producing poor human controls, because when the exceptional case arrives, the reviewer may approve it anyway, since the system itself has trained them, through hundreds of correct recommendations, to trust it.

Billingsley recommends assessing human-in-the-loop systems with the same rigor as any other security control: they should be monitored, tested and able to provide evidence that they are working as intended. A simple log showing that someone clicked "approve" is not enough; it must be demonstrated that the person had the necessary context, applied independent judgment and had real authority to override the AI.

Robert Blumofe, executive vice president and chief technology officer at cloud computing and security provider Akamai, sees the same problems: while some form of human oversight is preferable to a fully autonomous AI, the human in the loop can turn into a mind-numbing exercise. Large language models produce the right output often enough to make them seem more reliable than they really are, and after diligently checking the AI's output again and again without finding errors, that diligence fades and review becomes routine approval. Blumofe advises IT leaders to spend time understanding exactly how the system is designed and when and how the human will interact with the AI before accepting a promise from a vendor or from their own team. He also suggests exploring other technologies as a safeguard, rather than relying on unreliable human oversight: non-AI tools that automate validation of outputs, flag problems and have the ability to pause the system's work, thereby reducing risk without depending solely on human approvals.

Finally, several of those consulted agree that the human in the loop is not always the right solution. In cybersecurity, for instance, waiting for a person's approval to mitigate an incident can come too late: if an endpoint is compromised, it may be preferable for the system to isolate it immediately, since waiting 20 or 30 minutes for someone to approve that action could allow the attack to spread. The goal, Kimura sums up, is not to put a human into every AI decision, but to place the right human, with the right context and authority, at the precise point in the workflow where they can actually make a difference.