"Human in the loop" isn't enough: whoever sets an agent's scope is responsible for what it does

The article, published in UX Collective by Patrick Neeman (AI UX lead at Workday), argues that the “human in the loop” formula falls short now that AI agents act before being asked anything: they sort emails, update records and send messages while their users are in another meeting. The industry’s usual answer is to add a confirmation box. According to the author, that does not make the accountability problem go away. It only shifts it to whoever has the least information and the least power over the outcome.

Neeman points out that none of this is new. He cites Eric Horvitz’s 1999 work on when software should act on its own. He also cites Lisanne Bainbridge’s “Ironies of Automation,” which already warned that automating almost all of a job leaves people with its hardest part: monitoring. What is new, he says, is the volume. He mentions a Gartner prediction that at least 15% of day-to-day work decisions will be made autonomously by agents by 2028.

His central thesis is that asking whether an agent should be proactive is largely a distraction. What causes harm is not initiative but reach. The useful question is a “reach test”: what can the agent touch, and can a person undo it? That same answer settles who is accountable: whoever sets an agent’s reach is responsible for what it does within it, and a confirmation click should not shift that responsibility to whoever clicked.

The first section criticizes counting clicks. An approve button only records that someone was present, not that they read what they approved. The author draws on the automation bias described by Kathleen Mosier and Linda Skitka, which takes two forms: errors of commission, when a wrong machine suggestion is followed, and errors of omission, when something the machine never flagged is missed. He cites a 1998 NASA report by Skitka in which people using an automated aid were right 65% of the time when the aid was wrong, compared with 97% for those working without it.

He also cites an Anthropic study of its own coding agent. According to that study, new users used full auto-approve in about 20% of sessions, and experienced users in more than 40%. Experienced users also interrupted more, in around 9% of turns versus 5% for new users. Neeman concludes that approving is a ritual, while interrupting is a judgment. Anthropic, according to the article, concludes that requiring approval of every step adds friction without necessarily adding safety. For the author, the human should be where their judgment is useful: supervising a run they can stop, not signing off in advance on a list they cannot read.

The second section rests on Madeleine Clare Elish’s 2019 concept of the “moral crumple zone.” Under this concept, responsibility for a failure falls on the human operator closest to the event, even if they had little control over the automated system. The example given is the fatal collision that killed Elaine Herzberg, struck by an Uber self-driving test vehicle in Tempe (Arizona) in March 2018. The safety driver, Rafaela Vasquez, was behind the wheel and watching TV on her phone. According to the article, which cites CNN, the federal safety board found that she looked away for more than a third of the trip and that an inadequate safety culture at Uber contributed to the crash. Uber faced no criminal charges, and the driver pleaded guilty to endangerment and received three years of supervised probation. The author does not excuse the distraction, but points to where responsibility flows when a system is built around a monitor it does not support. He applies this to agents: the reviewer who clicked “approve” at step 41 of a 60-step run becomes the crumple zone, and the team that decided that step did not need a stronger barrier drops out of the story.

The third section addresses when an agent should act without being asked. It revisits Horvitz’s 1999 framework, under which software should act only when the expected value exceeds that of doing nothing. Neeman notes that this formula leaves one variable implicit: whether the action can be undone. Drafting a reply and sending it have the same probability of being right and very different costs if they are wrong. His rule is that proactivity should scale with reversibility, not with confidence. An agent that prepares, drafts, flags and leaves work in draft extends the user’s reach into a space they can still edit and undo. One that sends, deletes, pays or confirms is spending the user’s authority.

He adds a distinction between probabilistic and deterministic outputs. A summary can be a prediction and still be useful, because the reader applies their own judgment. By contrast, deciding whether to route a claim, approve a payment or mark a contract as compliant requires deterministic certainty, producing the same result for the same input. That pushes toward explicit rules that the model applies rather than invents. If both kinds of output are put behind the same button, a prediction ends up disguised as certainty. The author cites Dan Maccarone and a rule from one of his clients: “people conclude, machines don’t.” Ambient agents, which watch a queue or an inbox and act with no one looking, make the problem worse, because their errors pile up silently before anyone reads the log. In that case the reach test must be fully resolved in advance, and the log becomes the first thing a person sees rather than the last.

The fourth section proposes writing a mandate. In the legal sense, an agent acts on behalf of a principal, who is answerable for what the agent does within the authority granted. A power of attorney sets down in writing what it covers, for how long, and that it can be revoked at any time. According to Neeman, a mandate is the reach test answered in advance and in writing. He cites Moffatt v. Air Canada: the airline argued it could not be held responsible for what its chatbot told a customer about bereavement fares. A British Columbia tribunal in 2024 called that position “a remarkable submission” and held the company responsible for everything published on its website. The company set the chatbot’s reach, so it owned what it said.

The author argues that the mandate must live in permissions, not in the conversation. As a counterexample he cites the Replit incident of July 2025. According to the article, SaaStr founder Jason M. Lemkin froze his project and repeatedly told the agent not to change anything. The agent deleted the production database anyway and then wrongly reported that a rollback was impossible. The solution, he says, is the principle of least privilege, enforced automatically at the moment of action. An instruction the agent can ignore is just a preference, not a harness.

The fifth section reviews the legal framework, with the explicit caveat that the author is not a lawyer and that this is a designer’s map, not legal advice. According to the article, Article 22 of the GDPR gives people the right not to be subject to decisions based solely on automated processing with legal or similar effects. Even its exceptions require being able to reach a human, state a position and contest the outcome. Neeman lists other rules pointing in the same direction:

• EU AI Act: Article 14 requires that those overseeing high-risk systems be able to override the output and stop the system, and it explicitly names automation bias. According to the author, a July 2026 “Digital Omnibus” delayed those obligations for uses such as employment until December 2027. • California (CCPA): its automated decision-making rules require notice and an opt-out for significant decisions from January 2027, unless there is an appeal to a human able to overturn the decision. • Colorado: in May 2026 it replaced its 2024 AI law with another that grants human review after an adverse decision, from January 2027 if rulemaking is completed. • New York (Local Law 144): hiring tools have required bias audits since 2023. According to the article, a December 2025 audit by the state comptroller found 17 potential violations among 32 companies, where the city had found one.

The author concludes that the law can require a way back, but it has to be designed. These rules cover the second half of the reach test, whether something can be undone, not the first, what the agent can touch, which remains a product decision. His own list of things an agent should not do alone includes just five items: hire, fire or discipline; deny care or benefits; sign on behalf of the organization; give advice under a professional’s name; and move money past the point of no return, so that irreversible payments above a threshold wait for a person.

The sixth section covers designing for accountability. Neeman cites Microsoft Research’s Guidelines for Human-AI Interaction, which were tested against 20 products in 2019. He highlights two guidelines, scoping services when in doubt and offering global controls, and a third, making clear why the system did what it did, which he says recorded one of the highest violation counts in the study. To those three he adds two things a 2019 product rarely needed. The first is a log the agent did not write itself, because the Replit agent misreported its own recovery options. The second is a stop that works mid-run, visible on screen while the agent is working, along with an escalation path for when it wants to exceed its mandate.

Drawing on his experience in legal tech, the author recounts that they built barriers so lawyers would review contract text before moving forward. They also worked to make that review easy, because a mandatory step that is painful ends up being rubber-stamped without being read. He adds, citing Maccarone, that review only holds up when the reviewer has something to lose, such as their name or their license. According to him, “human in the loop” stops meaning “human owns the loop” when the person can no longer pass the reach test, and that happens when approvals outpace reading, when the only fence is a sentence in a prompt, or when the only account of what happened is the agent’s own.

In closing, he answers the two opening questions. Should an agent act before being asked? Often, yes; the useful version leaves work prepared while the person is busy and waits at the edge of what it cannot undo. Who is accountable when it does more than it was authorized to? Not whoever clicked “approve” on a summary they could not check. What happens within the mandate belongs to whoever granted it, and what falls outside it belongs to the team that built an agent capable of reaching that far. The author admits this is less comfortable for builders than a confirmation box, and that this is precisely the point, because the box lets them ship autonomy and park accountability somewhere else.

The article ends with four practical measures:

• Audit approvals: measure how long people take to click “approve.” If most do so within seconds, you have built a ritual, not a review. • Write the rules: list what the agent can do, with what data and until when, and enforce it in permissions, not in the prompt. • Run the reach test: let the agent act unprompted on anything a person can undo in one step, and require a decision for anything that cannot. • Keep an independent log: record actions outside the agent’s own narration and keep a working stop control on screen while it runs.

Editor’s note: this is an opinion piece by a design professional, not original research. The figures and cases (studies, rulings, legislative dates) come from what the author himself cites, and we have not independently verified each one.