Neither Blind Faith nor Panic: The Science of Knowing When (Not) to Trust an AI

TWO RESCUES IN ONE WEEK. The first week of September produced two stories that look like opposites and are in fact the same story. Three inexperienced hikers had to be rescued on Mount Shasta (California, 14,179 feet) after planning their route and packing list with Gemini, Google’s AI assistant. According to the Siskiyou County Sheriff’s Office, the AI recommended insufficient food and water, and the group started their descent long past the safety turnaround time; they ended up off-track in Mud Creek Canyon with one knee injury. Days later, Nvidia CEO Jensen Huang declared that “AGI has arrived” because Astra, OpenAI’s new model, had scored 99.9% on ARC-AGI-3, a test designed to measure whether a model generalizes to novel problems. The benchmark’s creators answered with their own measurement: 62.7% under their standard harness — the neutral software scaffolding that connects any model to the test under equal conditions. Thirty-seven points of difference without touching the model: only the exam environment changed. Our thesis: the central problem of AI in 2026 is no longer how much it knows, but that almost nobody — not users, not executives, not CEOs — knows how to calibrate how much to trust it on any given task. And that, unlike AGI, has 40 years of science behind it.

FORTY YEARS OF SCIENCE. Aviation solved this problem the hard way, and we should copy its homework. In 1997, researchers Raja Parasuraman and Victor Riley named the two failure modes, which are symmetric: “misuse” (trusting automation when you shouldn’t) and “disuse” (switching it off or ignoring it when you should trust it). Both kill. That same year, American Airlines captain Warren Vanderburgh coined the phrase “children of the magenta”: pilots who followed the magenta line on their navigation display all the way into disaster; according to his famous training talk, 68% of airline accidents at the time involved automation dependency. The cases are grimly famous: on Air France 447 (2009), the autopilot disconnected over a minor fault and the crew could not recover the aircraft by hand; on Asiana 214 (2013), investigators cited over-reliance on automation as a major factor in the San Francisco crash. The mature lesson of that literature was formulated by John Lee and Katrina See in 2004: the goal is not to trust more, nor to trust less — it is “calibrated trust,” meaning your level of confidence should match the system’s actual reliability on that specific task. Aviation applied that science, and flying is now the safest way to travel. It did not throw out the autopilot: it learned when to watch it.

THE JAGGED FRONTIER. Generative AI adds a complication the autopilot never had. In 2023, researchers from Harvard and the consulting firm BCG measured it with 758 real consultants (the study by Dell’Acqua and colleagues). On 18 tasks that fell within GPT-4’s capabilities, consultants using AI completed 12.2% more tasks, 25.1% faster, with over 40% higher quality. But on one task designed to fall just outside those capabilities, consultants using AI were 19 percentage points less likely to get the right answer than those working without it. The authors called it the “jagged frontier”: AI is superhuman on some tasks and useless on others that look identical to the user. And the frontier is invisible from the outside: Gemini plans a tourist itinerary through Rome brilliantly and plans a high-mountain route terribly, yet both questions are typed the same way, into the same text box, and both answers come back impeccably written. This is where the best analogy in this debate fits: nobody lets an intern perform open-heart surgery, because the intern hesitates, stumbles and asks for help. The problem is that this intern always speaks with the confidence of a chief surgeon.

FLUENCY IS NOT COMPETENCE. Why do we fall for it? Because we use mental shortcuts. Since users cannot verify every answer, the brain swaps the hard question (“is this correct?”) for an easy one (“does it sound confident and well written?”). The data says that shortcut fails with today’s models. A Carnegie Mellon study published in Memory & Cognition (July 2025) compared the self-confidence of humans and chatbots: humans adjust their expectations after failing; the models do not. In a Pictionary-style game, Gemini predicted it would get about 10 sketches right, got 0.93 on average, and afterwards claimed it had gotten 14.4. Another 2025 academic paper documented that users systematically rely more on answers expressed in confident language, and that the effect holds across every language tested. The Harvard/BCG study added an unsettling detail: when consultants pointed out the AI’s errors, it did not acknowledge its limits — it escalated its persuasion. Let us say it plainly: fluency is a product of the model’s training, not an indicator of its reliability. In humans, confidence and competence correlate, however imperfectly. In today’s models, barely at all.

OUR READ. Three conclusions. First: the Astra episode confirms our long-standing thesis that measurement beats marketing. A 99.9% under the vendor’s own protocol and a 62.7% under the neutral harness is not a technical nuance: it is the difference between demonstrated capability and narrative. And ARC Prize was right to remind everyone that even saturating its test “would not represent proof of AGI.” Second: Parasuraman’s two symmetric errors are now playing out at social scale. Misuse: the Shasta hikers, or the executive who cuts staff because “AI already does that” without ever verifying it. Disuse: the hospital or school that bans a tool capable of reducing errors. The second error makes fewer headlines, but it also costs lives and opportunities. Third: today, calibration falls on the user because the product does not provide it. Chatbots do not display their error rate by task type, do not distinguish “I know this” from “I am generating this,” and are optimized to please. That is a product-design failure, not a law of nature: recent human-computer interaction research already treats trust calibration as a design problem, not as user clumsiness.

PRACTICAL RULES AND IMPLICATIONS. While the design catches up, the literature distills into a two-axis heuristic: cost of error and ease of verification. If the error is cheap and verification easy (a draft, code you test by running it, a summary you will read in full), delegate freely: there, AI is already superhuman in speed, and the BCG study quantifies it. If the error is expensive and verifying it requires expertise (health, money, law, a 14,000-foot mountain), AI is a first draft, never the final word: demand sources and cross-check with a human or official authority. The Siskiyou sheriff said it precisely: call the ranger station before you climb. And a third rule that cuts across everything: distrust the tone; a model’s verbal confidence carries no information about its accuracy. Long term, we keep our usual, tempered optimism. Aviation took a generation to turn automation into the most effective safety system in the history of transport, and there is no physical reason AI cannot walk the same path: models that reliably say “I don’t know,” interfaces that display reliability per task, and a public literacy that does not yet exist. The abundance we expect from this technology — earlier diagnosis, faster science, personalized education — depends less on the next benchmark point than on this: that millions of people learn, as pilots did, when to follow the magenta line and when to look up.