The 'humanoid lesson': will AI leave database research anything to do?

The article is a critical review, published on A. Jesse Jiryu Davis's blog, of an academic paper by Berkeley and Stanford researchers titled 'What Happens When the Model Eats the Stack? Rethinking the Research Agenda for Data Agents to Withstand the Bitter Lesson'. The underlying thesis it discusses is uncomfortable for any field of applied research: if foundation models keep improving in line with the 'Bitter Lesson' (the now-classic idea in AI that general methods leveraging more compute eventually beat any handcrafted technique designed to compensate for specific model weaknesses), what is left for a database researcher who does not work at OpenAI, Anthropic or NVIDIA?

The author notes that we have seen this pattern repeat before: every time the community designs a 'harness' (a scaffolding of tools, prompts or processes) to paper over an LLM limitation, the next generation of models makes that scaffolding unnecessary. Meta-harnesses have even been built to automate the design of harnesses themselves, and still the pattern repeats: the Bitter Lesson is never contradicted, it is simply learned again.

The paper under review answers that question with a 'yes': it proposes a research agenda for adapting data systems to use by AI agents. But the author is skeptical and openly describes it as a kind of 'cope' (a form of self-consolation in the face of an uncomfortable reality). His underlying doubt is that AI will leave no software 'on the plate': the problems that survive are, in his view, the same ones humanity has been dragging along since the dawn of the information age, because AIs and humans share a similar difficulty in accessing organizations' tacit or 'tribal' knowledge.

The text connects this paper to an earlier one by two of the same authors, titled 'Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First', which predicted that AI agents would flood databases with speculative, poorly optimized queries, and that data systems should therefore be redesigned to absorb that load pattern. The new paper qualifies that prediction: it argues that the latest coding-agent models are drastically reducing the average number of turns and tokens needed per query, which would call into question the premise that future data systems will be constrained by large-scale, inefficient agentic speculation. The authors state that new models natively improve efficiency on a given task, solving each one with fewer actions and failures, generation after generation.

The reviewer concedes it is reasonable that newer models 'fail less', but points to an inconsistency: the 'Overlords' paper also warned about the heterogeneity of agent queries, and in his judgment there is no new evidence that such heterogeneity has stopped being a problem for databases traditionally designed for repetitive OLTP workloads. He also notes in passing that the study measures efficiency and accuracy only on GPT models (citing specifically results with 'GPT-5.6 Sol' on the Data Agent Benchmark) without including models from the Claude family, something he finds odd.

One finding of the paper the author does value is the change in the nature of errors as models improve: failures from incorrect tool use or execution errors decline, but the remaining errors come to be dominated by gaps in knowledge of the environment. As he quotes verbatim, more than 60% of the failures of the most advanced models stem from poor semantic interpretation: not understanding the real meaning of the task, relying on invalid proxies, using the wrong sources, tables, fields or metrics, or mishandling entities, join keys and identifiers.

From there, the paper's authors propose that the research community should study how to give AIs 'persistent semantic context': curated knowledge about what data exists, what it means and which sources are authoritative. They give as an example a typical business task: drafting a quarterly review explaining why European revenue missed forecast, reconciling the discrepancy with Finance's official figures and recommending corrective actions — a task that requires identifying authoritative datasets spread across multiple systems, resolving conflicting metric definitions, understanding business conventions specific to the organization, interpreting semi-structured documentation and validating intermediate results.

The blog's author accepts the diagnosis but rejects the idea that it is a new problem: he argues we have been talking about exactly this kind of 'persistent semantic context' for decades under the name tribal knowledge. Any competent professional joining a company, or moving to a different team, runs into the same difficulty finding information that is not formally documented. For him, AI changes nothing substantial on this point.

It is here that he proposes his own idea, as an ironic counterpoint to the Bitter Lesson: the 'Humanoid Lesson'. His formulation is that, to guess an LLM's strengths and weaknesses, a good first approximation is to anthropomorphize it: AIs have the same problems people do. Drop a brilliant person into a large corporation and they will spend most of their time uncovering the organization's informal knowledge, especially if they also — as happens with an LLM — suffer a kind of total amnesia every few minutes. And just as that brilliant person performs better if given freedom rather than a rigid process imposed on them, he suggests something similar happens with models.

The author criticizes the paper for framing, as though it were an unprecedented question specific to the AI era, who should learn about a knowledge update, how it should propagate across overlapping domains, and when conflicting semantic views should be reconciled or coexist. His answer is that the question is identical for humans and machines, and that the few genuinely interesting questions in the paper would still be interesting even if LLMs did not exist.

In the section he titles 'Just Use Git', the author takes his argument to very concrete ground, drawing on his own experience. He recounts that in MongoDB's early days the documentation lived in a wiki separate from the code, and the two fell out of sync constantly, which generated interesting — but avoidable — questions about consistency. The solution was not a novel technique but a process change: they began keeping architecture guides in the Git repository itself, modifying them in the same commit as the code they describe, and developed a process in their ticket tracker and in code reviews to check which documentation was affected by each change. His conclusion is that the only real difference between humans and LLMs in this area is that humans are lazy and LLMs are 'maniacal'.

On whether stale context should be fixed immediately or on demand — another question the paper leaves open as a research problem — he considers it equally over-analyzed: he takes it as given that documentation is read far more than it is written, so it almost always pays to update it immediately; if a piece of tribal knowledge is never consulted, the sensible move is to delete it or find out why nobody can find it, not to design a sophisticated consistency system to manage it.

The article's closing broadens the frame to a live market debate: the question of whether a given technology is a complement to or a substitute for LLMs. The author recalls that in February investors decided Claude was a substitute for all software, because it could build any application on demand, which led to a mass sell-off of SaaS stocks (an episode he calls the 'SaaSpocalypse'). Afterwards, those same investors decided some SaaS companies were in fact complements to AI, and their shares recovered; since then, he says, the market has kept swinging between the two readings.

He transposes that same uncertainty to data systems research: if you invent some technique that makes a model work better today (he mentions Claude Fable), it is worth asking whether that improvement will also compound in the next generation of models, or whether the model itself will end up absorbing that capability and rendering the technique obsolete. The paper under review bets on the first option, suggesting that the current technology stack is nearly obsolete but that a new stack will provide lasting value; the author declares himself skeptical and argues it is just as likely that future AIs will end up outperforming any existing system by writing new software from scratch for each specific application. He closes by acknowledging, without resolving it, that this would be a lesson as bitter as the original, and that he does not know how the community — readers included — is going to cope with it.