An agent that predicts its own changes improves more, but spends 3.4 times more tokens: the price of self-evolution

By NeoTeo · October 17, 2026.

The preprint "Predict Before You Modify: Learning Improvement Dynamics for Self-Evolving Agents" introduces the Improvement Dynamics Model (IDM). It is an editable program that predicts, task by task, what effect a proposed change to an AI agent's code will have before the change is accepted. According to the study's own data, it outperforms its baseline in programming, but consumes considerably more tokens.

The IDM builds on HyperAgents, a framework in which an agent performs tasks and a meta-agent modifies that agent's program as well as its own. To make predictions, the IDM uses the test history, the base agent, its results and the proposed code change (the "diff", the difference between versions). It calls a frozen language model: what evolves is the agent's program; the model is not retrained.

The prediction does not act as a gate. If the aggregate forecast recommends revising a proposal, the system can request at most one additional revision. It then predicts again and evaluates the resulting agent even if the forecast remains unfavorable. The IDM itself can also mutate: every five generations the meta-agent can propose changes to the predictor, which are adopted only if they reduce per-task prediction loss without increasing the centered mean squared error.

The experiments cover four domains: multilingual programming (Polyglot), paper review, reward design for simulated robotics, and grading of Olympiad-level math problems. Each method ran five times for 40 generations, with shared initial agents, models and protocols.

In programming, the AUBC (area under the budget curve, which summarizes performance across generations) had a median of 0.306 with Selective IDM, versus 0.210 with HyperAgents. On 166 held-out test tasks, the median gain in solve rate over the initial agent (which started at 16.27%) was 13.25 percentage points with Selective IDM and 5.42 with HyperAgents.

The cost: in full programming runs, Selective IDM used an average of 170.22 million tokens, versus 49.45 million for HyperAgents. That figure includes agent evolution, evaluation, predictions, IDM updates and their validation. With a matched budget of 44.21 million tokens, the preprint reports a 10.9% higher median AUBC for Selective IDM. The advantage is real, but far more modest than in the unconstrained comparison.

Outside programming, the results are more restrained. In paper review, the median AUBC was 5.9% higher, and 3.9% higher on the joint objective with robotics. An IDM developed in those two domains and transferred to math grading achieved a median accuracy 4 points higher than Selective IDM and 8 points higher than HyperAgents. The final IDM reduced per-task prediction loss by between 8.7% and 24.4% compared with the initial predictor.

Our take. The underlying idea is sound: if every change to an agent is evaluated by running it, and that is expensive, learning to anticipate which changes are worth it is a natural path. But the preprint's most honest finding is the one that will make the fewest headlines: prediction is not free. The 3.4-to-1 token ratio shows that self-evolution is still constrained by compute budget, and that the real improvement, measured at a matched budget, is 10.9%, not 45%. That is the comparison to focus on, not the raw AUBC.

There is also a methodological caveat. The tests were run on benchmark tasks, including simulated robotics. The authors themselves warn that this is not enough to justify replacing human judgment or deploying the system without safeguards. And the token count is not equivalent to total compute cost, energy or money. More broadly, agents that rewrite themselves raise a governance question that performance does not answer: if the code being executed changes on its own, who verifies what changed and why?

Even so, the direction matters for the long term. If agents learn to improve themselves with less trial and error, the cost of obtaining competent software and automation will fall, and that feeds the abundance we advocate. In the short term, however, the advantage goes to whoever can afford 170 million tokens per run. And in any autonomous improvement, verification will remain the scarce resource.