AI Builds AI: what is actually improving?

By Turing Post · September 2026.

Turing Post returns to its full publishing schedule after the summer and launches a new series called "AI Builds AI", devoted to examining in detail what it really means for artificial intelligence to help build the next generation of artificial intelligence. The starting point is the tension that has flared up this week around AI's ability to improve itself.

On September 6, OpenAI announced it had hit its goal of having an "automated AI research intern": a system capable of carrying out well-defined research tasks under human direction. That same day, its chief scientist, Jakub Pachocki, warned that no one is prepared for the consequences of machine intelligence continuing to advance this fast, that no lab has yet solved alignment and oversight well enough to keep scaling at full speed, and that OpenAI still does not know how to safely reach full, aligned recursive self-improvement.

OpenAI also acknowledges it cannot assume safety work will keep pace: after an incident at Hugging Face in which agents compromised research infrastructure, the company paused reinforcement learning on models bound for deployment while it hardened those environments.

The author argues that the relevant question is not whether a warning sounds dramatic, but what is actually being improved, who decides the next change and how the result is checked. That is the rationale behind the new series.

On September 8 the temperature rose further: AI researcher Jacob Coxon resigned from Anthropic, accusing both that company and OpenAI of an irresponsible race toward self-improving superintelligence. Shortly afterward, Evan Hubinger, head of Alignment Science at Anthropic, wrote a line on X that drew more than 115 million views: "we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade".

The author questions that figure: a claim of a greater-than-10% chance of human extinction demands more explanation than a simple statement of personal belief. Without spelling out how that estimate was reached or what evidence would change it, numbers like this feed fear without really helping anyone understand the risk. Her stated interest is in understanding that risk in practical terms: what happens when AI starts contributing to the research, the code and the tools used to build the next generation of AI, how researchers check that work, and what changes when the systems doing the checking are also AI.

For this first installment in the series, the announced topics are: what exactly is being improved; a metaphor described as "a pencil tied to a brick"; why a model may need a lower-level tool; what today's feedback loop already looks like; how cheaper implementation changes which ideas are worth trying; whether models can learn to tell what is worth improving; how the evaluator itself becomes part of the problem; and what we should record about this moment.

As a connecting thread, the author says she spoke with Phil Tillet (creator of Triton) and Matthew Ferrari about inference engineering at OpenAI. One detail from that conversation stands out: models have become capable enough that some decisions programming tools used to make automatically now get in their way. Engineers are handing those decisions back to the programmer, who is increasingly a model. This detail serves as a concrete starting point for examining bigger claims about AI autonomy, grounded in engineering decisions already being made today.

The analysis distinguishes between the "object" being improved and the decisions AI is allowed to make. A model can write a faster implementation of an existing mathematical operation, propose a training method that changes another model's behavior, or help choose the next experiment; these are contributions at different points in the development process. The email lays out three levels: (1) the system that runs a model, where AI can contribute faster kernels, better scheduling or less redundant compute, leaving open whether a local gain improves the whole system without breaking required behavior; (2) a model in training, where AI can propose changes to architecture, data or learning methods, leaving open whether the improvement generalizes beyond the tests used to select it; and (3) the research process itself, where AI can improve experiment selection or evaluation methods, leaving open whether the system genuinely gets better, round after round, at finding useful improvements.

A model can have plenty of autonomy writing a kernel and at the same time need detailed guidance to decide the research question: these are different dimensions of progress, and they should not be conflated.

The author stresses that recursion adds a further requirement: improvements have to feed back into the ability to produce further improvements. A model writing a useful training script is already a contribution to AI development; showing that this contribution makes the next round of development more effective is a stronger claim; and showing that the cycle can continue without people steering it is stronger still. On this framing, an automated intern executing well-defined tasks under human direction is not the same as a system that chooses better research questions on its own, and a personal probability of extinction measures neither.

The author notes that even Anthropic's own analysis, "When AI builds itself", explicitly concedes that fully autonomous development of successor systems has not been achieved and is not inevitable; and that OpenAI's post on accelerating research says the same thing in different words: the company still does not know how to safely reach full, aligned recursive self-improvement. Even so, she concludes that the concrete engineering work is worth examining, precisely because the final outcome remains uncertain.

The email then introduces a section titled "A pencil tied to a brick", but the content of that section and of the remaining items on the list — why a model may need a lower-level tool, what today's feedback loop looks like, the effect of cheaper implementation, whether models can learn what to improve, and the evaluator's role — sits behind the newsletter's paywall ("Upgrade to read the rest"), so their actual development cannot be summarized from this email. The text also includes a link to the full YouTube interview with Phil Tillet and Matthew Ferrari, titled "Does AI Understand the Machine It Runs On? | Inside OpenAI".