OpenAI's math solutions still fall short of the standards the field itself asked it to follow

OpenAI published hundreds of purported solutions to some of the world's hardest math problems this week, presenting it as a responsible exercise: the company said it had consulted an advisory group of top-tier mathematicians to avoid the controversy generated the previous time one of its models solved a historic problem in the field. According to TechCrunch, however, OpenAI did not fully meet those standards, especially on the point mathematicians considered most important: ensuring there is real human understanding behind each result.

The body in question is the Advisory Group on Mathematics and Artificial Intelligence (AGMAI), hosted by the Institute for Advanced Study and made up of nine prominent researchers from institutions around the world. In late September, AGMAI published a set of guidelines aimed at frontier labs solving mathematical problems. On this new batch of proofs from OpenAI, the group said only that it is up to the mathematical community itself to assess the extent to which its recommendations were followed, and did not respond to TechCrunch when asked for a more detailed assessment.

One of those recommendations — in fact the first on the list — explicitly called for "stopping testing advanced math problems on proprietary models". OpenAI ignored that starting point: its own statement acknowledges that it is evaluating its proprietary models precisely by means of open research problems, that is, doing the opposite of what was requested.

In other respects, the company did follow some of the guidelines, such as publishing the results as soon as possible and including information on how the models arrived at their conclusions. But compliance was partial and uneven: of the 719 manuscripts published, only 10 included the model's reasoning (its "chain of thought"). And for proofs the community cannot understand on its own, AGMAI had suggested formalizing them; even so, according to the source, 42% of the proofs published by OpenAI had not gone through that formalization process.

The backdrop to all this is an uncomfortable question: who takes responsibility for making sure these results end up being understood? Under AGMAI's own guidelines, the lab publishing a proof must take responsibility for ensuring that publication is followed by human understanding; the advisory group further suggested that OpenAI should help fund the work of the human mathematicians who will have to digest and make sense of those solutions. It is not clear that OpenAI is holding up that end.

Terence Tao, one of the mathematicians who had already criticized OpenAI's approach, summed it up bluntly on social media after the release: the problems are being solved autonomously by AI "prompters" who have no real interest in the field beyond marking their initial goal as "solved", and who do not understand the model-generated result well enough to answer questions about it, give talks or engage with the rest of the community.

That underlying problem is illustrated by a study published this same week by mathematicians at the University of Cambridge and King's College London, which casts doubt on the procedure frontier labs are using to tackle these challenges. When an AI model solves a math problem, it first writes an explanation in "natural language" and then tries to translate that reasoning into Lean, a programming language that in theory certifies the proof's validity by compiling it as code. The study documents at least two discrepancies between the natural-language explanation and the Lean code underpinning a solution offered by OpenAI to a problem derived from the Navier-Stokes equations, which describe the complex behavior of fluids.

Those discrepancies do not necessarily invalidate either version of the proof, but they do raise a legitimate doubt: can the models be trusted to formalize their own solutions correctly without human supervision? That is precisely why AGMAI had asked OpenAI to include machine-readable metadata correlating the natural-language text with the formal artifacts, something the company did not do in this release. The study's authors, who informally dubbed it "lost in translation", conclude that, because of the mistranslation phenomenon they document, neither OpenAI's natural-language proof nor other self-formalized Lean proofs should be taken as sound without being subjected to the same peer review and scrutiny any other mathematical proof receives.

Mathematicians insist that when a new result is discovered by a human being, that person takes responsibility for it and defends it before the community through papers, talks and seminars — a process that broadens collective understanding of the result, makes it possible to find strategies applicable to other problems, and helps turn that knowledge into practical uses. Melanie Wood, a professor of mathematics at Harvard, puts it starkly: when a model is asked to solve a hard problem and it delivers a solution, at that moment of publication there is no human understanding of the results, and that is precisely where the real work begins.

All told, the episode paints a picture of an AI sector advancing faster in producing results than in building the verification, translation and human-appropriation mechanisms those same results require in order to be useful. The very existence of an advisory group like AGMAI, and the fact that its recommendations were only partially followed shortly after being published, suggests that the tension between the speed of these releases and the far slower pace at which the mathematical community can digest and validate them will remain a source of friction.