AGI 'has arrived' on OpenAI's own harness — the benchmark's authors measured 62.7%

Two sentences, three days apart, did the work. On September 3, according to the report, OpenAI president Greg Brockman told the press "Welcome to the AGI era." On September 6, NVIDIA CEO Jensen Huang posted a single line on X: "From ChatGPT to o1 to Astra in 4 years, AGI has arrived. Congratulations @OpenAI team." The number underneath was 99.9% on ARC-AGI-3 — the score OpenAI reported for Astra, its new frontier model released under the GPT-6 designation — alongside 98% on FrontierMath Tier 4. The organization that built ARC-AGI-3 then published its own figure from its reference evaluation harness: 62.7%. Thirty-seven percentage points apart.

Both numbers are real, and that is the uncomfortable part. As the article makes clear, OpenAI's internal evaluation setup and the benchmark creators' reference protocol are not the same procedure, and the same model scores differently under each. A harness is the scaffolding around a test: how many attempts, what tools, what prompting, what time budget. This is not a caught-lying story. It is a story about which of two legitimate numbers gets to be the headline — and 99.9% versus 62.7% is precisely the distance between "very strong result" and "a threshold has been crossed."

Worth saying plainly, because the reflex is to read the lower figure as a debunking: 62.7% on ARC-AGI-3 is a serious result. The article notes the benchmark's predecessors had saturated — models absorbed material resembling the questions and pushed scores up without getting better at novel reasoning — and ARC-AGI-3 was redesigned to demand abstraction rather than pattern retrieval. A score in the sixties on that test was out of reach two years ago. The researchers who pushed back, per the report, were not disputing the capability. They were disputing the leap from a benchmark score to a declaration of arrival, which requires an intermediate step nobody in the field has: a stable, shared definition of AGI. OpenAI has an internal one. A company grading its product against a rubric it wrote itself is a different class of evidence than one measured against an external standard.

Then there is the verification gap. As of the article's publication, Astra appeared in neither the Artificial Analysis Intelligence Index nor Arena.ai's rankings — the two most-watched independent evaluation systems for frontier models, and the kind of external input we rely on ourselves when we score models. Independent numbers for a major release usually land within days. Their absence proves nothing about the model's quality; it means the outside check that should accompany a claim of this magnitude simply is not there yet. Declare first, verify later is a workable sequence for a product launch. It is a bad sequence for a civilizational milestone.

The incentive geometry deserves attribution rather than accusation, and the article is careful here: Astra was trained on NVIDIA chips, Huang sells NVIDIA chips, and the AI infrastructure market moves on narrative. If Astra is AGI, the silicon that trained it becomes the defining infrastructure of whatever comes next, and the buyers funding the next training run purchase toward that benchmark. Nobody needs to act in bad faith for a congratulation to also function as a product statement. That is structure, not conspiracy — but structure is exactly what readers should be told about, and the article's charge is that most coverage passed both claims along without comment.

Our read: what is missing here is not capability, it is a norm. Frontier claims should ship with the harness — protocol, attempt budget, tooling, scaffolding — and any headline score that cannot be reproduced under the benchmark authors' reference conditions should be reported as a vendor number and labeled as one. We keep repeating this because it keeps mattering: measurement over marketing, demonstrated capability over aspiration. And the long-term case survives this episode intact. Systems that genuinely abstract over unfamiliar problems are how we get compressed drug discovery, earlier diagnosis, more healthy years, and eventually the kind of abundance that frees people to work on what they love. That road runs through verifiable measurement. Every premature "we have arrived" spends a little of the credibility the field will need on the day the claim is finally true — and makes it harder for anyone outside the labs to tell the difference.