OpenAI's shifting launch numbers: a 100%-to-39% drop is the benchmark crisis in miniature

The facts, as reported: five of the figures OpenAI published alongside a launch were subsequently found to have changed. The most striking case is an "exploit bench" — a test of a model's ability to find and use software vulnerabilities — where the reported score fell from 100% to 39% once evaluators ran a fresh version of the test, built specifically so the model could not fall back on answers it had already seen during training.

That single number deserves more attention than the correction itself. A 100% is not a good result; it is a broken instrument. Any test where a model answers everything correctly has stopped discriminating between systems and started measuring exposure — whether the questions leaked into the training data. The 61-point drop between the original and the fresh version is, in effect, the contamination tax made visible. It is rare to see it quantified this cleanly.

This is why we build our own cybersecurity index on expert, unguided tasks rather than the saturated classics. Public benchmarks like Cybench have been pushed to near-perfect pass rates by frontier models, at which point they tell you nothing about the frontier. The discriminating evaluations today are the ones with held-out problems, no scaffolding, and no published solution sets — precisely the ones that are expensive to run and unflattering to quote in a launch post.

A note on fairness: quietly revised numbers are not automatically bad faith. Evaluation harnesses are genuinely fragile, and re-running a suite with a fixed prompt template or a corrected parser legitimately moves scores. The problem is procedural, not moral. Software ships with changelogs; benchmark tables ship without them. If a figure changes after launch, the industry norm should be a dated note explaining what changed and why — not a silent edit that only outside observers catch.

Our read: this is what the adolescence of a measurement discipline looks like, and it is fixable. The credible path is already visible — held-out test sets maintained by third parties, fresh variants generated per evaluation, and vendors reporting on benchmarks they cannot see in advance. Companies that adopt that discipline early will end up with the only numbers buyers trust, which turns transparency into a competitive asset rather than a cost. In the meantime, the practical rule for anyone deploying these systems is unchanged and now better evidenced: treat any published score near 100% as a claim about the test, not about the model, and run your own evaluation on your own tasks before you believe anything.