A fresh version of the test cut a 100% score to 39% — the benchmark was measuring memory, not skill

Five of the numbers OpenAI published around a launch changed after the fact, according to the report. The most telling revision involves an "exploit bench" — a test that measures whether a model can find and exploit software vulnerabilities — where the score fell from 100% to 39% when the evaluation was re-run on a fresh version of the test, written specifically so the model could not lean on answers it had already encountered during training.

That gap has a name in evaluation work: contamination. When a benchmark's problems have leaked into training data, a model can reproduce solutions it memorized rather than derive them. The fix is exactly what was done here — rewrite the tasks so they are new to the model — and the drop from 100 to 39 is the size of the illusion. Worth noting on its own: a 100% is never a good sign. A test everybody aces has stopped discriminating between systems; it has saturated, and saturated benchmarks measure nothing except that the ceiling was too low.

Context matters for how harshly to read this. Correcting published figures is not, by itself, bad faith, and we have no basis here to allege intent — the report describes changed numbers, not a proven attempt to mislead. Post-launch revisions are common across the industry, and a vendor that publishes corrections is behaving better than one that quietly leaves stale numbers up. The legitimate criticism is about sequencing: the fresh-variant score is the one that carries information, and it should be in the launch materials, not appear afterwards once someone else has checked.

This is a recurring theme for us, and cybersecurity is where it bites hardest. Offensive-security benchmarks are already known to saturate — Cybench-style suites have been run to near-perfect pass rates — which is why the evaluations that still discriminate are the hard, expert, unguided ones. These numbers are not marketing trivia. Regulators, enterprise buyers and the labs' own safety teams use claimed cyber capability to decide what to deploy and what to restrict. A number inflated by memorization pushes those decisions in both wrong directions at once: it invites panic about a capability that isn't there, and it erodes trust in the warnings that will eventually be real.

Our read: the weakest link in AI governance right now is not the models, it's the measurements. We should treat every headline capability score as provisional until someone re-runs it on tasks the model has never seen — and the labs should build that step into their own launches rather than outsourcing it to critics. The short-term picture is messy, and honesty about it is the point. But there's a constructive reading too. The scrutiny that turned a 100% into a 39% is the immune system working. The future we're arguing for — models that genuinely patch vulnerabilities, genuinely accelerate drug discovery, genuinely take drudgery off people's hands — depends entirely on being able to tell real capability from a memorized answer key. A credible 39% is worth more than a hollow 100%, because you can build on it.

One caveat we'll keep honest: the material available to us is brief. It does not identify which launch, which model version, or who re-ran the test. Those details matter, and we'd want them before drawing conclusions about anyone's intent.