METR's evaluation of unreleased Anthropic models shows the frontier is ahead of what the public can see

METR is an independent organization that evaluates the capabilities of frontier AI models. According to the brief note we reviewed, it has identified several internal Anthropic models that haven't been publicly released. The top performer, labeled "Model 4," reportedly scored above 50% on all the benchmarks METR evaluated.

That is all our source says. We don't have the benchmark names, the exact scores or how they compare with Anthropic's public models. We also can't confirm whether "identified" means METR found these models on its own or tested them with Anthropic's cooperation, as usually happens in pre-deployment evaluations. The word "secret" in the original framing should be read with that in mind. Labs routinely keep models internal before release. Having unreleased models isn't wrongdoing, and the note doesn't accuse Anthropic of any.

Our thesis: the more useful takeaway isn't the 50% mark. It's the gap between the public frontier and the internal one. Our own rankings already put Anthropic at the top of the public field. If the internal models are stronger still, every public comparison we and others publish is a snapshot of the past, not the present. A score above 50% on every evaluated test can only be judged against those tests. Saturated benchmarks tell us little. Hard, unguided tasks tell us much more.

Our reading: this is a governance story more than a capabilities story. External evaluators seeing models before the public does is exactly the kind of evidence-based oversight we have argued for. The open question is whether that visibility should stay voluntary. As internal models pull ahead of released ones, regulators and the public need a reliable view of what labs are running behind closed doors, not only what they choose to ship. Independent evaluation like METR's points in the right direction. The next step is making it systematic.