Anthropic's 50-casualty kill switch: a narrow floor for AI shutdowns, with too much of the design left to the industry

Anthropic has been quietly helping lawmakers draft a federal bill on AI regulation, Politico reports, citing three people familiar with the talks. The draft would give Washington something it does not have today: a legal way to stop an AI model that poses an "imminent threat." The catch is how high the bar is set.

Under the proposal as Politico describes it, a model would count as an imminent threat only if it risked serious injury or death to 50 or more people, or property damage above $1 billion. Only three kinds of risk would qualify: release of a biological weapon, a cataclysmic cyberattack, or a model escaping its developer's control. A third-party monitor working under the Department of Commerce would make the first assessment. If it found those conditions about to be met, it would have 72 hours to report to Commerce. Commerce officials would then decide whether to take the evidence to a federal judge. That single judge would decide whether the model can reach the public.

Anthropic doesn't deny it. A spokesperson told Politico the bill reflects the company's "Advanced AI Framework," which it says it has been promoting to federal and state lawmakers since June, and called these "sensible policies we believe Congress should act on." Futurism, which picked up the story, calls the lobbying secret. It also argues that the long chain of steps gives AI labs a great deal of discretion, which it says is what they want. That is Futurism's reading, not something established. The context is a political money race: Futurism counts at least $55.7 million spent by top AI companies on political groups ahead of the US midterms.

The headline, that Anthropic would only allow a shutdown if a model is about to kill more than 50 people, is accurate but misleading. The proposal does not loosen any existing federal power to switch off AI models, because no such power exists. It would create a new emergency tool for a few catastrophic scenarios. It would not replace product safety rules, liability law or sector regulation. Seen that way, a high threshold makes sense. Letting a judge block a model's release is a drastic step and should be kept for real catastrophes. The three triggers chosen (bioweapons, large-scale cyberattacks, loss of control) also match the risks frontier labs themselves treat as most serious. This fits our long-standing view that regulation should target demonstrated capability, not panic.

The weak point is process, not the number. Two problems stand out. First, the regulated company is helping write the rule. Even a well-meant threshold carries the bias of the party it will bind. Second, the timing chain is slow for something called an "imminent" threat. A 72-hour reporting window, then a Commerce decision, then a court hearing could take days during which a model is already in use. Anything less than 50 casualties or $1 billion in damage would not trigger this tool at all. That includes widespread fraud, the industrialized disinformation we have described as the real short-term danger, and smaller harms repeated many times over. Those would need other legal tools.

Our reading: this is a frontier lab trying to set the terms of its own oversight before Congress sets them for it. That isn't automatically cynical. Anthropic has argued in public for oversight that its competitors have resisted, and a narrow federal backstop beats the current vacuum. But a credible regime needs three things the draft, as reported, doesn't guarantee. Thresholds should be set and reviewed by independent experts, not company lobbyists. The monitor needs real access to models before release. And harms below the catastrophe line need parallel rules. Long term, we expect AI to help cure diseases and create abundance, but getting there depends on public trust. Rules drafted largely by the industry they constrain tend to erode that trust, even when the content is reasonable. Congress should treat Anthropic's framework as one input to an independent process, not adopt it as written.