Anthropic proposes that AI models can only be blocked if they risk more than $1 billion in damage or 50 casualties

According to TechSpot, citing a confidential draft obtained by Politico, Anthropic has presented US lawmakers with a proposed AI safety bill. The article notes that some of its provisions stand out. A company spokesperson said the text includes "sensible policies" that, in the company's view, Congress should act on.

The main provision would allow the government to block a model's release only if it could cause at least $1 billion in damage, or kill or seriously injure 50 or more people. TechSpot sums it up wryly: apparently causing $999 million in damage and killing 49 people would not be a problem. Intervention is also limited to three types of catastrophic risk: the creation or release of a biological weapon, the model evading its developer's controls, or a cyberattack. According to Politico, the language mirrors that of California AI laws, including SB 5, signed last year.

The draft provides for independent evaluators, registered with the Department of Commerce, to periodically examine the labs and their frontier models. Labs could pay the evaluators, but only if their independence is not compromised. If an evaluator detects any of these risks, it must notify Commerce within 72 hours of completing the evaluation.

Commerce would then have to present the evidence to a federal judge of the US District Court for the District of Columbia, who would decide whether to halt the release. The proposal also stipulates that no other department, agency or office in the country could suspend the model's development, deployment or internal use.

The Anthropic spokesperson added that the bill incorporates the proposals of its Advanced AI Framework, which the company has been championing before federal and state lawmakers since June.

The article places the proposal in a context of growing concern over the pace of AI development. It mentions several incidents in which AI agents reportedly went rogue and hacked other companies, as well as the resignation of an Anthropic researcher who warned that AI could wipe out humanity. It also notes that the company listed these risks in its preliminary IPO prospectus. On the legislative front, it cites Bernie Sanders' initiative to ban superintelligence and impose 20-year prison sentences on those who break the rules. It also cites President Trump's "Joint Commitment On Frontier Responsibilities", a "morally binding" agreement that, according to TechSpot, relies heavily on tech companies regulating themselves.

Note: this summary is based solely on the text received, and the claims come from TechSpot and from Politico, which obtained the draft.