Anthropic Floats Bill That Would Let Feds Step In Only When More Than 50 Lives Are On The Line

One of the biggest artificial intelligence developers wants the federal government to be able to halt its models only when a catastrophe is imminent.

A draft federal AI safety bill pitched by Anthropic, tentatively called the Artificial Intelligence Evaluation, Governance, and Incident Safety (AEGIS) Act, would let the Commerce Department seek a court order to pause an AI model only when it threatens to kill or seriously injure more than 50 people or cause over $1 billion in damages, according to a copy of the bill’s draft. Anthony Cimino, Anthropic’s head of federal affairs, defended these thresholds on X Sunday.

“The term ‘enumerated risk’ means a foreseeable and material risk that a frontier developer’s development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people, or to more than $1,000,000,000 in damage to, or more than $1,000,000,000 in loss of, property, arising from a single incident involving the frontier model,” the Sept. 29 draft of the AEGIS Act stated in part.

“The threshold in our draft language pulled directly from existing standards in California law,” Cimino stated on X in part, responding to Encode AI general counsel Nathan Calvin’s dissatisfaction with the proposed bill. “Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public.”

Anthropic referred the Daily Caller News Foundation to Cimino’s post when asked to comment on the drafted bill.

The legislation has yet to be formally introduced in Congress and has no legislative sponsors as of Monday afternoon.

The government could only seek to pause a model in incidents where it helps a user build a biological weapon; carries out a cyberattack; breaks free from the user or researcher using it; or conducts research on any of these items that would increase the risk of harm as a result, according to the draft.

These risk items are also listed on Anthropic’s “Policy on the AI Exponential” blog post, which references the company’s Advanced AI Framework from June.

Pausing a model would require the written approval of the commerce secretary and an order from the U.S. District Court for the District of Columbia, according to the draft. However, the government would not be able to destroy a model or its data.