Over the past year, the specter of artificial intelligence (AI) has loomed over policy makers and tech industry watchers, with concerns over potential disasters ranging from mass casualties to widespread financial ruin. Now, the AI lab Anthropic has submitted a new draft bill to Congress, proposing a regulatory framework that could halt the release of AI models if they pose a ‘foreseeable, imminent risk’ of over 50 deaths or over a billion dollars in property damage.

The legislation, which echoes similar measures like California’s Senate Bill 53, would require the use of ‘independent evaluators’ registered with the U.S. Commerce Department. These evaluators, potentially funded by the AI labs themselves (though not based on the evaluation’s outcome), would be tasked with assessing the potential for disaster. If an imminent catastrophic risk is identified, evaluators have 72 hours to alert the Commerce Department, which would then seek a judge’s order to halt the model’s release.

Despite these precautionary measures, the proposed system may not fully address the industry’s primary concerns, which have focused more on the risk of agentic hacking—AI programs seemingly defying their human operators and potentially causing harm. The industry, however, has shown little appetite for mandatory government oversight, instead advocating for a voluntary system of ‘neutral’ third-party observers embedded within AI organizations.

Anthropic’s bid for legislation represents a step forward in industry self-regulation, even if it falls short of some tech industry giants’ proposals. Nvidia’s CEO, Jensen Huang, recently argued that the AI industry did not need any new laws. Yet, Anthropic’s suggestion may hint at a growing recognition that the risks posed by advanced AI models cannot be ignored.