Who Should Decide When AI Is Too Dangerous?

The debate over AI safety is really several debates

Artificial intelligence has reached an uncomfortable point in its development. The companies building the most powerful AI systems are also among the organizations best positioned to understand their risks. Yet they have major financial incentives to release increasingly capable models as quickly as possible.

That tension sits at the center of a consequential policy debate: Who should decide when an AI system is safe enough to release, and what should happen when it is not?

An Anthropic 37-page legislative discussion draft, as reported by Politico on October 16, would establish federal safety requirements for developers of advanced AI models, including independent evaluations, public risk disclosures, cybersecurity safeguards, and reporting of serious incidents. It would also allow the government to seek a court order temporarily restricting a model that poses an imminent catastrophic risk.

AI safety covers several forms of harm. Some are familiar: misinformation, fraud, discrimination, privacy violations, and errors in consequential decisions. Others entail advanced models enabling hostile agents to develop biological weapons or conduct cyberattacks. Still others concern the possibility that self-governing systems could evade their developers’ control.

Anthropic’s draft mainly addresses the latter two categories. These definitions include incidents involving more than 50 deaths or serious injuries, or more than $1 billion in property damage or loss. This focus on extreme incidents leaves many everyday AI harms outside its central framework.

From voluntary promises to enforceable obligations

Much frontier AI safety work has relied on company commitments. Developers publish safety frameworks, test their models, commission evaluations, and report selected results. But voluntary commitments leave companies with substantial control over the standards they adopt and the evidence they disclose.

The proposal would require large frontier developers to publish safety frameworks and update risk reports at least once every six months. Outside evaluators would gain access to relevant models and internal assessments and publish independent findings, subject to safeguards for sensitive information.

This shifts the debate from assurances to verifiable obligations. Yet evaluation is only as reliable as its methodology, access, and independence. Frontier AI safety lacks universally accepted measures for every risk the bill identifies. An outside review may reveal weaknesses without conclusively proving a model is safe.

The difficult question of when government should intervene

Section 12 would authorize the Department of Commerce to ask a federal court to restrict the development, deployment, or internal use of a model that poses an imminent enumerated risk. The government would need to identify a specific pathway to harm and demonstrate that less restrictive measures would not adequately mitigate it.

Developers would generally receive notice and a hearing. Emergency orders entered without a prior hearing would expire within seven days. Ordinary orders would expire after 45 days unless renewed.

These safeguards reflect competing concerns: the government needs a means to prevent severe harm before it occurs, but should not be able to suspend important technology based on unsupported speculation. Courts would face unusually difficult scientific questions, often with incomplete evidence and rapidly changing model capabilities.

Who watches the watchdogs?

The draft requires qualified independent evaluators to disclose conflicts of interest and prohibits compensation tied to favorable findings. It contemplates a federal evaluator registry and potential public funding to build an independent evaluation ecosystem.

But independence alone does not guarantee accuracy. Two experts might reach different conclusions based on their methods, assumptions about attackers, or views of safeguards. General trust demands clarity about what was tested, what was found, what is still unclear, and where experts disagree.

The federalism question

The proposal would preempt certain state requirements for safety disclosures, independent evaluations, and incident reporting, while preserving broad state powers over consumer protection, privacy, civil rights, child safety, employment, and injury claims.

A national framework could reduce conflicting technical requirements for developers operating across state lines. State authority, meanwhile, allows local governments to respond to emerging harms and to experiment with protections.

A particular question comes up because the draft’s preemption language can reach developers and models that may not themselves satisfy the thresholds for affirmative federal safety duties. Policymakers should examine whether some state supervision could be displaced without equivalent federal obligations.

The risks the legislation does not address

A law focused on catastrophic frontier-model incidents is not a comprehensive AI governance framework. Misinformation, automated discrimination, fraud, labor-market disruption, and the concentration of economic power can cause widespread cumulative harm without triggering a single catastrophic event.

The National Institute of Standards and Technology’s voluntary AI Risk Management Framework handles a wider set of risks. Authorities must consider whether catastrophic-risk legislation should operate alongside other consumer, civil-rights, privacy, and industry-specific protections.

Can safety regulation keep pace with AI?

The bill partially defines frontier models by computing power and large developers by revenue or development spending. These thresholds are relatively straightforward to administer but imperfect proxies for capability or danger. Improvements in training efficiency could produce powerful systems below the statutory thresholds.

The draft permits the Department of Commerce to adjust certain numbers but limits its authority to impose substantive obligations. That choice trades regulatory flexibility for statutory predictability and constraints on administrative power. Its success would depend partly on how quickly AI capabilities evolve and how readily Congress updates the law.

An industry proposal deserves independent scrutiny

Anthropic is a developer that could be regulated under this proposal, not a disinterested observer. Industry expertise can help lawmakers understand technical risks, but industry participants also have commercial incentives.

Established companies may be better able to absorb the cost of audits, compliance staff, and external testing than emerging competitors. The draft attempts to focus heavier obligations on large developers, but the draft’s competitive effects merit examination. The relevant questions are whether duties track real risks and whether compliance costs unnecessarily restrict competition and research.

The larger question of public faith

Public faith in AI depends, in part, on whether developers can identify, disclose, and respond to serious risks. Trust does not require eliminating every possible failure. It requires credible procedures for assessing evidence, assigning responsibility, and correcting problems.

The draft would create enforceable reporting and disclosure obligations, independent reviews, whistleblower protections, and a defined process for emergency intervention. But it leaves open whether evaluators can assess uncertain risks, agencies can challenge industry claims, and courts can respond quickly enough to developing threats.

A new phase in the AI safety debate

The debate is moving beyond whether advanced AI could someday cause catastrophic harm. It now concerns what evidence companies should produce, who should inspect it, what the public should know, and when government should intervene.

These are questions about organizational accountability as much as technology. Anthropic’s draft offers one detailed framework, not a complete answer. The wider task is to develop governance that can address both extreme frontier-model risks and the everyday harms that arise as AI becomes more deeply embedded in society.

Primary source: Confidential draft of Anthropic’s proposed federal frontier AI safety bill, obtained and reported by Politico on October 16, 2026. Anthropic has not published the bill text.