Anthropic Urges California to Let Courts Halt Risky AI Models

Anthropic has proposed giving California authorities the power to go to court to stop the release of AI models that independent evaluations show to be a serious danger. The idea is meant to strengthen oversight of advanced models and limit the potential harms of their use.

The proposal was set out in a letter the company sent to California Governor Gavin Newsom on 15 October, responding to ideas under discussion on AI safety. Anthropic called for a mechanism allowing the state's attorney general to seek an injunction to suspend the deployment of a model, its internal use or its further development, if evaluations reveal an imminent risk of catastrophic harm.

Independent oversight and regular reporting

The plan would let independent bodies that specialize in verifying model safety submit risk reports to the attorney general, so legal action can follow when needed. That would mean risk assessment no longer rests on the decisions of the developers alone. Anthropic also asked developers of advanced models to publish risk reports at least every six months. Independent evaluators would get access to the information they need, roughly comparable to what in-house safety teams see, while customer data and sensitive information remain protected.

Why a kill switch is not enough

The company also addressed the so-called kill switch, which lets a model be disabled quickly when security or operational risks emerge. Anthropic stressed that firms should be able to shut down models they offer to users or run internally. But it warned against treating shutdown as sufficient on its own. Disabling a model entirely may not suit every case, and it can create a false sense of security without a broader system for monitoring and responding to risk.

Instead, Anthropic pointed to circuit breakers: mechanisms that watch the behavior of AI systems, detect problems and intervene, whether at the level of the model, the agent or the user. The company noted that a shutdown does not cover every dangerous scenario, including a model copying its weights to computers outside its developer's control.

A wider debate

The proposal arrives as California weighs stricter AI safety rules amid a broader US debate over high-risk models and developer responsibility. OpenAI also sent proposals to the governor on 20 October, backing measures to control high-risk models along with oversight and auditing mechanisms. Anthropic's stands out for calling for court action to halt deployment of models that pose a grave danger.

Together, the proposals show the debate moving from what AI can do toward clear accountability, independent evaluation and legal intervention, particularly as AI agents capable of carrying out many tasks with little supervision spread.