Anthropic’s Dario Amodei wants governments to have the power to block ‘dangerous’ AI systems

Anthropic PBC Chief Executive Dario Amodei is calling on the U.S. government to block the deployment of dangerous artificial intelligence models in the same way as it prevents unsafe airplanes from taking off.

In a new post today on his personal blog, Amodei (pictured) said there need to be mandatory third-party audits of frontier AI systems, and suggested that governments be able to shut them down if any unacceptable risks are discovered. At the same time, he called on the government to provide more economic support to people who are financially hurt by the rise of AI.

Amodei’s AI safety proposal is far more aggressive than what most people have suggested. It exceeds the executive order issued by U.S. President Donald Trump last week, which mandates that intelligence agencies play a role in testing new models for risks. He suggested that governments should audit new AI models based on the amount of resources thrown into their training process. He proposed there should be a “compute threshold,” and that if any new model exceeds this specified level of compute, it would automatically trigger an independent investigation into its capabilities before it can be launched publicly.

The essay identified four specific risk categories that need to be evaluated: cybersecurity vulnerabilities, biological weapons capabilities, the ability to accelerate automated research in dangerous domains, and the potential for models to grow beyond human control. Should the auditors decide that a model poses an unacceptable risk in one of those areas, Amodei wants governments to be able to prevent their deployment.

Amodei said existing regulations for aircraft, automobiles and drugs should serve as models for regulating AI. “I believe the best analogy, at least at the current stage of the exponential, is to cars, airplanes, or drugs — powerful technologies essential to the modern economy, but capable of killing large numbers of people if designed or operated poorly,” he wrote.

Amodei’s proposed framework goes much further than Trump’s June 2 executive order. The President simply encouraged AI firms to share their models voluntarily with government auditors a month before their public release. The order also emphasized how the intelligence community should be involved in testing new models for cybersecurity vulnerabilities. But it stopped short of enforcing anything, or introducing mandatory testing.

The Anthropic CEO used much stronger language. There’s no mention of the word “voluntary” and he doesn’t ask for cooperation but instead compliance. Whereas Trump’s order leans on intelligence agencies, Amodei calls for independent auditors to be given effective veto power.

Anthropic has been pushing for more stringent controls on AI for a while. Back in September 2023, it published a Responsible Scaling Policy that has since been updated several times, introducing concepts such as AI Safety Levels or ASLs. Essentially, it’s a tiered framework for managing risk as AI model capabilities improve. The company has also voiced its support for other legislation, such as California’s SB 53 bill.

In addition to safety, Amodei also warned that the disruption to the labor market caused by AI’s emergence could be much more significant than earlier technological advances. It could also last much longer, he said. “The key challenge in such a world won’t be incentivizing growth, but finding a way for everyone to share in the benefits,” Amodei wrote.

Amodei said he has long warned of AI displacing jobs, not because he wants to be a “prophet of doom,” but because by doing so, “policymakers and the private sector have the best chance to adapt and respond.”

In his post, he called for the government to collect more data to keep track of AI job displacement, as well as pro-employment incentives that could help to slow this down. He also wants the government to look at “mechanisms such as universal basic income,” if job displacement becomes more profound and permanently reduces labor demand. It could be financed by taxing “relevant companies,” or increasing the capital gains tax, he added.

Photo: World Economic Forum/YouTube