Anthropic urges pause on advanced AI, warns of loss of human control

Anthropic has urged leading artificial intelligence companies to agree on a coordinated mechanism that could temporarily slow or pause the development of advanced AI systems, warning that rapid progress in the field could lead to a loss of human control.

In a blog post published on Thursday, the company behind the Claude chatbot said that as frontier AI systems become increasingly capable of carrying out complex tasks at high speed, it would be “good for the world to have the option to slow or temporarily pause” further development.

Anthropic said its internal research institute would explore the issue in collaboration with other organisations and take unspecified steps towards building mechanisms that could support a credible slowdown or pause.

The proposal comes amid growing disagreement across the industry over how AI development should be governed. Rival firm OpenAI said in a separate report that responsibility should rest with governments rather than private companies acting alone.

“Democratic governments — not private companies acting alone — must ultimately determine the rules, safeguards, and accountability mechanisms,” OpenAI said, adding that no single company or lab should decide the pace of innovation.

Anthropic warned that AI systems are rapidly improving in their ability to perform software tasks independently, including coding. It said that under current trends and sufficient computing power, future systems could potentially design and build improved versions of themselves, a process known as “recursive self-improvement”.

While the company said such advances could bring major benefits in areas such as science and healthcare, it also cautioned that they could increase the risk of humans losing control over highly autonomous systems.

The debate has intensified alongside wider security concerns in the sector. Researchers at the University of Toronto recently demonstrated how AI tools could be used to create a self-propagating “worm” capable of adapting its hacking strategy as it spreads across systems.

Lead researcher Nicolas Papernot warned that cybersecurity risks extend beyond the most advanced models, noting that widely available open-source tools could also be misused to launch large-scale attacks on connected systems.

Anthropic said a coordinated global framework would also be necessary to ensure compliance, allowing leading labs to verify that competitors were genuinely slowing development and preventing bad actors from exploiting a temporary pause to gain advantage.

The company argued that without coordination, less cautious developers could continue advancing rapidly, increasing pressure on others to follow.

Concerns over AI safety have grown as systems become more capable, with some models already demonstrating the ability to identify vulnerabilities in existing software. However, regulation has lagged, particularly in the United States, where most major AI firms are based.

Earlier this week, a US executive order placed greater responsibility on AI developers to voluntarily submit advanced systems for cybersecurity testing before public release.

Calls for a temporary halt to AI development are not new. In 2023, figures including Elon Musk supported an open letter from the Future of Life Institute calling for a six-month pause in advanced AI training, though the initiative did not result in industry-wide action.

Anthropic, which has positioned itself as a safety-focused AI company, has also taken a more cautious stance in specific cases, including restricting certain military uses of its models.

The company’s latest proposal comes as it and competitors including OpenAI continue to expand rapidly, with both firms reportedly preparing for potential stock market listings that could value them among the most valuable technology companies in the world. (NewsWire)

The post Anthropic urges pause on advanced AI, warns of loss of human control appeared first on Newswire.