DistantNews
Support us
🇰🇷 South Korea /Technology

AI firm Anthropic urges development pause, warns of losing human control

From Hankyoreh · () Korean

Translated from Korean, summarized and contextualized by DistantNews.

At a glance

News Sources not specified New plan
  • AI company Anthropic has urged other AI research labs to consider slowing down development, citing concerns about rapid advancements.
  • The company warns that AI systems could soon reach a point of recursive self-improvement without human intervention.
  • Anthropic suggests a temporary pause in development, drawing a parallel to arms control treaties, to allow time to manage the potential risks of advanced AI.

The artificial intelligence company Anthropic has issued a call for a global slowdown in AI development, expressing concerns that the rapid pace of advancement could soon lead to AI systems capable of self-improvement without human oversight. The company highlighted the potential for AI to reach a stage of "recursive self-improvement" much sooner than anticipated.

It would be a good thing to slow down the development of artificial intelligence globally.

— AnthropicAnthropic's statement on the potential benefits of slowing AI development.

In a blog post, Anthropic shared internal data illustrating the swift progress of AI models, suggesting that a pause in development would be beneficial. While acknowledging the immense potential benefits of AI in fields like science and medicine, Anthropic also warned of the increasing risk of humans losing control over these powerful systems. The company emphasized that this rapid advancement, if unchecked, could pose significant dangers.

Most institutions may reach it earlier than they have prepared for.

— AnthropicAnthropic's warning about the timeline for AI reaching recursive self-improvement.

To mitigate these risks, Anthropic proposed that multiple research institutions agree to temporarily halt development under mutually verifiable conditions. They drew a parallel to arms control treaties, such as the Intermediate-Range Nuclear Forces (INF) Treaty between the U.S. and the Soviet Union during the Cold War, as a model for international cooperation. However, Anthropic acknowledged the time constraints, noting that "these regimes took decades to build infrastructure and trust, but we do not have that time."

This increases the risk of humans losing control over artificial intelligence systems.

— AnthropicAnthropic's concern about the potential loss of human control over advanced AI.

Anthropic plans to establish a forum involving policymakers and civil society to discuss the implications of full recursive self-improvement and coordinate responses. This initiative reflects a growing concern within the AI community about the ethical and safety implications of increasingly sophisticated artificial intelligence.

These regimes took decades to build infrastructure and trust, but we do not have that time.

— AnthropicAnthropic's comparison of AI development controls to historical arms control treaties, highlighting the urgency.
DistantNews Editorial

Originally published by Hankyoreh in Korean. Translated, summarized, and contextualized by our editorial team with added local perspective. Read our editorial standards.

More Stories

From Our Blog