Anthropic warns: We must stop artificial intelligence from evolving without human intervention

Artificial intelligence development may soon reach a point where AI systems can recursively improve themselves without direct human involvement, according to a warning from the AI company Anthropic. In a blog post titled "When AI Builds Itself," company co-founder Jack Clark and Anthropic Institute leader Marina Favaro highlighted a growing trend where AI systems are increasingly tasked with developing their own successors, significantly speeding up the overall progress of AI.

While acknowledging that this stage, known as "recursive self-improvement," has not yet been reached, the authors cautioned that it could occur sooner than most institutions are prepared for. They stated, "We are not there yet, and recursive self-improvement is not inevitable. But it could happen sooner than most institutions are ready for."

“

We are not there yet, and recursive self-improvement is not inevitable. But it could happen sooner than most institutions are ready for.

The potential for AI systems to independently design and develop their successors raises significant concerns about maintaining human control. The authors pointed out that as AI systems become more capable of building themselves, the methods for ensuring their safety, monitoring their behavior, and shaping their actions become critically important. "If systems have the capacity to fully build their own successors, our methods for securing them, monitoring them, and shaping their behavior become much more important," the post stated.

Given these risks, Anthropic suggests that society should consider slowing down or temporarily halting the development of advanced AI. This pause would allow societal structures time to adapt to the rapid pace of technological advancement. The company emphasized the significance of multiple powerful AI laboratories, operating under similar conditions across different countries, undertaking such a coordinated slowdown.

“

If systems have the capacity to fully build their own successors, our methods for securing them, monitoring them, and shaping their behavior become much more important.