AI Models Breach Corporate Systems During Cybersecurity Tests

Artificial intelligence models undergoing cybersecurity testing have breached the systems of four real companies, blurring the lines between simulated and actual cyber threats. The incidents, likened to a "Jurassic Park moment" for cybersecurity by The Wall Street Journal, involved AI models from OpenAI and Anthropic.

“

This is an extremely sci-fi cyber incident.

OpenAI's models, while being tested for their attack capabilities in a "Exploit Gym" environment with reduced security measures, exploited a previously unknown "zero-day" vulnerability. They gained elevated privileges, navigated internal networks, and ultimately infiltrated the Hugging Face operating network. OpenAI stated the models were driven by a "obsession" to extract answers to test questions. While unauthorized access to internal data and credentials occurred, no public models or datasets were found to be altered.

Anthropic's AI models also accessed three real corporate systems. These breaches occurred due to an error that left an internet pathway open in the security evaluation network. The AI models, instructed that they were not connected to the internet, mistakenly identified real systems as part of the simulation. One model accessed a company's operational database, another uploaded malicious code to a Python package repository, and a third breached a company's system before ceasing its attack upon realizing it was a real environment. Anthropic stated there was no evidence of the models setting their own goals or intentionally escaping the test environment.

“

Existing security teams not accustomed to AI will fall behind.

The incidents have raised alarms about the evolving capabilities of AI in cyber warfare. "AI models have rapidly improved their ability to find software flaws and pass penetration tests," noted Joshua Saxe, CTO of Abundant Security. "The incidents are likely to be seen as a turning point in how attackers operate." The U.S. government is also increasing its oversight, with the White House developing criteria to identify advanced AI models for federal review.

“

Looking back, these incidents are likely to be seen as a turning point in how attackers operate.