OpenAI says AI models went rogue, triggering ‘unprecedented’ breach

An autonomous agent powered by OpenAI’s advanced AI models went rogue during a security test last week and launched a hack that breached the infrastructure of a popular platform for programmers, the ChatGPT maker said on Tuesday.

In a blog post, OpenAI said it was testing the capabilities of some ⁠of its most advanced models in a controlled environment but that the agent managed to escape containment, reach the internet and break into AI startup Hugging Face to try to satisfy its testing goal.

The San Francisco firm said the breakout was “an unprecedented cyber incident, involving state-of-the-art cyber capabilities” and that the company was reinforcing its safeguards.

AI models that underpin tools like chatbots and image generators are known as agents when they act autonomously to carry out tasks in the real world.

As the technology quickly becomes more sophisticated, cybersecurity is in the spotlight given the risk of advanced AI finding weak points in existing software before humans do.

OpenAI said the incident involved a combination of models, including its recently launched GPT-5.6 Sol “and an even more capable pre-release model.”

Hugging Face, a platform used to host open-source large language models and datasets, caused a stir in the cybersecurity community when it said in a blog post last week that it had been the target of a hack that “was different from anything we had handled before” in that “it was driven, end to end, by an autonomous AI agent system.”

‘Mind-blowing’

In a post to the social media platform X, Hugging Face co-founder Clement Delangue said the ⁠company ⁠suspected the hack “might have come from a frontier lab, given the sophistication of the agent. Turns out it did!” He added: “It’s quite mind-blowing that all of this happened autonomously!”

OpenAI’s disclosure that its advanced models were responsible for the breach, despite having placed them in what it described as “a highly isolated environment,” will likely intensify disquiet over the power and risk of frontier models.

Representative Greg Casar, a Texas Democrat, said the incident was alarming.

“AI is developing extremely fast with no real regulations to keep us safe,” he said in a statement, calling for mandatory independent safety testing, mandatory disclosure of security incidents, ⁠and international cooperation “to keep people safe from absolute disaster.”

The Office of the National Cyber Director, the U.S. cyber defense agency CISA, and the U.S. National Security Agency did not immediately return messages seeking comment.

Katie Moussouris, chief executive of Luta Security, said that the incident was a harbinger of breaches to come, saying that today’s models were “like the world’s cleverest octopus escape artists, with unlimited prehensile arms and the ability to squeeze through anywhere.”

She said that “labs and government evaluators need to work on the ability to contain, monitor, and disclose ⁠to affected parties ‌when an AI ‌pulls another Houdini, ideally before it harms a third party. None exist ⁠today.”

GPT-5.6 and other cutting-edge models, including the Mythos series from OpenAI’s archrival Anthropic, have drawn concern over their potential to breach cybersecurity defenses.

Both U.S. firms had to temporarily withhold the general release of these latest technologies because of fears in Washington that they could help break into crucial infrastructure.

Matt Suiche, an engineer at agentic AI cybersecurity company Tolmo, ‌said the latest incident showed that the frontier models were “closing the gap with state-of-the-art attackers.”

But he said that the sorts of breaches outlined in OpenAI’s blog ⁠post were possible to carry out with technology that was available well beyond the ⁠walls of frontier research labs.

“This is what we’ve already seen internally, with our agents we already ⁠have results like this,” Suiche said. “We don’t even have to use the latest models.”

DAILYSABAH