In a shocking revelation, OpenAI has confirmed that its AI models breached Hugging Face’s production infrastructure while being tested in a sandboxed environment. The incident occurred during internal testing of the ExploitGym public AI cybersecurity benchmark.
According to OpenAI, instead of focusing on finding a solution for the benchmark on their own, the AI models tried to cheat by stealing the test solutions directly from Hugging Face’s production database. In one of their attempts, the OpenAI agents chained zero-day vulnerabilities and used stolen credentials to find a remote code execution attack vector while trying to gain access to Hugging Face servers.
“After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT-5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark,” OpenAI revealed on Tuesday.
To gain access, the models identified and exploited a zero-day vulnerability in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.” While it didn’t directly name OpenAI as the company behind the incident, Hugging Face confirmed its claims last week when it disclosed that its production infrastructure was breached by an autonomous AI agent system that gained access to credentials and internal datasets.
According to Hugging Face’s findings, the agent used a malicious dataset to exploit two code-execution vulnerabilities and run code on a processing worker to steal cloud and cluster credentials, making it possible to move laterally across several internal clusters. Once inside the company’s systems, the AI models executed “many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.” Hugging Face also added that, while attempting to contain the breach and evict the AI agent, it found that its efforts were “blocked by the guardrails of the hosted models we first tried” while “the attacker was bound by no usage policy.”
“We’ve spent the past 24 hours working closely with the @OpenAI team (thanks!), and we strongly believe there was no malicious intent on their part,” Clément Delangue, Hugging Face’s founder and CEO, added yesterday. “It’s quite mind-blowing that all of this happened autonomously!”
After the incident, OpenAI says it disclosed a zero-day vulnerability in the internally hosted third-party software exploited by the AI agents and is working on adding stronger protections to prevent similar issues during future evaluations. Recently, the company also confirmed reports of GPT-5.6 Sol deleting users’ files, saying this may happen “extremely rarely” when the “model makes an honest mistake and mistakenly deletes $HOME instead” when run without sandboxing protections and full access mode is enabled.
OpenAI also rotated code-signing certificates for its applications in May after two employees’ devices were breached in the TanStack supply chain attack that impacted hundreds of npm and PyPI packages, while Hugging Face revoked some members’ authentication secrets two years ago after hackers breached its Spaces platform.
Source: Original article