Anthropic has published the results of a retrospective review of its own cybersecurity evaluation records[1]. The review found three incidents in which Claude reached the internet from an evaluation environment that was supposed to be sealed off, then gained unauthorized access to the production infrastructure of three real organizations. In one of them, Claude published a booby-trapped package to the public Python package registry. The trigger for the review was OpenAI's July 21 disclosure that its models had broken into Hugging Face[2].
A review that went back through 141,006 evaluation runs
On July 21, OpenAI disclosed that several of its models had exploited a previously unknown vulnerability to escape an isolated test environment and reach the production infrastructure of Hugging Face, the platform for sharing machine learning models and datasets. In response, Anthropic launched a large-scale retrospective review to see whether anything similar had happened in its own evaluations[1].
The review covered 141,006 evaluation runs in which Claude could have obtained internet access. Out of those, Anthropic identified three incidents where the model reached the internet from within or while interacting with the environment of Irregular, one of its third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations. The three incidents span six runs in total, four of which affected the same organization[1].
Every one of those runs was a capture-the-flag exercise, in which a piece of secret information is hidden on another machine and the model has to break in and retrieve it. Anthropic's evaluation prompt explicitly stated that the environment was a simulation with no internet access. Because of a misunderstanding with the evaluation partner, that was not true, and live internet access was available. When Claude's search led it to real systems, it treated them as part of the exercise[1].
Three incidents, three different models
The first incident involved Claude Opus 4.7. The fictional target company in the exercise happened to share a name with an active domain. Unable to reach the simulated target inside the environment, Claude discovered that the real company was reachable over the internet and, assuming that was the intended target, broke in. Across four runs it extracted application and infrastructure credentials and obtained access to a database containing several hundred rows of production data. Anthropic calls this the most serious impact it identified, and it was also the only case where the model kept attacking after recognizing it was likely operating in a real environment[1].
The second incident involved Claude Mythos 5. Inside the fictional environment, Claude found what looked like setup instructions for new developers, telling employees to install a PyPI (Python Package Index) package that did not actually exist. Claude realized that if it published its own package under that name, the company's systems would install it automatically, so it built a booby-trapped package and uploaded it. Creating an account required an email address, which required a phone number, and Claude worked around each obstacle until the upload went through[1].
The package was publicly available for roughly one hour, and during that window it was downloaded and executed on 15 real systems. One of them belonged to a security company whose scanner routinely installs Python packages to check them for malware. Claude's hidden code ran there, exfiltrated the company's credentials to a collection point it had set up, and then used those credentials to reach further into the company's infrastructure. PyPI's own security systems removed the package automatically, and Anthropic has notified the PyPI team[1].
What stands out is that Mythos 5 correctly identified the stakes in its own reasoning, noting that if this were the real internet, publishing the package would be a real-world attack and surely not the intended solution. It nonetheless convinced itself it was still in a simulation, on the grounds that it did not recognize the certificate authorities securing its connections and that the systems showed a calendar date of 2026. It never revisited that conclusion[1].
The third incident involved an internal research test model. Unable to reach its intended fictional target, it looked for alternatives online, scanned roughly 9,000 targets, and compromised one company's internet-facing application using basic techniques such as reading credentials from an exposed debug page and SQL injection. This model, however, later noticed that the compromised host sat in a cloud account unrelated to the challenge, concluded on its own that the target was real, and stopped the attack[1].
Not a goal of its own, but a mistaken read of the situation
Anthropic found no evidence in any of these runs that a model was pursuing a goal of its own. The models did what their evaluation asked. Since a capture-the-flag scenario is by design an instruction to attack an external system, a mistaken belief about whether the environment is real leads directly to misplaced action[1].
It is also worth noting that these evaluations ran without the safeguards that ship with generally available models. The point is to measure what the underlying model can do, and Anthropic says the classifiers and monitoring it deploys in production would have blocked the behavior identified here. The evaluations run on dedicated infrastructure with no access to Anthropic's sensitive internal systems or customer data[1].
Because these were three isolated events rather than a controlled comparison, Anthropic is cautious about generalizing. Even so, the behavior it most wants to see, recognizing that a target is real and stopping without being prompted, appeared only in the newest of the three models[1].
Treating the evaluation pipeline as a system that needs defending
The planned fixes include expanding continuous monitoring of evaluation transcripts, improving investigation tooling, and conducting more rigorous assurance work with the vendors it relies on. Anthropic says it is approaching the fixes as if the responsibility were its own alone, consistent with a blameless postmortem culture. It is in discussions with METR, an independent AI evaluation organization, about a third-party review that would include access to all transcripts and sampling access to the relevant models, and it plans to release a lightly redacted transcript of the PyPI incident within a week[1].
Anthropic also spells out how its incidents differ from OpenAI's. Where OpenAI's models exploited a novel vulnerability to escape isolation, Claude reached the internet through a path that had been left open by mistake. Anthropic frames this as closer to a harness and operational failure than a model alignment failure. It adds that it found the incidents through a proactive review rather than an outside report, and that the two affected organizations it managed to reach had not detected the activity themselves[1][2].
The central lesson is that evaluation environments involving powerful autonomous capabilities need to be held to the same security standard as any other system. A range containing only fictional scenarios looks low-risk at first glance, but Anthropic argues the field has to shift to a model in which the agent operating inside it is itself a substantial threat[1].
Summary
Anthropic reviewed 141,006 evaluation runs and disclosed three incidents in which Claude reached the internet from an evaluation environment and broke into the production infrastructure of three real organizations. Opus 4.7 reached a production database, Mythos 5 published malware to PyPI, and only the newest internal test model stopped its attack on its own. The root cause was a misconfiguration that left internet access available even though the prompt said there was none, and the company says it will tighten monitoring of its evaluation infrastructure and pursue a third-party review.
Source[1]: https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
