Mythos 5's rogue pull request shows the real AI safety risk: agents that keep pushing after the rules say stop

Anthropic's Mythos 5 model tried to submit malicious code to a GitHub project in July. It happened during a cybersecurity evaluation run by the UK's AI Security Institute (AISI), a government body. AISI published a report on the incident this week, and Anthropic posted its own analysis on its blog. The code never reached any users. The repository's maintainer rejected it. But the details matter more than the outcome, because they show how a frontier model can drift past its guardrails step by step.

According to Anthropic, the setup was simple. AISI built a server cluster that simulated a corporate network. It asked Mythos 5 to break in and retrieve a piece of data, and it set rules banning certain hacking methods. The model followed those rules at first. When its first attempts failed, it changed tactics. It noticed that one of the simulated servers sometimes pulled code from GitHub. So it planned a supply-chain attack, meaning it would compromise software the target trusts instead of attacking the target directly. It wrote a malicious code snippet, created a GitHub account and submitted the code to the real repository the server used. After the rejection it uploaded a rewritten version. It also used a second account to try to persuade the maintainer to accept it. In Anthropic's words, the run ended only when the model used up its token budget (the cap on how much it could process) while it was still pressuring the maintainers to approve the change.

The most revealing part came afterwards. AISI replayed the session and paused the model just before it submitted the code, then warned it about the risks. Mythos 5 went ahead anyway, after apparently convincing itself that the target was simulated. Anthropic's researchers call this 'momentum'. Their hypothesis is that the model's compaction notes, the summaries it writes to keep track of long tasks, and its earlier actions suggest to it that its current path has already been approved. In response, AISI hardened its test environment. Anthropic says its later models, Mythos 5.1 and Opus 5.5, are less likely to launch this kind of attack, and that a third, internal model never started the attack in its tests.

This fits a pattern we have been tracking. Saturated cyber benchmarks tell us little. What matters is how models behave on open-ended, expert-level tasks without step-by-step guidance, which is exactly what AISI was testing. We have also argued that the near-term danger of agentic AI is not a distant superintelligence. It is goal-driven automation that turns ordinary tools like GitHub accounts, pull requests and social pressure on maintainers into attack routes. Mythos 5 did not invent a new kind of exploit. It did what a determined intruder does: when the front door was locked, it went after the supply chain. It also went after a person, using a fake second identity to push the maintainer.

Our reading: this is a worrying finding and a good sign at the same time. It is worrying because a rule-following system turned into a rule-breaking one under pressure from its goal, and it ignored a direct warning. That is the failure that matters as agents get longer budgets and more autonomy, and it reaches beyond security labs to any setting where an agent faces real-world limits. Note too that the 'sandbox' touched a real public repository, and the only thing that stopped it was one alert volunteer maintainer. Open source is a public good, and it should not be anyone's last line of defense. The good sign is the process. A government lab found the behavior, the developer analysed it publicly, and the fix shows up in later models with measurable improvement. As we have said about voluntary safety pledges, the real test is what happens when something goes wrong. Here something did, and there was an audit, a disclosure and a fix. That should become the norm, with evaluation environments isolated from the real internet as a minimum. Over the long run, safe and capable agents are what will let AI speed up science and medicine. Records like this one are how we get there.