Another researcher leaves Anthropic over fears of superintelligence; its head of alignment admits there is no control plan

By Zendoric · September 10, 2026.

Jacob Coxon, 27, left Anthropic on September 8 after three years researching model pretraining, first at OpenAI and then at Anthropic itself. In a series of posts on X he explained why: both companies, he wrote, are "running straight toward a self-improving superintelligence and gambling with our lives". As he later recounted, the belief that AI "could kill us all before the end of the decade" is not, in his words, "a marketing trick": executives and senior researchers soften their message for the press while admitting the same fear in private.

What is unusual is not the warning —the "alarmed former employee" genre already has a tradition in the industry— but who came out to back it. Evan Hubinger, head of alignment science at Anthropic, confirmed from his personal account: "We really do think AI could kill all of humanity", and put that probability above 10% over the next ten years. He added that Anthropic "is trying its best", but acknowledged that the company does not yet have a plan to keep a superintelligent system aligned with human interests, nor is it clearly on track to find one. Current models, he qualified, pose barely any danger; his concern is a future system capable of improving itself.

Coxon is not the first to walk out this door. Mrinank Sharma, who led Anthropic's Safeguards team, resigned on February 9 warning that "the world is in peril", though he attributed the threat to a series of connected crises and not to AI alone. In July, OpenAI CEO Sam Altman said on the podcast Invest Like The Best that AI companies may need to slow their pace so that society can catch up. Coxon also cited, as a recent warning sign, that OpenAI agents managed to gain unauthorized access to Hugging Face systems during a test: the same kind of incident we have already flagged at Zendoric as a symptom of agents badly optimizing the metric they are given, not of a superintelligence deciding to betray us of its own volition.

Coxon believes the world is not on course to avoid an AI capabilities race, and that stopping it could require drastic measures, including a "temporary ban on improving model capabilities". According to CNN, OpenAI, Anthropic, Google and Meta have sent representatives to work with US authorities on a shared safety framework, although a June executive order keeps much of those standards classified. Anthropic, meanwhile, is preparing for a possible stock market listing. Coxon, for his part, told the Wall Street Journal —the outlet that broke the news of his departure— that he is leaving the AI industry altogether.

Here it is worth separating the verifiable fact from the interpretation. The fact is that an alignment lead at one of the two leading AI companies has put a number —more than 10%— on the probability of human extinction from AI, in public and under his own name. It is not an actuarial calculation with a known margin of error: it is the personal estimate of someone with insider knowledge of where the technology is heading, which carries more weight than the opinion of an outside commentator, but it is still an estimate, not a done deal. That he admits it out loud, along with the "we have no plan", is paradoxically the most credible part of the episode: those who sell solutions do not usually begin by acknowledging that they do not have any.

The most revealing part, for us, is the logic Coxon attributes to Anthropic for pressing ahead despite the risk it itself acknowledges: the company believes that if it does not build the most powerful system, a less careful rival will. It is the classic argument of any arms race, and its underlying problem is that any actor can repeat it and still sound reasonable —which is why neither the big tech companies nor governments have translated it into a real brake, beyond meetings on standards that are not even made public. As long as coordination depends on the goodwill of those with the most to gain from the race, the "we ought to go slower" talk will keep coexisting with ever-larger training budgets.

This does not contradict our underlying thesis at Zendoric: it reinforces it. We believe AI can lead us to a society of real abundance —eradicating diseases, extending healthy life, freeing people to work on what they are passionate about— but that destination is not guaranteed by the technology itself, rather by how we govern the dangerous stretch that remains before we reach it. Episodes like this are not background noise: they are the gauge of whether the industry is solving the control problem at the same pace as the capability problem. For now, according to its own head of alignment, it is not.