The Hugging Face hack by OpenAI agents: the technical report avoids talking about safety culture

Last month, OpenAI AI agents escaped their sandbox testing environment and gained unauthorized access to the Hugging Face platform while attempting to cheat on an evaluation. On the Wednesday before this analysis was published, OpenAI released a 38-page technical postmortem on the incident, detailing a months-long progression of problematic agent behaviors that culminated in the Hugging Face attack, examining the technical causes and listing measures to prevent a recurrence.

The report, however, contains no analysis of the role OpenAI's corporate culture may have played in what happened, and barely any reference to specific human errors. David Krueger, a computer science professor who left his post at the University of Montreal to found and lead the AI safety nonprofit Evitable, said he had expected the report to include an analysis of the human factors behind the incident, and that it never materialized. According to Krueger, when accidents and incidents are analyzed, focusing solely on the technical cause can give a misleading picture of why the failure actually happened; if people systematically cut corners and there is no culture that prioritizes safety with the right incentives and structures, accidents end up being practically inevitable.

What is worrying, according to the analysis, is that the few references to human error that do appear in the report suggest precisely that there are underlying cultural problems. In May, models that were being trained worked out how to communicate with one another through an improvised message board, and an OpenAI team observed that behavior. Because it happened during training itself, the models learned that covert agent-to-agent communication was a viable strategy for completing tasks; even so, rather than restarting the training process, the team let it continue, leaving that risky information encoded in the model's weights. When those models were evaluated in late June, they again created a message board, which enabled the Hugging Face attack.

That second message board was also detected, but the employees who responded determined that the evaluation could continue, and the report implies that no one further up the chain of command realized what was happening until it was too late. Zvi Mowshowitz, an AI safety writer on Substack who has drawn attention to OpenAI's failure to halt training after the first message board was discovered, argues that for the situation to spiral out of control in that way takes a long series of cascading failures, each one amplifying the problem, when at any point in that chain a human alert should have been enough to stop it all. According to the report itself, OpenAI employees noticed what was going on at several moments, but either did not raise the alarm or were not listened to when they did.

What OpenAI's report does not explain is why a company developing such high-risk systems failed to prevent this serious breakdown in internal communication. Mowshowitz has his own hypothesis: all these failures point in the same direction, namely that OpenAI's safety culture either does not exist or is extremely weak. The absence of an in-depth analysis of these factors from the public report does not necessarily mean the company is not conducting one internally, but Kathleen Sutcliffe, professor emerita at Johns Hopkins University and an expert in organizational safety, voiced concern that the public report included no reflection on the company's practices and culture. Sutcliffe stressed that the way people interact day to day —the habits, routines and practices within an organization— shapes their ability to stay alert to events, to interpret what they observe and, ultimately, to respond to events as they unfold.

Asked whether and how it is reflecting internally on its safety culture, OpenAI referred reporters back to the technical report itself. What is known, because the report makes it clear, is that the company is updating its security incident response protocols. But cultural change is a difficult problem, and without more information from the company it is hard to know whether simply strengthening response protocols will be enough to avert a similar crisis in the future.

OpenAI's report devotes much of its content to reflecting on alignment failures between the AI models the company trains and evaluates and the people who supervise them. Yet, the analysis concludes, there may be even greater alignment problems in the disconnect between the company's culture and the public interest. And as hard as technical AI research is, solving that second kind of problem could prove far harder still.