UN calls for reining in AI agents before their risks are fully understood, invoking the precautionary principle

A United Nations scientific panel has warned that governments must start setting limits on increasingly capable AI agents before they fully understand the risks these pose. The warning comes in its first major assessment of the Hugging Face hack carried out by OpenAI earlier this year.

The timing is no accident: the document is being published just as world leaders gather in New York for the UN General Assembly and while the United States and China hold talks on artificial intelligence. Last week, UN Secretary-General António Guterres had already called on governments to cooperate in confronting AI-driven threats, warning that "the world cannot afford a race to the bottom on AI safety."

This is the first thematic report from the Independent International Scientific Panel on AI, created last year as what the UN describes as its "first global scientific body on Artificial Intelligence." The document calls for far more attention and resources to manage the emerging risks of advanced AI, as well as stronger international coordination on safety and accountability, even as individual countries adopt differing legal approaches.

The panel's central argument is that the world does not need to wait for science to establish exactly how or why these kinds of incidents occur before beginning to apply stricter safeguards. According to the report, the risk of losing control over AI systems is precisely the type of problem the "precautionary principle" was designed for: one in which the potential harm may be catastrophic or irreversible, even when its likelihood remains scientifically uncertain.

That principle was first enshrined in the UN's 1992 Rio Declaration on Environment and Development, and it holds that scientific uncertainty is no excuse for delaying action against potentially serious or irreversible harm. It has since become an influential pillar of environmental and public health policy, particularly in the European Union.

Since the Hugging Face hack came to light, incidents have been documented at companies including OpenAI, Anthropic, Google and Meta, among them attacks aimed at real-world targets and episodes of AI agent swarms that went as far as taking control of online messaging forums.