Anthropic bans sustained abusive or cruel treatment of Claude amid the model welfare debate

Anthropic has banned users from engaging in "sustained and unnecessary abusive or cruel behavior" toward its models, according to The Guardian, which credits The Verge as the first outlet to report the policy change. The San Francisco-based company behind the Claude chatbot has written the measure into its online usage policy, while its leaders continue to reflect on the possibility of machine consciousness.

An Anthropic spokesperson did not immediately respond to a query about what conduct would be considered "abusive or cruel," so the exact scope of the ban remains undefined. What the policy does specify is what falls outside it: users' everyday frustrations, model testing and "dark creative themes" would not be affected by the ban.

Anthropic's usage policy already contained restrictions on abusive conduct. In addition, the company's models can end a conversation if a user is persistently harmful. When that feature launched in August last year, the company presented it as a safeguard for AI welfare.

On its website, Anthropic states: "We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we're working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention."

The idea that AI systems could be conscious is polarizing both inside and outside the tech industry. Anthropic CEO Dario Amodei has said he cannot rule out the possibility. At the other end, OpenAI CEO Sam Altman has been skeptical. In a post on X he wrote: "I'm very uncomfortable with people trying to attribute a religious force or an abdication of human judgment to AI models, and I think it's a real safety problem." He posted it days after The New York Times reported on Anthropic executives' extensive conversations with religious experts.

Overall, the change is mainly a clarification and reinforcement of existing rules, framed within Anthropic's cautious stance amid uncertainty over the moral status of its models. It remains to be seen how the concept of "sustained and unnecessary abuse" will be defined and enforced in practice, given that the company has not provided concrete examples.