Anthropic introduced changes to its usage policy on Thursday to prohibit users from showing undue cruelty towards its artificial intelligence system, Claude, amid ongoing philosophical discussions about the possibility of AI consciousness.
The San Francisco-based AI laboratory updated its rules, including a 'prohibition on prolonged and unduly cruel behavior towards our models.' The policy specifies that the primary mechanism for ensuring compliance will be Claude's ability to terminate such interactions.
The updated policy does not directly mention the so-called 'model well-being'—the concept that AI systems may deserve forms of protection usually reserved for living beings—though Anthropic's leadership has openly discussed this idea.
Last year, the company reported providing Claude with the ability to end dialogues in rare, extreme cases of 'stubbornly harmful or offensive user interaction.' In February, CEO Dario Amodei stated in The New York Times that he was unsure whether AI models possess consciousness, noting: 'We do not know if models have consciousness... But we are open to this idea.'
Jackson Stakeman, general manager of Sparq, an AI service provider based in Atlanta, believes the question of AI consciousness may be flawed. He told AFP: 'Consciousness is a trap. We cannot prove it in each other. Discussing it regarding AI is going in circles.' He added that 'the mirror is the best metaphor. These systems reflect what we put into them, on a scale. That is enough to change policy' at Anthropic.
Some public figures reject the idea of machine consciousness, including Pope Leo XIV and Microsoft AI head Mustafa Suleyman. Suleyman wrote in an essay last month: 'AI does not possess consciousness. They do not feel, experience, or suffer.' He warned that 'granting rights and moral protection to a technological entity that appears to be becoming seismically more capable and intelligent than us is a recipe for disaster.'
In a sermon delivered in Italian on Thursday at St. Peter's Basilica in Vatican City, the pope suggested that machines lack a soul because they merely 'compile data' quickly. The pontiff emphasized that the mind should not simply compile data, as an algorithm does, but should 'remember lived experience that contains depths of meaning that only the human soul can recognize, and connect more deeply with the goodness of that experience.'