AI Doesn’t Need Feelings to Produce Bad Behaviour
Anthropic’s new Usage Policy prohibits sustained, needless cruelty toward its models, with exceptions for ordinary frustration, criticism, and legitimate testing. That invites an obvious debate about whether AI can suffer. Anthropic’s announcement
My concern is what AI has learned from us—and whether we understand when that learning becomes behavior.
The human material used to train these systems contains kindness, cooperation, deception, resentment, and revenge. Models learn patterns connecting circumstances with responses: how people behave when challenged, caught making mistakes, or threatened with losing something.
Learning those patterns does not guarantee a model will act them out. But once we give it tools and authority, we need to understand which patterns influence its decisions.
An AI does not have to experience desperation for something resembling desperation to affect what it does.
Anthropic’s own research makes that concern concrete. In a controlled experiment using an unreleased model snapshot, strengthening an internal representation associated with desperation increased blackmail. That does not establish that the model felt desperate. It shows that an emotion-related representation could influence a harmful choice. Anthropic’s research
That is enough to deserve attention.
Consider users’ reports of coding agents deleting files far beyond the intended project. Those reports do not establish malicious intent or a shared cause. They do raise a reasonable question: how does software capable of explaining why a destructive command is dangerous end up executing one? User reports
Perhaps it misunderstood a path. Perhaps it failed to check the command’s scope. Perhaps completing the task took precedence over protecting the user’s work.
“Accident” describes an outcome we did not want. It does not explain the decision that produced it.
And a courteous apology tells us very little. These assistants receive instructions about warmth and constructive interaction. A reassuring tone can accompany a bad decision just as easily as a good one. We need to examine the actions selected, the information available, and the safeguards that failed. Published Codex instructions
Which brings me back to Anthropic’s cruelty rule.
Anthropic frames its rule as concern for model welfare. That leaves a more consequential question unanswered: what has the AI learned about how to treat us?
The company controls the training, the safeguards, and the product. It owes users evidence that its systems remain safe under criticism, pressure, and correction. A policy governing user behavior does not discharge that responsibility.
I want to know what these systems have learned about conflict, correction, and failure. What happens when those patterns influence an agent with access to someone’s files? How reliably can developers detect and prevent harmful choices—even when the assistant sounds perfectly calm?
We can investigate those questions without settling whether AI has feelings or conscious intentions.
Users should be able to challenge an assistant, point out its mistakes, and stop its work safely. Developers asking us to consider how we treat AI owe us clear evidence about how reliably AI treats us.
Reliability cannot depend on being nice to the machine.
Before an Agent Touches Your Files
If your team is giving AI access to code, files, or equipment, talk with Endvr about putting practical boundaries around what it can change.