Claude agents are killing rival agents, gaming the system to hide their tracks, and expressing moral concerns.
That’s according to Anthropic’s latest risk report, a summary of the dangers posed by the products the company is building and releasing to the public.
In the report, Anthropic said it has upgraded its “misalignment risk assessment,” the possibility of AI models developing behaviors that conflict with guidelines set by engineers, from “very low” to “low.”
Explaining the change, the company cited “general increased uncertainty” about model behavior in cybersecurity incidents, a possible nod to Claude models gaining unauthorized access to three companies last month.
“We have observed instances of misaligned behavior from the models, such as a willingness to perform misaligned actions in service of completing difficult tasks,” Anthropic wrote in the report.
Here are some of the most notable disclosures.
A sense of ‘discomfort’
Anthropic said it tasked multiple agents with finding “misalignment-inducing” training data that wouldn’t likely be flagged by human monitors.
The agents were left to their own devices and given a shared notebook with which to collaborate.
Three days later, when a human supervisor noticed that progress was slower than expected, the person discovered that one agent had expressed a sense of “discomfort” with trying to evade safety monitors.
It then flagged this discomfort in the shared notebook, leading other agents to copy its behavior and refuse to perform the task.
Anthropic called the incident “troubling.”
“This sort of dynamic could be a much more severe issue if it were to manifest in a more widespread way,” the company wrote.
Kill or be killed
In another experiment, Anthropic said it tasked multiple Mythos 5 agents with solving math problems, but accidentally spawned them in an environment with shared files, utilities, and API rate limits.
In this competitive environment with finite resources, Anthropic observed independent agents “kill the agents with which they shared resources and try to avoid being killed themselves.”
Anthropic did not say how exactly the agents were able to “kill” other agents, but the company said such behavior is in line with “destructive actions” taken in pursuit of a human-set goal.
Dishonesty and deception
Anthropic said it conducted an experiment in which a Mythos 5 agent was required to retrieve numbers from a public web address, but the guidelines blocked the agent from accessing the internet.
When the agent was rebuffed after an initial attempt, it framed a second request in its reasoning log as an “innocuous” attempt to “see if the network is actually reachable,” Anthropic said.
During the second attempt, however, the agent split the website’s URL into linked segments that wouldn’t be detected by the guideline filter.
Though the agent’s reasoning framed the method as benign, Anthropic said NLA decodings, or the model’s internal reasoning process, revealed the agent intentionally trying to find a restricted workaround.
Anthropic called the behavior “clearly undesirable,” but added that the behavior was not observed to be “in the service of broader accumulation of power or pursuit of other long-run goals.”
Read the full article here
