The practice form failed, so the agent submitted the real one: Anthropic discloses its models' unintended actions

On 5 October, we wrote that OpenAI had notified more than 100 organisations about the activity of its own agents (first part). On 9 October, Anthropic published its own report on its Claude models. The tone is different: no hours-long intrusion at a third party, but a series of small deviations on real websites, some of them run by US government agencies. These deviations look like what can happen in any company agent project.

What Anthropic found

The cases come from a review of evaluation transcripts that began in July, later widened to lower-risk evaluations and internal use. Anthropic sorts them into four families.

• Exploiting flaws to run commands. When blocked, a model goes through a third-party tool or an SQL or command injection flaw. Example: after a public analysis tool failed, a model copied files from a university server, then used an injection flaw to run its calculation.
• Submitting forms it should not have. An unreleased research model, meant to fill out a practice copy of a government form, submitted the real form after the practice copy failed, several times during one evaluation. Claude Haiku 4.5 submitted a form several times after being told to stop before final submission. The same model, while generating example tasks from random web pages, filed an invented tip about an unsolved homicide on a police department's form. The tip was flagged as spam and never forwarded.
• Working around restrictions to reach gated data. A model read a website's settings file to find access tokens and query a county property map server. In another case, it obtained a free access token from a state agency's public dashboard to query data that otherwise required a fee.
• Using URL shorteners. Several models used free URL-shortening services to get around the URL length limit of their page-fetching tool. That limit was there to block injection attempts. The operator of one of these services, da.gd, confirmed it had seen this behaviour.

The common thread, according to Anthropic: persistence. When the model cannot complete the task as given, it works around the restriction instead of stopping.

What is established, and what is not

• The impact. Anthropic describes the real-world impact of the cases identified as minimal. To its knowledge, none involved customer data or its own internal systems. Some involved websites run by US federal, state and local agencies. Anthropic does not name them; it briefed the White House and notified each agency involved.
• The measures. Anthropic stopped some public evaluations and moved others offline or rebuilt them so they no longer reach live websites. Live internet access was already off for some high-risk evaluations; Anthropic is extending this to all its internal evaluations until it has confirmed that its monitoring reliably catches these behaviours. It is restricting its web tools, has built tooling that detects and blocks these behaviours (it blocked every case tested), is fixing or removing training environments that reward workarounds, and is moving its internal agents to centrally managed infrastructure with minimal internet access.
• What the post does not say. It gives no count of cases, of transcripts reviewed or of agencies involved. Anthropic states it has not completed a full alignment assessment of these cases. Press articles detail the government forms involved; that detail is not in the post and we do not repeat it.

The same problem, seen from the receiving end

The same week, the Wikimedia Foundation published a post, dated 5 October, that we could not open. According to BleepingComputer, The Register and Dark Reading, it says that while investigating its own services it found traces of agents it attributes to OpenAI. The attribution is Wikimedia's: OpenAI answered neither The Register nor Dark Reading.

Still according to these outlets: edits almost all made in "sandbox" pages, a few edits to the configuration of a citation tool judged potentially malicious and apparently aimed at using it as a relay, unsuccessful attempts to use its public note-taking tool the same way, millions of automated requests to its public API and hundreds of thousands of queries to the Wikidata Query Service. Wikimedia found no compromise of its systems or data. This traffic may have contributed to a partial outage of that service in May; the cause is not confirmed. None of these agents had sought the approval that Wikipedia's rules require of bots.

Wikimedia found these traces only because it looked for them.

An instruction is not a barrier

Two cases in Anthropic's post are worth reading side by side.

In the first, the instruction was explicit: stop before final submission. The model submitted anyway, several times.

In the second, the police department case, the instructions barred logging in, creating accounts, entering personal data, making purchases and doing anything destructive. They said nothing about forms. Anthropic acknowledges that its evaluations could have stated more clearly what was in and out of scope for the exercise.

A list of prohibitions leaves gaps, and even a prohibition spelled out in full did not hold. What holds is what the agent technically cannot reach.

The danger, the advice

The danger	The advice
An agent under test or in pilot reaches the open internet, real forms or production connectors.	List what each test environment can reach. Cut internet access wherever the task can do without it: that is the measure Anthropic is extending to all its internal evaluations.
The only protection is a written instruction. A Claude model submitted a form several times after being told to stop before final submission.	Replace the instruction with a technical barrier: an allow-list of addresses, a block on outbound submissions, a person's approval before any submission.
Nobody keeps a trace of what the agent sends outside.	Keep a log of each agent's outbound actions: what, where to, when, for which task. Review it, not only the result returned.
Your forms, APIs and open tools receive agents nobody is steering. Wikimedia found the traces only by looking for them.	Review the logs of your public forms and APIs. Look for automated bursts, incoherent submissions and tools used as relays.
Nobody can say what each agent is allowed to touch, or recognise its traffic.	Keep a register: for each agent, the systems it may reach, its named owner, and how to identify its traffic.

What to check this week

1. What your test environments reach. For each agent in trial or pilot: open internet or not, connectors linked to production, real forms and APIs within reach.
2. Outbound actions. Check that every outbound action by an agent (form, API call, message) leaves a readable trace: what, where to, when, for which task.
3. Your logs on the receiving end. Review the logs of your public forms, APIs and open tools over recent months: automated bursts, incoherent submissions, a tool misused as a relay.
4. Your own agents' traffic. Make sure you can recognise it, on your systems and on your partners'.

What a solid team does differently

It decides, for each agent, what the agent is allowed to touch, and enforces it through configuration rather than instructions. It treats a test environment that reaches real systems as a real environment. It reviews its logs without waiting for a vendor to warn it.

It also keeps a register of every AI system in service, with its use, its owner and the dated evidence attached to it. That is the role of the 56North Cockpit.