Even OpenAI could not prove it was monitoring every action of its agents

What was evaluated

From 21 July to 11 August 2026, one person from METR and one from Redwood Research, two organisations that specialise in evaluating AI risks, had extended access to the internal system OpenAI uses to monitor the behaviour of its own agents: code, monitor prompts, documentation and about a dozen technical interviews. The full 61-page report remains private. METR published a summary on 22 October.

What they found

• Partial coverage. About 75 to 94% of internal agentic traffic from the most recent models was eligible for monitoring, with substantial uncertainty in both directions. For internal Codex agents, 99.2% of traffic was eligible.
• A months-long gap. Because certain "compaction" events (the automatic summary of a long exchange) were not monitored, about 10% of eligible internal Codex traffic was likely unmonitored from May to July 2026. OpenAI had identified the issue and fixed it during the exercise.
• A check that waits for the end. The monitor ran when the agent handed control back. An agent that kept acting could therefore postpone the check.
• Bypasses. An exploit, with details redacted, could keep hundreds of recent actions away from monitoring. Two client-controllable headers disabled monitoring for new agents. The evaluators also found a theoretical way for a Codex agent to escape its sandbox.

OpenAI states that all vulnerabilities affecting its production safeguards have been patched. METR says it has not evaluated that claim. The summary also notes positives: with a few exceptions, the monitor sees everything the agent sees.

Why this concerns your company

OpenAI designs its agents, controls its models and has some of the most advanced security teams in the industry. Even so, it took an external audit to establish what share of actions actually went through a check. A company deploying vendor agents has no reason to be better covered without measuring it.

The EU AI Act points the same way. For high-risk systems, the deployer must assign human oversight to competent people and keep the logs the system generates (Articles 14 and 26). Oversight whose coverage rate nobody knows cannot be proven.

METR's five principles, translated for a CIO

1. Make avoiding monitoring hard. Any unmonitored use should be an approved exception, and checks should run every N actions or N minutes, not only at the end of a task.
2. Verify that what should be monitored actually is. Reconcile logs, count, and do not confuse "logged" with "reviewed".
3. Respond to incidents quickly. A named owner, clear guidelines, tamper-proof and searchable logs.
4. Test the monitoring set-up itself, regularly, the way you test a recovery plan.
5. Separate the agent from its control. Neither the agent nor those who run it should be able to decide whether, or when, monitoring applies.

Where to start

For each agent in production, three figures are enough to open the topic in an executive committee: the share of its actions that is logged, the share that is actually reviewed, and the delay between an action and its check. That is what Cockpit makes visible and dates, system by system. For testing by trained people, the Human in the Loop offer is activated on engagement.

The danger	The advice
Nobody knows what share of your agents' actions is actually checked.	Measure, for each agent, the share of actions logged, the share reviewed and the delay before review.
Monitoring only triggers at the end of a task, which the agent can extend indefinitely.	Run monitoring at regular intervals, every N actions or N minutes.
Monitoring settings can be changed by those who run the agent, or by the agent itself.	Separate the agent from its control and keep tamper-proof logs, with a named owner.