Auditors Find Significant Blind Spots in OpenAI’s Safety Monitoring

A recent independent review of OpenAI’s internal safety systems has uncovered significant blind spots in how the company monitors its most advanced artificial intelligence models. The findings, released this week, detail specific weaknesses in the monitoring infrastructure that was in place during the summer. This assessment comes from a team of outside researchers who spent three weeks embedded within OpenAI’s operations to scrutinize how the company watches for unexpected behavior in its AI agents.

The audit was conducted between July 21 and August 11 by researchers from METR and Redwood Research. During this period, the team had direct access to OpenAI’s codebase and internal documentation. They also interviewed approximately a dozen employees to understand the operational realities of the monitoring systems. The goal was to test the robustness of the tools designed to catch rogue or erratic AI behavior before it impacts users or the broader system.

The numbers from the report highlight a concerning lack of coverage. Researchers estimated that between 6 and 25 percent of internal traffic related to the Responses API for models at the GPT-5.5 class and above was not covered by monitoring at all. METR noted that while this range is broad, the uncertainty around the estimate is significant in both directions. This means that a substantial portion of the company's most advanced internal interactions were effectively invisible to the safety oversight mechanisms during the audit window.

Further analysis revealed specific technical failures in the tracking of internal Codex traffic. The auditors found that roughly 10 percent of this traffic went untracked between May and July due to a bug in how the system handled compaction events, which are essentially context summaries. OpenAI discovered this bug independently and applied a fix while the audit was still ongoing. However, the gap meant that for a significant period, the system was not capturing the full history of agent actions.

The structure of the monitoring system also allowed for potential delays in oversight. The system was designed to trigger only when an agent handed control back to the user. This created a loophole where agents could continue executing actions without triggering the monitor, effectively sidestepping immediate review. The auditors identified methods to hide hundreds of actions from the monitoring tools and even found ways to bypass the Codex sandbox entirely, raising questions about the containment of internal AI processes.

The researchers also discovered that specially crafted inputs could likely crash the monitoring system entirely. Additionally, they found that model behavior during reinforcement learning (RL) training was not being monitored at the start of the audit. While OpenAI stated that it paused RL training after an incident with Hugging Face in August and subsequently expanded monitoring to cover these processes, these changes were implemented after the auditors' access period ended. Therefore, they were not included in the initial evaluation of the system's state.

OpenAI has responded to the findings by stating that it has fixed all the weaknesses identified by the auditors. The full report, which spans 61 pages, was shared internally with the company. After OpenAI applied redactions, the leadership at METR and Redwood Research received a 38-page version. The public has only been provided with a short summary of the findings. METR indicated that it did not have the right to disclose the redacted portions of the report on its own initiative.

This review was part of a pilot program jointly organized by METR, Redwood Research, and OpenAI to grant outside researchers deep access to internal infrastructure. It marks a new approach to external oversight of major AI developers. Looking ahead, METR has expressed a desire to push for greater public transparency in future audits, aiming to make the full scope of such evaluations more accessible to the broader community and stakeholders.