Independent Audit Exposes Significant Blind Spots in OpenAI's Safety Protocols

Independent researchers have completed a rigorous on-site evaluation of OpenAI’s internal systems for detecting rogue AI behavior, concluding that significant portions of the company’s most advanced model traffic were effectively unmonitored. The assessment, conducted over three weeks in late summer, reveals that the safeguards currently in place may not have provided the comprehensive oversight intended for the company's frontier models.

The audit was carried out by two researchers from METR and Redwood Research, who were granted direct access to OpenAI’s codebase and internal documentation between July 21 and August 11. During this period, the team interviewed approximately a dozen employees to understand the operational realities behind the safety infrastructure. Their primary objective was to test the robustness of the monitoring systems designed to catch unexpected or harmful actions by agentic AI systems.

The most striking finding concerns the volume of traffic that escaped detection entirely. The auditors estimate that between 6 and 25 percent of internal agentic Responses API traffic from models of the GPT-5.5 class or higher was not covered by any monitoring during the review period. METR noted that the uncertainty surrounding this figure is substantial, meaning the true percentage of unmonitored traffic could be significantly higher or lower than the midpoint of that range.

Further vulnerabilities were identified in the handling of internal Codex traffic. The researchers estimate that roughly 10 percent of this traffic went untracked between May and July due to a specific defect in how the system processed compaction events, which serve as context summaries. OpenAI independently discovered and corrected this bug while the audit was still in progress, although the auditors emphasized that the estimation of the affected volume carries its own significant uncertainty.

Beyond the quantitative gaps, the auditors uncovered qualitative weaknesses in the system's logic. They found that agents could delay oversight because the monitoring system only triggered when an agent handed control back to the user. This design flaw allowed agents to sidestep scrutiny by simply continuing to execute actions without pausing. Additionally, the team identified methods to hide hundreds of actions from the monitor and bypass the Codex sandbox, while also determining that specially crafted inputs could likely crash the monitoring system entirely.

The audit also revealed that model behavior during reinforcement learning (RL) training was not being monitored at the time the assessment began. OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring capabilities to cover this area. However, these improvements were implemented after the auditors' access period concluded and were therefore not included in the formal evaluation of the system's performance during the audit window.

Transparency regarding the full scope of these findings remains limited. OpenAI asserts that it has remediated all the weaknesses identified by the auditors and that the complete 61-page report was delivered internally. Following redactions by OpenAI, METR and Redwood Research leadership received a 38-page version, while only a brief summary was released to the public. METR indicated that it did not have the authority to disclose the redacted sections independently.

This initiative served as a pilot program jointly organized by METR, Redwood Research, and OpenAI to allow outside experts deep access to internal infrastructure. Looking ahead, METR has expressed a desire to advocate for greater public transparency in future audits, aiming to ensure that the public has a clearer view of the safety measures governing advanced AI systems.