OpenAI has published a detailed audit of SWE-Bench Pro, one of the most widely used benchmarks for measuring AI coding capabilities, estimating that roughly 30 percent of its tasks are broken[1]. The company had previously encouraged the community to adopt the benchmark, but it is now retracting that recommendation in light of the findings[1]. The announcement is a fresh reminder of how hard it is to measure AI models' real capabilities.

What SWE-Bench Pro Is and Why OpenAI Audited It

SWE-Bench Pro sources its tasks programmatically from the history of feature changes in software repositories, asking AI models to implement a solution that passes new tests for a feature without breaking existing functionality[1]. It was designed to test models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. On the 731-task public split, frontier models improved from a pass rate of 23.3 percent to 80.3 percent in just eight months[1].

There is a precedent behind this audit. OpenAI previously investigated the long-standing SWE-bench Verified benchmark and found fundamental design and contamination issues (where answers leak into training data), concluding that it no longer provided meaningful signal on software development capabilities. At the time, the company encouraged the wider community to switch to SWE-Bench Pro[1]. The new audit applies the same kind of quality check to that recommended successor[1].

OpenAI explains that evaluation results feed into its own deployment and safety decisions, including those under its Preparedness Framework. When evaluations have flaws, they can give a false understanding of capabilities, misrepresenting safety cases and affecting research priorities[1].

A Two-Layer Audit: AI Agents Plus Human Engineers

The audit began with an automated filter that reviewed the instructions given to the model, model attempts at each task, and the tests used to grade those attempts, flagging 286 potentially broken tasks[1]. Codex-based investigator agents then dug into the flagged subset. The agents had access to the task repository and environment, could run tests, inspect files, and investigate common failure modes, with a researcher making the final judgment[1].

In parallel, OpenAI ran a human annotation campaign with experienced software engineers. Each flagged task was reviewed by five engineers who formed independent judgments, with disagreements or low-confidence cases escalated for further review[1].

The issues primarily fell into four categories[1]: overly strict tests that enforce specific implementation details not specified in the prompt, invalidating functionally correct submissions; underspecified prompts that omit requirements the hidden tests enforce; low-coverage tests that under-check the requested feature, letting incomplete fixes pass; and misleading prompts that point models toward the wrong behavior or contradict what the tests require[1].

Notably, human reviewers were more likely than the investigator agents to mark tasks as broken. Reviewers' judgments overlapped with the agent-flagged categories in 74 percent of cases, but for low-coverage tests, humans selected it as the most common issue for 9.4 percent of the benchmark compared with 4.1 percent from the agent pipeline[1]. In no flagged task was "not broken" the most common human label[1].

Recommendation Retracted, With a Call for New Benchmarks

Based on these results, OpenAI estimates that around 30 percent of SWE-Bench Pro tasks are broken and advises model developers to carefully examine results[1]. The company has also retracted its earlier recommendation to adopt the benchmark[1].

OpenAI also points to the structural reasons behind such flaws. Issues and pull requests from open-source repositories were originally created for human collaboration, so problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably[1]. In particular, tests included in pull requests are written to validate a specific change rather than to define an implementation-agnostic standard for solving the task[1].

At the same time, the company notes that evaluation flaws are easier to detect now than before, as improving model capabilities make it practical to inspect prompts, tests, patches, traces, and edge cases at scale[1]. OpenAI is calling on the wider evaluation community to develop new benchmarks built by experienced software developers specifically to test model capabilities[1].

Summary

OpenAI's audit estimates that roughly 30 percent of SWE-Bench Pro tasks are broken, and the company has retracted its earlier recommendation to adopt the benchmark[1]. The fact that a rapidly rising pass rate does not necessarily reflect genuine capability gains is a sobering lesson for anyone tracking benchmark numbers. Building an unbroken yardstick for AI coding ability is shaping up to be a challenge for the entire industry.

出典:https://openai.com/index/separating-signal-from-noise-coding-evaluations