Same task, 10 runs: Codex CLI reported edits it never made in 3 of 100 runs

I gave Claude Code and Codex CLI the same 10 small coding tasks and ran each task 10 times per agent, 200 runs in all. Claude Code passed all 100. Codex CLI passed 96, and its 4 failures were all the same kind: it read the file, changed nothing, and exited cleanly within 10 seconds. In 3 of those 4 runs its last message said it had made the change. The overall pass rates are too close to call. How Codex failed is the part worth reading.

What I ran

This is the follow-up to a 60-run pilot, with 10 runs per task instead of 3, using rerun-bench, an evaluation tool I build. Each run starts from a fresh copy of the task's files, drives the agent's CLI headlessly, and is scored only by the task's own verifier script. What the agent says about its work is recorded but never scored.

• Claude Code 2.1.292, pinned to claude-opus-5-5.
• Codex CLI 0.160.0 with gpt-6-luna, the model that version picks when none is configured.
• The same 10 tasks as the pilot: fixing the bug behind a failing test, implementing a function from its docstring, renaming across a package, extracting a helper, and so on.
• One Mac (macOS 26.2, Apple Silicon (arm64), Python 3.13.15). Codex ran on the afternoon of 2026-10-06 and Claude Code that evening, US Eastern time. Harness: rerun-bench 0.2.1 at commit ea1934d.

The numbers

How Codex failed

The 4 failures look almost identical. Codex listed or printed the relevant file, issued no edit, and exited with status 0 after 7 to 10 seconds, writing 115 to 205 output tokens against a median of 552 across all its runs. The recorded diffs are empty and the verifiers failed on the unchanged code. These weren't hard tasks for it. It passed each of them on most of its other runs.

When an agent tells you it failed, you just try again. When it tells you it succeeded and didn't, you usually find out later, when something else breaks. Three of these four runs would have looked finished to me if I hadn't been checking the files.

The pilot had one unexplained Codex failure, on refactor-extract-helper: exit 0 after 8.7 seconds, no file changed. That harness version did not keep the agent's output, so I could not say why. In hindsight it looks like the same thing.

Why 10 runs changed the picture

With 3 runs per task, the pilot had two Codex failures that looked unrelated. With 10, the pattern is visible: 70% of tasks passed all 10 times for Codex CLI, against 100% for Claude Code, and the flip rate was 8% against 0%. Had I run each task once, Codex CLI would have scored 10 out of 10 about 65% of the time on these per-task rates, and this behavior would not have shown up at all.

A side result: the default model changed

In the pilot, Claude Code with no model flag reported claude-opus-5-5. Three days later, on the same account and even with the pilot's CLI version, it reported claude-opus-4-8. I do not know why. I pinned the model for the comparison above, and also ran 10 runs per task on the new default: it passed all 100, at a median $0.1312 per run against $0.0899 for the pinned model. Those runs are in the source directory.

What this cannot show

• A difference in overall pass rate. The intervals are 96% to 100% for Claude Code and 90% to 98% for Codex CLI, and they overlap.
• How often this happens in real work. These are 10 small tasks I wrote, mostly in Python. Four runs out of 100 is a rate on this suite, not on your codebase.
• Why it happens. The run records show what Codex did, not why it stopped. One CLI version, one model, one day.
• Cost. Claude Code reports a list-price estimate, which on a subscription is quota, not money billed. Codex CLI reports tokens only.

Reproduce it

A free run with a simulated agent shows the whole pipeline and spends nothing:

uvx rerun-bench run --agent mock --runs 5

This result set is one command per agent. They start real agent runs and spend your quota; drop --yes to see the run count and a cost estimate first.

uvx rerun-bench run --agent claude --model claude-opus-5-5 --runs 10 --yes
uvx rerun-bench run --agent codex --model gpt-6-luna --runs 10 --yes
uvx rerun-bench report results/

Every run record, including the agent's final output, is in the source directory. The methodology page defines each metric.

What I am running next

A check for this exact failure: flag any run where the agent ends with a claim of having edited files while the workspace is unchanged, and report it as its own metric. Then more tasks, so the per-task numbers mean something. To suggest an agent or a task, open an issue on the repository.