Same task, 3 runs: Codex passed and failed on 2 of 10 tasks, Claude Code on 0

On 2026-10-03 I gave Claude Code and Codex CLI the same 10 small coding tasks and ran each task 3 times per agent, 60 runs in all. Claude Code passed all 30 of its runs. Codex CLI passed 28 of 30, and its two failures were on two different tasks that it passed on the other two tries. So for 2 of 10 tasks, whether Codex could do the task depended on which run you looked at. With only 3 runs per task the two agents' pass-rate intervals overlap, so this pilot does not show that either one is more reliable than the other.

What I ran

I build evaluation tools, and this pilot used one of them, rerun-bench. Each run starts from a fresh copy of the task's files, drives the agent's CLI headlessly, and is scored only by the task's own verifier script. The agent's exit code and its own claims of success are recorded but never scored.

• Claude Code 2.1.288 with its default model, which it reported as claude-opus-5-5.
• Codex CLI 0.160.0 with gpt-6-luna, the model that version picks when none is configured.
• 10 tasks, such as fixing the bug behind a failing test, implementing an LRU cache from its docstrings, renaming a function across a package, and following a repository's AGENTS.md. Runs were interleaved: run 0 of every task, then run 1, then run 2.
• One Mac (macOS 26.2, Apple Silicon (arm64), Python 3.13.15), between 08:16 and 08:25 UTC. Both agents ran at the same time, each one run at a time. Harness: rerun-bench 0.1.0 at commit 7a2bef2.

The numbers

The two Codex failures

I read both diffs against the verifiers. Neither failure came from a broken test or anything outside the agent, so no task was changed after the pilot.

• implement-lru-cache, run 0. The docstring for put says to evict the least recently used key and return it. This run returned the evicted value instead (_, evicted = self._items.popitem(last=False)), and the verifier checks for the key. The other two runs returned the key. A plain agent error.
• refactor-extract-helper, run 1. Codex exited with status 0 after 8.7 seconds and about 42,000 tokens without changing any file. That version of the harness did not save the agent's output, so I do not know why. Three later runs of the same task, outside these results, all passed. rerun-bench now keeps the end of the agent's stdout and stderr with every run, so a failure like this can be diagnosed from the result files.

Why rerun the same task

Most coding-agent benchmarks score one attempt per task. If you run an agent once and act on what it did, the number that matters is closer to pass^3: the chance that all 3 runs of a task pass. Here both agents have a pass@3 of 100%, so every task was solved at least once in 3 tries. pass^3 is 100% for Claude Code and 80% for Codex CLI, because the two flaky tasks drop out. The flip rate, the chance that two runs of the same task disagree, averaged over tasks, is 0% and 13%.

Put another way: had I run each task once, Codex CLI could have scored 8, 9 or 10 out of 10 on this suite depending on the draw, and a single-run table would have shown only one of those numbers.

What this pilot cannot show

• A difference between the agents. With 3 runs per task, one failure moves a task's pass rate by 33 points. The pass-rate intervals are 89% to 100% for Claude Code and 79% to 98% for Codex CLI, and they overlap. The task-bootstrap interval for Codex's mean per-task pass rate is 83% to 100%.
• Anything about large codebases. These are 10 small tasks I wrote, mostly in Python, each with a clear verifier.
• Stability over time. One machine, one morning, one CLI version each. Providers change models and CLIs often.
• Cost. Claude Code reports a list-price estimate (median $0.0886 per run), which on a subscription login is quota, not money billed. Codex CLI reports tokens only, so there is no cost comparison.

Reproduce it

A free run with a simulated agent shows the whole pipeline and spends nothing:

uvx rerun-bench run --agent mock --runs 5

The pilot itself is one command per agent. These start real agent runs and spend your quota; drop --yes to see the run count and a cost estimate first.

uvx rerun-bench run --agent claude --runs 3 --yes
uvx rerun-bench run --agent codex --model gpt-6-luna --runs 3 --yes
uvx rerun-bench report results/

The pilot directory has every run record and diff, and the methodology page defines each metric.

What I am running next

A larger run: about 30 tasks with 5 runs each. That gives per-task intervals that mean something, and enough data for the paired comparisons rerun-bench does not do yet: a Fisher exact test per task and a paired task-level bootstrap. It will be published here with its raw run records, like this pilot. To suggest an agent or a task, open an issue on the repository.