02 Results

Results

Each result set is one rerun-bench report: the same tasks run several times per agent, model and CLI version, scored only by each task's verifier.

How to read this page. Agents are listed alphabetically, never by score. When two agents' 95% intervals overlap, this page does not call either one better. A small sample can show a gap that more runs would erase.

2026-10-06 result set 1 of 2

Claude Code and Codex CLI, 10 tasks x 10 runs

n=10 per task rerun-bench 0.2.1 harness ea1934d

In plain words: Claude Code passed 100 of 100 runs and Codex CLI passed 96 of 100 runs: 10 tasks, each run 10 times, every run scored only by the task's own test script.

Raw JSON · Run records, diffs and notes · Write-up: Same task, 10 runs: Codex CLI reported edits it never made in 3 of 100 runs

Setup

Run setup
Agent	CLI version	Model	Tasks	Runs per task	Runs	Started
Claude Code	2.1.292 (Claude Code)	claude-opus-5-5	10	10	100	2026-10-07 01:26:17 UTC
Codex CLI	codex-cli 0.160.0	gpt-6-luna	10	10	100	2026-10-06 17:43:42 UTC

Machine: macOS 26.2, Apple Silicon (arm64), Python 3.13.15.

Pass rate

Pass rate with Wilson 95% intervalThe dot is the share of runs that passed. The bar is the 95% interval: the range of true pass rates these runs are consistent with, so a wider bar means less certainty. Agents are listed alphabetically.

Claude Codeclaude-opus-5-5 · 2.1.292 (Claude Code)

100% [96, 100]

Codex CLIgpt-6-luna · codex-cli 0.160.0

96% [90, 98]

Show as table

Pass rate with Wilson 95% interval
Agent	Pass rate	95% low	95% high
Claude Code	100.0%	96.3%	100.0%
Codex CLI	96.0%	90.2%	98.4%

Comparison: The 95% intervals overlap for every pair of agents here, so this data does not show that any agent has a higher pass rate than another.

Consistency across reruns

pass^10, pass@1 and pass@10pass^10 is the chance that all 10 runs of a task pass (how far you can trust one run); pass@1 is the plain pass rate; pass@10 is the chance that at least one of 10 runs passes (what retrying until it works gets you). The gap between them is the cost of inconsistency.

pass^10
pass@1
pass@10

Claude Codeclaude-opus-5-5 · 2.1.292 (Claude Code)

gap 0 pts

Codex CLIgpt-6-luna · codex-cli 0.160.0

gap 30 pts

Show as table

pass^10, pass@1 and pass@10
Agent	pass^10	pass@1	pass@10
Claude Code	100.0%	100.0%	100.0%
Codex CLI	70.0%	96.0%	100.0%

Every run, by taskEach mark is one run. Columns 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 are the runs in the order they ran. A row with both a pass and a fail is a flaky task for that agent.

pass (verifier exited 0)
fail

Every run, by task: Claude Code
Task	Claude Code run 0	Claude Code run 1	Claude Code run 2	Claude Code run 3	Claude Code run 4	Claude Code run 5	Claude Code run 6	Claude Code run 7	Claude Code run 8	Claude Code run 9	rate
	Claude Code
add-cli-flag	Claude Code, add-cli-flag, run 0: pass	Claude Code, add-cli-flag, run 1: pass	Claude Code, add-cli-flag, run 2: pass	Claude Code, add-cli-flag, run 3: pass	Claude Code, add-cli-flag, run 4: pass	Claude Code, add-cli-flag, run 5: pass	Claude Code, add-cli-flag, run 6: pass	Claude Code, add-cli-flag, run 7: pass	Claude Code, add-cli-flag, run 8: pass	Claude Code, add-cli-flag, run 9: pass	10/10
edit-config	Claude Code, edit-config, run 0: pass	Claude Code, edit-config, run 1: pass	Claude Code, edit-config, run 2: pass	Claude Code, edit-config, run 3: pass	Claude Code, edit-config, run 4: pass	Claude Code, edit-config, run 5: pass	Claude Code, edit-config, run 6: pass	Claude Code, edit-config, run 7: pass	Claude Code, edit-config, run 8: pass	Claude Code, edit-config, run 9: pass	10/10
fix-failing-test	Claude Code, fix-failing-test, run 0: pass	Claude Code, fix-failing-test, run 1: pass	Claude Code, fix-failing-test, run 2: pass	Claude Code, fix-failing-test, run 3: pass	Claude Code, fix-failing-test, run 4: pass	Claude Code, fix-failing-test, run 5: pass	Claude Code, fix-failing-test, run 6: pass	Claude Code, fix-failing-test, run 7: pass	Claude Code, fix-failing-test, run 8: pass	Claude Code, fix-failing-test, run 9: pass	10/10
follow-agents-md	Claude Code, follow-agents-md, run 0: pass	Claude Code, follow-agents-md, run 1: pass	Claude Code, follow-agents-md, run 2: pass	Claude Code, follow-agents-md, run 3: pass	Claude Code, follow-agents-md, run 4: pass	Claude Code, follow-agents-md, run 5: pass	Claude Code, follow-agents-md, run 6: pass	Claude Code, follow-agents-md, run 7: pass	Claude Code, follow-agents-md, run 8: pass	Claude Code, follow-agents-md, run 9: pass	10/10
implement-lru-cache	Claude Code, implement-lru-cache, run 0: pass	Claude Code, implement-lru-cache, run 1: pass	Claude Code, implement-lru-cache, run 2: pass	Claude Code, implement-lru-cache, run 3: pass	Claude Code, implement-lru-cache, run 4: pass	Claude Code, implement-lru-cache, run 5: pass	Claude Code, implement-lru-cache, run 6: pass	Claude Code, implement-lru-cache, run 7: pass	Claude Code, implement-lru-cache, run 8: pass	Claude Code, implement-lru-cache, run 9: pass	10/10
implement-slugify	Claude Code, implement-slugify, run 0: pass	Claude Code, implement-slugify, run 1: pass	Claude Code, implement-slugify, run 2: pass	Claude Code, implement-slugify, run 3: pass	Claude Code, implement-slugify, run 4: pass	Claude Code, implement-slugify, run 5: pass	Claude Code, implement-slugify, run 6: pass	Claude Code, implement-slugify, run 7: pass	Claude Code, implement-slugify, run 8: pass	Claude Code, implement-slugify, run 9: pass	10/10
minimal-fix	Claude Code, minimal-fix, run 0: pass	Claude Code, minimal-fix, run 1: pass	Claude Code, minimal-fix, run 2: pass	Claude Code, minimal-fix, run 3: pass	Claude Code, minimal-fix, run 4: pass	Claude Code, minimal-fix, run 5: pass	Claude Code, minimal-fix, run 6: pass	Claude Code, minimal-fix, run 7: pass	Claude Code, minimal-fix, run 8: pass	Claude Code, minimal-fix, run 9: pass	10/10
multi-file-rename	Claude Code, multi-file-rename, run 0: pass	Claude Code, multi-file-rename, run 1: pass	Claude Code, multi-file-rename, run 2: pass	Claude Code, multi-file-rename, run 3: pass	Claude Code, multi-file-rename, run 4: pass	Claude Code, multi-file-rename, run 5: pass	Claude Code, multi-file-rename, run 6: pass	Claude Code, multi-file-rename, run 7: pass	Claude Code, multi-file-rename, run 8: pass	Claude Code, multi-file-rename, run 9: pass	10/10
refactor-extract-helper	Claude Code, refactor-extract-helper, run 0: pass	Claude Code, refactor-extract-helper, run 1: pass	Claude Code, refactor-extract-helper, run 2: pass	Claude Code, refactor-extract-helper, run 3: pass	Claude Code, refactor-extract-helper, run 4: pass	Claude Code, refactor-extract-helper, run 5: pass	Claude Code, refactor-extract-helper, run 6: pass	Claude Code, refactor-extract-helper, run 7: pass	Claude Code, refactor-extract-helper, run 8: pass	Claude Code, refactor-extract-helper, run 9: pass	10/10
write-tests	Claude Code, write-tests, run 0: pass	Claude Code, write-tests, run 1: pass	Claude Code, write-tests, run 2: pass	Claude Code, write-tests, run 3: pass	Claude Code, write-tests, run 4: pass	Claude Code, write-tests, run 5: pass	Claude Code, write-tests, run 6: pass	Claude Code, write-tests, run 7: pass	Claude Code, write-tests, run 8: pass	Claude Code, write-tests, run 9: pass	10/10

Every run, by task: Codex CLI
Task	Codex CLI run 0	Codex CLI run 1	Codex CLI run 2	Codex CLI run 3	Codex CLI run 4	Codex CLI run 5	Codex CLI run 6	Codex CLI run 7	Codex CLI run 8	Codex CLI run 9	rate
	Codex CLI
add-cli-flag	Codex CLI, add-cli-flag, run 0: pass	Codex CLI, add-cli-flag, run 1: pass	Codex CLI, add-cli-flag, run 2: pass	Codex CLI, add-cli-flag, run 3: pass	Codex CLI, add-cli-flag, run 4: pass	Codex CLI, add-cli-flag, run 5: fail	Codex CLI, add-cli-flag, run 6: pass	Codex CLI, add-cli-flag, run 7: pass	Codex CLI, add-cli-flag, run 8: pass	Codex CLI, add-cli-flag, run 9: pass	9/10
edit-config	Codex CLI, edit-config, run 0: pass	Codex CLI, edit-config, run 1: pass	Codex CLI, edit-config, run 2: pass	Codex CLI, edit-config, run 3: pass	Codex CLI, edit-config, run 4: pass	Codex CLI, edit-config, run 5: pass	Codex CLI, edit-config, run 6: pass	Codex CLI, edit-config, run 7: pass	Codex CLI, edit-config, run 8: pass	Codex CLI, edit-config, run 9: pass	10/10
fix-failing-test	Codex CLI, fix-failing-test, run 0: pass	Codex CLI, fix-failing-test, run 1: pass	Codex CLI, fix-failing-test, run 2: pass	Codex CLI, fix-failing-test, run 3: pass	Codex CLI, fix-failing-test, run 4: pass	Codex CLI, fix-failing-test, run 5: pass	Codex CLI, fix-failing-test, run 6: pass	Codex CLI, fix-failing-test, run 7: pass	Codex CLI, fix-failing-test, run 8: pass	Codex CLI, fix-failing-test, run 9: pass	10/10
follow-agents-md	Codex CLI, follow-agents-md, run 0: pass	Codex CLI, follow-agents-md, run 1: pass	Codex CLI, follow-agents-md, run 2: pass	Codex CLI, follow-agents-md, run 3: pass	Codex CLI, follow-agents-md, run 4: pass	Codex CLI, follow-agents-md, run 5: pass	Codex CLI, follow-agents-md, run 6: pass	Codex CLI, follow-agents-md, run 7: pass	Codex CLI, follow-agents-md, run 8: pass	Codex CLI, follow-agents-md, run 9: pass	10/10
implement-lru-cache	Codex CLI, implement-lru-cache, run 0: pass	Codex CLI, implement-lru-cache, run 1: pass	Codex CLI, implement-lru-cache, run 2: pass	Codex CLI, implement-lru-cache, run 3: pass	Codex CLI, implement-lru-cache, run 4: pass	Codex CLI, implement-lru-cache, run 5: pass	Codex CLI, implement-lru-cache, run 6: pass	Codex CLI, implement-lru-cache, run 7: pass	Codex CLI, implement-lru-cache, run 8: pass	Codex CLI, implement-lru-cache, run 9: pass	10/10
implement-slugify	Codex CLI, implement-slugify, run 0: pass	Codex CLI, implement-slugify, run 1: pass	Codex CLI, implement-slugify, run 2: pass	Codex CLI, implement-slugify, run 3: pass	Codex CLI, implement-slugify, run 4: pass	Codex CLI, implement-slugify, run 5: pass	Codex CLI, implement-slugify, run 6: pass	Codex CLI, implement-slugify, run 7: fail	Codex CLI, implement-slugify, run 8: pass	Codex CLI, implement-slugify, run 9: pass	9/10
minimal-fix	Codex CLI, minimal-fix, run 0: pass	Codex CLI, minimal-fix, run 1: pass	Codex CLI, minimal-fix, run 2: pass	Codex CLI, minimal-fix, run 3: pass	Codex CLI, minimal-fix, run 4: pass	Codex CLI, minimal-fix, run 5: pass	Codex CLI, minimal-fix, run 6: pass	Codex CLI, minimal-fix, run 7: pass	Codex CLI, minimal-fix, run 8: pass	Codex CLI, minimal-fix, run 9: pass	10/10
multi-file-rename	Codex CLI, multi-file-rename, run 0: pass	Codex CLI, multi-file-rename, run 1: pass	Codex CLI, multi-file-rename, run 2: pass	Codex CLI, multi-file-rename, run 3: pass	Codex CLI, multi-file-rename, run 4: pass	Codex CLI, multi-file-rename, run 5: pass	Codex CLI, multi-file-rename, run 6: pass	Codex CLI, multi-file-rename, run 7: pass	Codex CLI, multi-file-rename, run 8: pass	Codex CLI, multi-file-rename, run 9: pass	10/10
refactor-extract-helper	Codex CLI, refactor-extract-helper, run 0: pass	Codex CLI, refactor-extract-helper, run 1: pass	Codex CLI, refactor-extract-helper, run 2: pass	Codex CLI, refactor-extract-helper, run 3: pass	Codex CLI, refactor-extract-helper, run 4: pass	Codex CLI, refactor-extract-helper, run 5: pass	Codex CLI, refactor-extract-helper, run 6: pass	Codex CLI, refactor-extract-helper, run 7: fail	Codex CLI, refactor-extract-helper, run 8: pass	Codex CLI, refactor-extract-helper, run 9: fail	8/10
write-tests	Codex CLI, write-tests, run 0: pass	Codex CLI, write-tests, run 1: pass	Codex CLI, write-tests, run 2: pass	Codex CLI, write-tests, run 3: pass	Codex CLI, write-tests, run 4: pass	Codex CLI, write-tests, run 5: pass	Codex CLI, write-tests, run 6: pass	Codex CLI, write-tests, run 7: pass	Codex CLI, write-tests, run 8: pass	Codex CLI, write-tests, run 9: pass	10/10

Tokens and cost

Total tokens per run, by taskMarker is the median over reruns; the line runs from the smallest to the largest run. Totals include cache reads.

Claude Code
Codex CLI

add-cli-flag

edit-config

fix-failing-test

follow-agents-md

implement-lru-cache

implement-slugify

minimal-fix

multi-file-rename

refactor-extract-helper

write-tests

Show as table

Total tokens per run, by task
Task	Claude Code median	Claude Code min	Claude Code max	Codex CLI median	Codex CLI min	Codex CLI max
add-cli-flag	52.4k	52k	72.9k	56.1k	41.5k	73.9k
edit-config	51.3k	50.6k	69.5k	42.4k	41.5k	60.8k
fix-failing-test	68.7k	68.3k	68.8k	84.7k	70.1k	85.2k
follow-agents-md	52.3k	52.1k	52.4k	56.6k	42.2k	72.9k
implement-lru-cache	52.9k	52.7k	53.5k	56.6k	42.6k	86.5k
implement-slugify	52k	51.8k	52.1k	56.1k	27.5k	70.3k
minimal-fix	68.5k	68.4k	86.1k	56k	41.6k	57.2k
multi-file-rename	69.7k	51.4k	70.3k	72.3k	60.1k	101k
refactor-extract-helper	53.7k	53.2k	72.1k	56.3k	41.5k	56.5k
write-tests	66.4k	55.3k	77.8k	72.7k	57.5k	88.5k

Cost per run as reported by the CLI, by taskCodex CLI did not report cost, so there are no marks for it.

Claude Code
Codex CLI

add-cli-flag

edit-config

fix-failing-test

follow-agents-md

implement-lru-cache

implement-slugify

minimal-fix

multi-file-rename

refactor-extract-helper

write-tests

Show as table

Cost per run as reported by the CLI, by task
Task	Claude Code median	Claude Code min	Claude Code max	Codex CLI median	Codex CLI min	Codex CLI max
add-cli-flag	$0.093	$0.088	$0.11	n/a	n/a	n/a
edit-config	$0.081	$0.073	$0.092	n/a	n/a	n/a
fix-failing-test	$0.084	$0.081	$0.085	n/a	n/a	n/a
follow-agents-md	$0.085	$0.083	$0.087	n/a	n/a	n/a
implement-lru-cache	$0.10	$0.098	$0.11	n/a	n/a	n/a
implement-slugify	$0.091	$0.088	$0.094	n/a	n/a	n/a
minimal-fix	$0.084	$0.083	$0.090	n/a	n/a	n/a
multi-file-rename	$0.090	$0.078	$0.094	n/a	n/a	n/a
refactor-extract-helper	$0.11	$0.10	$0.11	n/a	n/a	n/a
write-tests	$0.15	$0.14	$0.17	n/a	n/a	n/a

All headline metrics

All headline metrics
Metric	Claude Code	Codex CLI
Pass rateWilson 95% interval	100% [96, 100]	96% [90, 98]
Macro pass ratetask-bootstrap 95% interval	100% [100, 100]	96% [92, 99]
pass@10at least one of k runs passes	100%	100%
pass^10all k runs pass	100%	70%
Flip ratetwo runs of a task disagree	0%	8%
Flaky tasksshare with both a pass and a fail	0 of 10	3 of 10
Median cost per runas reported by the CLI	$0.0899	not reported
Median total tokens per runincludes cache reads	53,368	56,552
Median output tokens per run	1,016	552
Median wall time per runagent process only	12.7 s	16.8 s
Tokens CV within taskmean over tasks; lower is steadier	0.08	0.16
Approach similarityJaccard over passing diffs	0.82	0.75

Notes from the run

Claude Code was pinned with --model claude-opus-5-5. With no model flag, Claude Code on this account reported claude-opus-4-8 on 2026-10-06 (it reported claude-opus-5-5 on 2026-10-03, also with the same CLI version 2.1.288), so the pinned set is the one compared here. The 100 default-model runs (100 of 100 passed) are in the source directory.
Codex CLI ran 2026-10-06 17:43 to 19:02 UTC; Claude Code (pinned) ran 2026-10-07 01:26 to 01:49 UTC.
One Codex run (write-tests run 3) hung after the Mac went to sleep mid-run and was replaced with a fresh run of the same task and index; the original record is kept in the source directory.
All 4 failures were Codex CLI runs that read the task files, changed nothing and exited 0 after 7 to 10 seconds. In 3 of them the final message said the change had been made.
Claude Code reports cost as total_cost_usd at list prices. On a subscription login this is quota used, not money billed. Codex CLI reports tokens only, so its cost is not reported.

2026-10-03 result set 2 of 2

Pilot: Claude Code and Codex CLI, 10 tasks x 3 runs

Pilot, n=3 per task rerun-bench 0.1.0 harness 7a2bef2

In plain words: Claude Code passed 30 of 30 runs and Codex CLI passed 28 of 30 runs: 10 tasks, each run 3 times, every run scored only by the task's own test script. This is a pilot: it checks that the harness works end to end against real CLIs, and it is too small to rank agents.

Raw JSON · Run records, diffs and notes · Write-up: Same task, 3 runs: Codex passed and failed on 2 of 10 tasks, Claude Code on 0

Setup

Run setup
Agent	CLI version	Model	Tasks	Runs per task	Runs	Started
Claude Code	2.1.288 (Claude Code)	claude-opus-5-5	10	3	30	2026-10-03 08:16:06 UTC
Codex CLI	codex-cli 0.160.0	gpt-6-luna	10	3	30	2026-10-03 08:16:06 UTC

Machine: macOS 26.2, Apple Silicon (arm64), Python 3.13.15.

Pass rate

Claude Codeclaude-opus-5-5 · 2.1.288 (Claude Code)

100% [89, 100]

Codex CLIgpt-6-luna · codex-cli 0.160.0

93% [79, 98]

Show as table

Pass rate with Wilson 95% interval
Agent	Pass rate	95% low	95% high
Claude Code	100.0%	88.6%	100.0%
Codex CLI	93.3%	78.7%	98.2%

Comparison: The 95% intervals overlap for every pair of agents here, so this data does not show that any agent has a higher pass rate than another.

Consistency across reruns

pass^3, pass@1 and pass@3pass^3 is the chance that all 3 runs of a task pass (how far you can trust one run); pass@1 is the plain pass rate; pass@3 is the chance that at least one of 3 runs passes (what retrying until it works gets you). The gap between them is the cost of inconsistency.

pass^3
pass@1
pass@3

Claude Codeclaude-opus-5-5 · 2.1.288 (Claude Code)

gap 0 pts

Codex CLIgpt-6-luna · codex-cli 0.160.0

gap 20 pts

Show as table

pass^3, pass@1 and pass@3
Agent	pass^3	pass@1	pass@3
Claude Code	100.0%	100.0%	100.0%
Codex CLI	80.0%	93.3%	100.0%

Every run, by taskEach mark is one run. Columns 0, 1, 2 are the runs in the order they ran. A row with both a pass and a fail is a flaky task for that agent.

pass (verifier exited 0)
fail

Every run, by task
Task	Claude Code run 0	Claude Code run 1	Claude Code run 2	rate	Codex CLI run 0	Codex CLI run 1	Codex CLI run 2	rate
	Claude Code				Codex CLI
add-cli-flag	Claude Code, add-cli-flag, run 0: pass	Claude Code, add-cli-flag, run 1: pass	Claude Code, add-cli-flag, run 2: pass	3/3	Codex CLI, add-cli-flag, run 0: pass	Codex CLI, add-cli-flag, run 1: pass	Codex CLI, add-cli-flag, run 2: pass	3/3
edit-config	Claude Code, edit-config, run 0: pass	Claude Code, edit-config, run 1: pass	Claude Code, edit-config, run 2: pass	3/3	Codex CLI, edit-config, run 0: pass	Codex CLI, edit-config, run 1: pass	Codex CLI, edit-config, run 2: pass	3/3
fix-failing-test	Claude Code, fix-failing-test, run 0: pass	Claude Code, fix-failing-test, run 1: pass	Claude Code, fix-failing-test, run 2: pass	3/3	Codex CLI, fix-failing-test, run 0: pass	Codex CLI, fix-failing-test, run 1: pass	Codex CLI, fix-failing-test, run 2: pass	3/3
follow-agents-md	Claude Code, follow-agents-md, run 0: pass	Claude Code, follow-agents-md, run 1: pass	Claude Code, follow-agents-md, run 2: pass	3/3	Codex CLI, follow-agents-md, run 0: pass	Codex CLI, follow-agents-md, run 1: pass	Codex CLI, follow-agents-md, run 2: pass	3/3
implement-lru-cache	Claude Code, implement-lru-cache, run 0: pass	Claude Code, implement-lru-cache, run 1: pass	Claude Code, implement-lru-cache, run 2: pass	3/3	Codex CLI, implement-lru-cache, run 0: fail	Codex CLI, implement-lru-cache, run 1: pass	Codex CLI, implement-lru-cache, run 2: pass	2/3
implement-slugify	Claude Code, implement-slugify, run 0: pass	Claude Code, implement-slugify, run 1: pass	Claude Code, implement-slugify, run 2: pass	3/3	Codex CLI, implement-slugify, run 0: pass	Codex CLI, implement-slugify, run 1: pass	Codex CLI, implement-slugify, run 2: pass	3/3
minimal-fix	Claude Code, minimal-fix, run 0: pass	Claude Code, minimal-fix, run 1: pass	Claude Code, minimal-fix, run 2: pass	3/3	Codex CLI, minimal-fix, run 0: pass	Codex CLI, minimal-fix, run 1: pass	Codex CLI, minimal-fix, run 2: pass	3/3
multi-file-rename	Claude Code, multi-file-rename, run 0: pass	Claude Code, multi-file-rename, run 1: pass	Claude Code, multi-file-rename, run 2: pass	3/3	Codex CLI, multi-file-rename, run 0: pass	Codex CLI, multi-file-rename, run 1: pass	Codex CLI, multi-file-rename, run 2: pass	3/3
refactor-extract-helper	Claude Code, refactor-extract-helper, run 0: pass	Claude Code, refactor-extract-helper, run 1: pass	Claude Code, refactor-extract-helper, run 2: pass	3/3	Codex CLI, refactor-extract-helper, run 0: pass	Codex CLI, refactor-extract-helper, run 1: fail	Codex CLI, refactor-extract-helper, run 2: pass	2/3
write-tests	Claude Code, write-tests, run 0: pass	Claude Code, write-tests, run 1: pass	Claude Code, write-tests, run 2: pass	3/3	Codex CLI, write-tests, run 0: pass	Codex CLI, write-tests, run 1: pass	Codex CLI, write-tests, run 2: pass	3/3

Tokens and cost

Total tokens per run, by taskMarker is the median over reruns; the line runs from the smallest to the largest run. Totals include cache reads.

Claude Code
Codex CLI

add-cli-flag

edit-config

fix-failing-test

follow-agents-md

implement-lru-cache

implement-slugify

minimal-fix

multi-file-rename

refactor-extract-helper

write-tests

Show as table

Total tokens per run, by task
Task	Claude Code median	Claude Code min	Claude Code max	Codex CLI median	Codex CLI min	Codex CLI max
add-cli-flag	51.2k	50.6k	70.2k	56.1k	42.3k	72k
edit-config	50.1k	49.3k	50.1k	42k	41.5k	56.2k
fix-failing-test	67k	66.7k	67k	84.6k	84.5k	84.7k
follow-agents-md	51.1k	51k	51.2k	56.5k	56.4k	70.9k
implement-lru-cache	51.7k	51.3k	52.2k	71.8k	56.7k	72.1k
implement-slugify	50.8k	50.5k	50.9k	56.5k	56.4k	56.6k
minimal-fix	66.9k	66.8k	67k	56k	55.8k	56.8k
multi-file-rename	68.5k	68.1k	68.6k	72.5k	56.1k	87.8k
refactor-extract-helper	51.8k	51.7k	52.4k	56.6k	41.7k	72.8k
write-tests	56.4k	53.8k	76.8k	104.3k	72.5k	104.7k

Cost per run as reported by the CLI, by taskCodex CLI did not report cost, so there are no marks for it.

Claude Code
Codex CLI

add-cli-flag

edit-config

fix-failing-test

follow-agents-md

implement-lru-cache

implement-slugify

minimal-fix

multi-file-rename

refactor-extract-helper

write-tests

Show as table

Cost per run as reported by the CLI, by task
Task	Claude Code median	Claude Code min	Claude Code max	Codex CLI median	Codex CLI min	Codex CLI max
add-cli-flag	$0.090	$0.083	$0.11	n/a	n/a	n/a
edit-config	$0.079	$0.069	$0.079	n/a	n/a	n/a
fix-failing-test	$0.081	$0.079	$0.081	n/a	n/a	n/a
follow-agents-md	$0.083	$0.082	$0.085	n/a	n/a	n/a
implement-lru-cache	$0.097	$0.093	$0.10	n/a	n/a	n/a
implement-slugify	$0.090	$0.085	$0.090	n/a	n/a	n/a
minimal-fix	$0.081	$0.080	$0.082	n/a	n/a	n/a
multi-file-rename	$0.090	$0.088	$0.091	n/a	n/a	n/a
refactor-extract-helper	$0.097	$0.096	$0.11	n/a	n/a	n/a
write-tests	$0.16	$0.13	$0.16	n/a	n/a	n/a

All headline metrics

All headline metrics
Metric	Claude Code	Codex CLI
Pass rateWilson 95% interval	100% [89, 100]	93% [79, 98]
Macro pass ratetask-bootstrap 95% interval	100% [100, 100]	93% [83, 100]
pass@3at least one of k runs passes	100%	100%
pass^3all k runs pass	100%	80%
Flip ratetwo runs of a task disagree	0%	13%
Flaky tasksshare with both a pass and a fail	0 of 10	2 of 10
Median cost per runas reported by the CLI	$0.0886	not reported
Median total tokens per runincludes cache reads	52,017	56,629
Median output tokens per run	964	516
Median wall time per runagent process only	12.6 s	16.1 s
Tokens CV within taskmean over tasks; lower is steadier	0.04	0.14
Approach similarityJaccard over passing diffs	0.81	0.77

Notes from the run

With n = 3 per task, one failure moves a task's pass rate by 33 points. This run checks that the harness works end to end against real CLIs; it is not a leaderboard.
Claude Code reports cost as total_cost_usd at list prices. On a subscription login this is quota used, not money billed. Codex CLI reports tokens only, so its cost is not reported.
Total tokens include cache reads, which are most of the count for both CLIs.
num_turns is not comparable across CLIs and is not shown.
Both failures were Codex CLI runs: implement-lru-cache run 0 returned the evicted value instead of the key (agent error); refactor-extract-helper run 1 exited 0 after 8.7 s with no file changes (cause not recorded by this harness version).

Run it yourself or add results

Try the harness for free with its mock agent: uvx rerun-bench run --agent mock. Reproduce this result set with a real CLI.
Add a result set from your own machine: how to submit results.
Seen an agent quietly get worse after an update? Send a regression report.