06 Findings
Findings
Each finding is a short write-up of one result set: what was run, the numbers with their intervals, and what the data cannot show.
- 2026-10-07 Same task, 10 runs: Codex CLI reported edits it never made in 3 of 100 runs
200 runs: Claude Code and Codex CLI on the same 10 coding tasks, 10 runs each. Claude Code passed all 100. Codex passed 96, and all 4 failures were clean exits with no file changed. - 2026-10-05 Same task, 3 runs: Codex passed and failed on 2 of 10 tasks, Claude Code on 0
A 60-run pilot: Claude Code and Codex CLI on the same 10 coding tasks, 3 runs each. What reruns showed, the two Codex failures, and what 3 runs per task cannot show.