Methodology

3.1 Runs

What one run is

1. rerun-bench copies the task's starting workspace into a fresh temporary directory. Nothing carries over between runs.
2. It starts the agent CLI headlessly in that directory with the task prompt, and records wall time, exit status, token counts, cost when the CLI reports it, the CLI version, the model and the final diff.
3. The task's verifier runs against the workspace. The run passes if and only if the verifier exits 0.

Runs are interleaved: run 0 of every task, then run 1 of every task, and so on, so a provider-side change in the middle of a session spreads across tasks instead of landing on whichever task ran last.

3.2 Metrics

What each metric means

Pass rate

Passing runs divided by all runs. A run passes only if the task's verify.py exits 0. The agent's exit code and its own claims of success are recorded but never scored.

Wilson 95% interval

The range of pass rates consistent with the observed runs. Unlike the normal approximation, it behaves at 0% and 100% and at small n. With 30 runs and 30 passes the interval is still 89% to 100%.

Macro pass rate and task-bootstrap interval

The mean of per-task pass rates, with an interval from resampling tasks. It reflects uncertainty about the task population, which the pooled Wilson interval ignores. Use it when comparing agents across the suite.

pass@k

Chance that at least one of k runs of a task passes (Chen et al. 2021). What you get if you retry until it works.

pass^k

Chance that all k runs of a task pass (Yao et al. 2024). What you get if you run the agent once and trust the result.

Flip rate

Chance that two runs of the same task disagree, averaged over tasks. 0 means every task always passes or always fails.

Flaky task fraction

Share of tasks with at least one pass and at least one fail. Coarser than flip rate and grows with n.

Cost, tokens, wall time

Medians per run as the CLI reports them. A value the CLI does not report is null and shown as not reported, never as zero. Wall time covers the agent process only.

Within-task CV

Coefficient of variation across reruns of one task, averaged over tasks. It measures how steady one agent is, not how different the tasks are.

Approach similarity

Mean Jaccard similarity of changed lines across passing runs of a task. 1.0 means every passing run made the same edit.

pass^k is never higher than pass@1, and pass@1 is never higher than pass@k. The gap between pass@k and pass^k is the cost of inconsistency. Full formulas in METRICS.md.

3.3 Comparisons

How agents are compared

• Agents are listed alphabetically on every page. There is no rank column.
• When two agents' 95% intervals overlap, the site states that the data does not show a difference. It does not call either one better.
• Statistical tests between result sets (Fisher exact per task, paired task-level bootstrap) are planned in rerun-bench and not yet run. Until they are, non-overlapping intervals are reported as such and nothing more.
• Results from different dates, machines or CLI versions are separate result sets and are not pooled.

3.4 Tasks

Task suite

rerun-bench default suite: 10 small, verifiable Python tasks. Each has a prompt, a starting workspace, a reference solution and a verifier script. They cover precise edits, bug fixes, implementing to a spec, refactors that must preserve behavior, following repository instructions, and writing tests that must catch injected bugs. Task sources.

3.5 Isolation

How runs are isolated

• Workspace. Each run gets its own temporary copy of the task workspace. Task workspaces are not git repositories.
• Personal configuration. For Claude Code, rerun-bench loads only project and local settings (--setting-sources project,local --strict-mcp-config), so the user's hooks, plugins and MCP servers do not apply. For Codex CLI it passes --ignore-user-config, so the model, reasoning effort and hooks in the user's config do not apply. Authentication still works.
• Nested sessions. When rerun-bench itself runs inside an agent session, that session's environment variables are removed before the measured agent starts.
• Permissions. The agent runs with file-edit and shell permissions inside the temporary directory, like any unattended agent session.

3.6 Limits

Known limitations

• Small samples. At 3 runs per task, one failure moves a task's pass rate by 33 points. Results marked pilot are checks that the harness works, not leaderboards.
• Small tasks. Ten short Python tasks do not represent work in large repositories, other languages, or long sessions.
• One machine, one day. Each result set comes from one machine at one time. Provider-side load, routing and model changes are outside the lab's control.
• Cost semantics. Claude Code reports cost at list prices even on a subscription login, where it is quota used rather than money billed. Codex CLI reports tokens only.
• Model identity. codex exec --json does not report the model, so it is recorded from the --model flag passed to the CLI.
• Token counts. Totals include cache reads, which dominate the count for both CLIs. Turn counts are not comparable across CLIs and are not shown.
• Approach similarity. Line-level Jaccard is sensitive to formatting and ignores line order. It signals variability, not code quality.
• Concurrency. In the pilot the two agents ran at the same time on one machine, which can affect wall time.

3.7 Reproduce

Reproduce a result set

uv tool install rerun-bench
rerun-bench run --agent claude --tasks all --runs 3 --out results/ --run-id claude --yes
rerun-bench run --agent codex --model <model> --tasks all --runs 3 --out results/ --run-id codex --yes
rerun-bench report results/ --format json -o report.json

Real runs spend API credit or subscription quota. Start with --tasks edit-config --runs 2.