Benchmarks

Every benchmark repowise publishes now lives on one page: repowise.dev/benchmarks.

This page used to carry its own copy of the figures. It no longer does, for a reason worth stating rather than hiding behind a redirect: the numbers were drifting. Two copies of the same measurement in two places is a guarantee that one of them goes stale, and the stale one is always the copy nobody remembers to update. The benchmark page is now generated from a single fact module, and a CI check compares that module against the methodology of record in docs/BENCHMARKS.md on every push.

Some of what this page previously said was also wrong by the time you read it. It quoted a token reduction of "up to 96%" as an agent-efficiency result. That figure measured a single context payload, not an agent session, and the two are not the same claim. It also said answer quality was "at parity with raw exploration". No significant difference is not the same as parity, and an equivalence claim needs a statistical test we have not run. Both are corrected on the current page.

What is measured#

Section 6 is a row we lose, by a wide margin, and it is on the page for the same reason the others are.

Documentation generation and PR review are not measured. They are capability comparisons rather than measurements, and they live in the README's feature table where a reader can tell the difference.

Reproducing any of it#

The harnesses, the pre-registration files with their commit timestamps, the arm-parity rules, the statistical tests, and every raw run including the invalidated ones with their invalidation notes are in repowise-bench. Nothing is measured on a private corpus.

Related#

• Code health methodology, for how the score the defect study evaluates is actually computed
• MCP tool reference, for the tools the agent-loop study measures
• repowise distill, for the compression study's subject