how to benchmark a codebase tool so the number survives a rerun

In July 2026, JetBrains took two popular token-saving tools and reran their headline claims on real agent work with Sonnet 5. Caveman advertised 65%. JetBrains measured 8.5%. RTK advertised 60 to 90%. JetBrains measured it 7.6% more expensive at low reasoning effort, with no change at high effort. Greptile's advertised 82% became 45% under Augment's independent rerun.

Three collapses is not bad luck. It is a category property. The axis that matters in this field is not whose number is biggest, it is whose number survives someone else running it. That is the standard we tried to build our own benchmark to, and this post is the method, written so it transfers to whatever you are trying to measure, not just to us.

1. Pre-register, as a commit, before you spend anything

Decide what you are testing, which harness is primary for which claim, and what result would falsify your hypothesis, and write it down before the first dollar is spent. Not in your head, not in a planning doc you can quietly edit later. A commit, with a timestamp, that exists before the numbers do.

The reason is not ceremony. It is that a benchmark you can still redefine after seeing partial results is a benchmark you will unconsciously redefine in your own favor. Committing the plan first is the only thing that makes "we didn't tune this" a claim anyone else can check.

2. Split and seal before you build anything

Our retrieval benchmark, ContextBench, was split 70 development instances and 42 sealed instances, stratified and pinned by instance id, and committed before rung 8 built a single index. All improvement work used the development half. The sealed half was touched exactly once, at publication.

The check that proves the split did its job is not the split itself, it is what a tool nobody tuned against does on both halves. CodeGraph scored 0.6093 on development and 0.6095 on sealed, three decimal places apart. Nobody optimized for either half, so if the halves were unevenly hard, CodeGraph's number would show it. It did not. That is how you demonstrate the two halves are equally hard, rather than asserting it.

Our own number moved the direction that matters: 0.810 on development against 0.876 on sealed. Overfitting makes the unseen half score worse. Ours scored better, which is weak evidence in our favor precisely because it is the opposite of what overfitting predicts.

3. Measure the loop, not the payload

Measuring one context load is easy. You show that your representation of a file is smaller than the file. That is a real measurement, and it is the one almost everybody in this category publishes, because it is the flattering one.

It is also not the question anyone actually has, which is whether an agent given your tool finishes the job having done less work. Agents re-read, backtrack, re-plan, and re-explore. A compression that looks like 90% on one payload routinely nets out near zero across a real session, and it can go negative when the agent has to work harder to recover what you compressed away. Measuring the loop instead of the payload is harder, slower, and more expensive, and it is the only version of the number that answers the question a reader actually has. The full accounting of how differently those two measurements can read for the identical tool is in 35x or 16%? we published both, and they measure different things.

4. Before you record a zero, prove the detector works and the arm was alive

This is where the real cost of the work lives, and it deserves specifics rather than a general warning, because the specifics are what make the rule usable.

A competitor once measured 0.012 on a ranking metric in our harness. That looked like a near-total failure. The real cause was that our path extractor required whitespace before a file path, and that tool emits paths as [src=path], with no whitespace before them. Its real score was 0.539, good enough for second place. The extractor was broken, not the tool.

Another competitor returned errors on 84 of 84 tool calls. Its tools carry a _tool suffix in their names, and one of them expected structured pattern and target parameters rather than the natural-language query string every other tool in the field accepts. We were calling it correctly for every other tool and incorrectly for that one, and the failure looked exactly like the tool being broken.

A third case cost us a false alarm against our own work. A hook-firing counter read five real firings as zero, because the shell redirect that fed it wrote UTF-16LE and the reader expected UTF-8. The control printed "POSITIVE CONTROL FAILED" while the thing it was checking was, in fact, working correctly. We nearly shipped a regression that did not exist.

The general rule, stated plainly: the arm that gets silently zeroed is never your own tool, because your own output format is the only one you already know cold. Every unfamiliar tool's output shape is a place a zero can hide, and it is on you to prove the zero is real before you publish it.

5. Give every competitor its full advertised surface

We under-allowlisted at first, and it looked like a reasonable setup rather than a mistake. Graphify serves 10 tools; we gave it 1. Serena serves 29; we gave it 3. code-review-graph serves 30; we gave it 1. Our own arm, meanwhile, ran the full surface we had pinned for ourselves.

An arm you handicapped is not an arm you measured. If a tool's real behavior depends on tools it never got the chance to call, your number is measuring your allowlist, not the tool. The fix is mechanical once you see it: every competitor gets every tool its own documentation says it serves, and the rule applies to your own tool too, so nobody can quietly under-serve the control.

6. Isolate the trees

Every tool we tested writes its index into a dotdir inside the repository it indexes. A shared checkout across arms means each arm's index build sees its predecessors' output sitting in the same tree. The bias runs in favor of whoever happens to run first, which in an unguarded setup was us, since we ran our own arm before the competitors' arms in the naive version of the harness.

The fix is one worktree per arm, so no arm can see another arm's artifacts. Clearing the artifact directories between cells looks like the same fix and is not: it is correct for keeping timing measurements clean, and it is fatal for any later rung that needs an earlier arm's index still on disk. Isolate the trees; do not delete the indexes.

7. Report what you lose

We are the slowest indexer in the field we measured, by 22x, and that number ships on our own benchmarks page next to every number we win. The reason is legible: the tools we compare against build a call graph, and in the same pass we mine git history, generate and embed documentation, extract decision records, and score code health. That does not make the losing row disappear, and it should not.

The same rule applies to runs, not just numbers. Invalidated runs are never deleted. They get an INVALID: banner explaining why and the numbers stay visible, because a benchmark that quietly removes its own bad runs is not more credible than one that shows them, it is just harder to audit.

8. Say which failures you retried and which you did not

Not every failed cell means the same thing, and treating them the same is a quiet way to inflate a result. An auth failure means the cell never ran at all, so it gets retried, because there is no data to lose by trying again. A turn-limit exhaustion is a real agent outcome: the agent tried, ran out of turns, and did not finish. That gets excluded from the comparison and named as what it is, an agent failure, not a plumbing failure.

Retrying turn-limit exhaustions until they happen to succeed quietly deletes exactly the hardest attempts your run produced, the ones most likely to show a tool's real limits. The rule has to be decided before you see which cells failed, or you will find yourself retrying the failures that embarrass you and accepting the ones that don't.

What this costs, honestly

None of the above is free, and pretending it is the eighth way a benchmark stops surviving a rerun. Producing the retrieval-layer table cost 748 index builds and roughly 78 machine-hours, because every arm builds its own index of every instance's repository at that instance's own base commit, with nothing shared and nothing cached across instances.

Producing the agent-loop tables cost 471 agent runs. About 13 hours of machine time and roughly $44 of API spend. Around a third of those runs are proof-of-life checks rather than headline numbers. That third does not appear in any published table, and it is most of what makes the tables trustworthy: confirming a server actually answers before a run counts, and confirming a control can correctly report a tool as unused rather than passing everything by default.

If a published benchmark in this category does not show you a cost like this, that is worth noticing. Rigor has a line item, and a number that cost nothing to produce is a number nobody checked. For the specific pattern that first taught us this, where a dollar figure and a token figure disagreed about the identical run, see a tool that did nothing came out 43% cheaper than the bare agent. Full tables and raw data for everything above live on the benchmarks page.

FAQ

Why did JetBrains's reruns of Caveman and RTK come out so different from the advertised numbers?

We don't know their internal cause, only the public numbers: Caveman's advertised 65% measured at 8.5%, and RTK's advertised 60 to 90% measured 7.6% more expensive at low reasoning effort. The general lesson we draw is not about those two tools specifically, it is that a claim measured once, by the vendor, on a payload rather than a full agent loop, is the shape of claim that tends not to survive an independent rerun.

What does "pre-registering" a benchmark actually mean in practice?

Writing down, as a committed file with a timestamp, what you are testing, which harness is primary for which claim, and what result would falsify your hypothesis, before you spend any money running it. It stops you from redefining success after you've seen partial results.

Why split a benchmark into development and sealed halves?

So you can tune your tool against one half without contaminating the number you publish. The sealed half is touched once, at the end, and a competitor nobody tuned against should score about the same on both halves, which is the check that the split itself is fair.

How do you know a zero score for a competitor is real and not a bug in your own harness?

You don't, until you've checked. Every zero in our runs got the same treatment: confirm the tool's server was actually alive, confirm the extractor reading its output understands that tool's specific output format, and only then record the zero as real. Every case we found where we skipped that step, the zero turned out to be ours, not the tool's.

Is measuring an agent loop always better than measuring a single context load?

They answer different questions, and both are legitimate as long as you label which is which. A single-payload measurement tells you your representation of something is smaller than the thing. An agent-loop measurement tells you whether an agent actually does less work with your tool. The mistake is publishing the first as if it answered the second.

What's the actual cost of running a benchmark this way?

For our retrieval layer, 748 index builds and about 78 machine-hours. For the agent-loop layer, 471 agent runs and roughly $44 of API spend, with about a third of that spent on checks and repeats rather than the numbers that get published.