a tool that did nothing came out 43% cheaper than the bare agent

Dollar cost per question is the number every tool in this category wants to quote. It is also close to meaningless as a measure of a tool, and we have a control that proves it rather than a thought experiment.

The control

We ran six arms, five tools plus a bare agent with no tools at all, against the same 15 questions on django/django under Claude Code. One of those arms, code-review-graph, never called its server. Not once, on any question. n = 15 · Claude Code (claude-sonnet-5) · django/django at a single pinned commit · 2026-08. It advertises 30 tools over a graph of 40,904 nodes and 380,168 edges, and the agent never asked it anything.

A tool the agent never calls is behaviourally identical to the bare agent. Same reasoning, same tool calls, same answer. The only difference is that code-review-graph's schema sits in the prompt the whole time, roughly 28,118 extra characters of tool descriptions the model has to read before it decides not to use any of them. That schema costs tokens. It should make the run more expensive, not less.

Measured on dollars, it came out 43% cheaper than the bare agent. A tool that did nothing produced a best-in-class saving.

Why: prompt caching

The cause is not the tool. It is the order the arms ran in.

Claude's prompt cache bills cached tokens at a fraction of fresh ones. Whichever arm happens to run first in a cycle pays full price to warm the cache, and every arm that runs after it inside the cache window reads the same prefix cheaply. Schema, system prompt, shared context, all of it. An arm's dollar figure is partly a measurement of the tool and partly a measurement of where it landed in the queue.

We checked this rather than asserted it. n = 90 · Claude Code (claude-sonnet-5) · django/django at a single pinned commit · 2026-08: -0.487. That is not a small correlation. It says close to a quarter of the variance in an arm's dollar cost is explained by nothing except when it happened to run.

The cache has a clock, and that made it worse

There is a second version of the same problem, and it is sharper because it does not need six arms to show up.

Prompt caching on Claude has a roughly five-minute TTL. How long an arm waits between two of its own cells is set by how many arms are interleaved in a run. Six arms sharing a rotation means a long gap between one code-review-graph cell and the next, long enough for the cache to go cold. Two arms sharing a rotation means a short gap and a warm cache the whole time.

We measured that difference directly, on the same tool, the same questions, changed only the number of arms sharing the schedule: 39 percentage points of movement on the measured dollar delta. Nothing about the tool changed. Nothing about the questions changed. The only variable was how many other things we benchmarked at the same time.

That is not a rounding error. It means a vendor could report almost any dollar-cost saving they wanted for the identical tool, just by choosing how many competitors to run it alongside.

Output tokens are immune, and that is the number we report

Cached tokens are the whole problem, so the fix is to measure something the cache cannot touch. Output tokens, the reasoning, tool calls and final reply the agent itself writes, are never cached. Whatever the model generates, it generates fresh every time.

We checked the same correlation on output tokens instead of dollars. n = 90 · Claude Code (claude-sonnet-5) · django/django at a single pinned commit · 2026-08: +0.010, which is nothing. Run position explains essentially none of the variance in how much an agent writes.

So that is what our agent-loop benchmark reports: output tokens, not dollars. It is a smaller, less impressive-looking number than the dollar figure would have been. On this same run, output tokens fell 15.9% against the bare agent when the agent actually used the tool, not 43%, and that smaller number is the one that survives a rerun on a different schedule. See how to benchmark a codebase tool so the number survives a rerun for the rest of what that discipline requires.

The number that goes with this one

This is not a case where our tool loses and a competitor wins. On the harness where every tool actually got called, the Codex run shows repowise cutting output tokens -31.6% against the bare agent, n = 43 · p = <0.0001 · Codex (gpt-5.6-sol) · django/django at a single pinned commit · 2026-08. That number has nothing to do with caching, because it is measured in tokens the agent wrote, not dollars it was billed.

The point of this post is not the size of that number. It is that a dollar figure and a token figure can point in completely different directions for the identical tool on the identical questions, and only one of them is describing the tool. For the deeper case that agent-loop measurements and single-payload measurements answer different questions entirely, see 35x or 16%? we published both, and they measure different things.

What to ask instead

If you read a token-savings or cost claim in this category and it does not say whether the measurement controls for cache state and arm ordering, that is the first thing to ask about. Not the second, not a footnote question, the first. A vendor who cannot answer it has not shown you their tool's cost. They have shown you the shape of their benchmark.

We are not naming names beyond what our own benchmarks page already names publicly, because the point generalizes past any one competitor: this is a property of how prompt caching bills, and it will produce the same illusion for anyone who runs a multi-arm comparison without controlling for it. We hit it in our own harness, on our own run, and the fix was to stop reporting the number the cache was corrupting.

FAQ

Why did a tool that was never called come out cheaper than no tool at all?

Because it ran later in the benchmark's schedule than the bare agent did. Prompt caching bills cached tokens at a fraction of fresh ones, and whichever arm runs first pays to warm the cache while every arm after it reads the same prefix cheaply. The saving belongs to the schedule, not to the tool.

Is this specific to code-review-graph?

No. It is a property of how Claude's prompt cache bills, and it will happen to any tool in any multi-arm benchmark that does not control for cache state and run order. We name code-review-graph because it is the cleanest control we had: a tool that provably did nothing still won on cost.

How long does the prompt cache stay warm?

Roughly five minutes on Claude. How long an arm waits between its own cells depends on how many arms share the rotation, which means the number of competitors in a benchmark changes the benchmark's own cost numbers.

Why does repowise report output tokens instead of dollar cost?

Because output tokens are never cached, so they are not sensitive to run order. The correlation between an arm's position in the run cycle and its output-token count measured at +0.010, versus -0.487 for dollar cost on the same runs.

Does this mean dollar-cost numbers are always wrong?

Not always, but they are conditional on a benchmark's schedule in a way most published numbers do not disclose. A dollar figure from a single-arm, single-run measurement with a stable cache is more trustworthy than one from a rotating multi-arm comparison. Ask how the run was scheduled before trusting the number.

Where can I see the full agent-loop numbers, not just the cost caveat?

The benchmarks page publishes the complete Codex and Claude Code tables, including which tools the agent actually called and by how much the loop got leaner. The companion post on benchmarking a codebase tool so it survives a rerun covers the rest of the methodology this post's control came out of.