we benchmarked ourselves against four open source tools and came last

We built a retrieval benchmark to measure repowise against the open source agent-context field: CodeGraph, Graphify, code-review-graph. Before running a single query we split the 112 test instances 70/42 by instance id, pinned the split in a committed file, and sealed the 42 so nothing downstream could see them. We agreed to that protocol in advance because we knew exactly what we'd be tempted to do once we saw a bad number.

We saw a bad number. We came last, at 0.228 file coverage against CodeGraph's 0.609. On the 20 Go instances in the set we scored 0.025 while every competitor cleared 0.50. We published it anyway, because the whole point of this workstream was to publish whatever came back.

This is the writeup of what came after: the diagnosis, the fix, and the check that made sure the fix wasn't benchmark tuning wearing a lab coat.

Why this comparison exists at all

repowise builds five things from one index: a dependency graph, a git-history layer, generated documentation, architectural decision records, and a code-health score. No single open source tool covers all five, so there's no honest way to declare a winner across the board. What we can do is measure the layers where a real competitor exists and publish each result on its own terms, including the ones where the honest answer is "we lose."

Retrieval, finding the right files for a question about a codebase, is one of those layers. CodeGraph, Graphify, and code-review-graph all build some kind of graph or index over a repository to answer where-is-this questions. ContextBench gives all of us the same test: a bug report from a real GitHub issue, and a gold set of files the actual fix touched. Grading is deterministic, with no LLM judge involved, which makes this the most reproducible number on the whole benchmarks page.

The protocol, agreed before we saw a single result

The plan was: measure, improve based on what we found, then publish when the numbers were good. That's legitimate product work, and it's also exactly how a benchmark gets quietly overfit. If you tune your retrieval against the questions you're about to publish results on, the first thing a skeptical reader should ask is whether you selected for the test, and they'd be right to ask.

So before rung 8 built a single index, we split ContextBench's 112 instances 70/42 by instance id and committed the split to a file. All improvement work happened against the 70. The 42 were touched exactly once, at publication. We also decided in advance to publish the arc rather than just the endpoint: "we measured, found the failure, fixed it, and it held on instances we'd never run against" is a stronger claim than one good number, and if the fix didn't hold on the sealed half, that was the result too.

We ran it and came last

The first run, on the 70 development instances, put us at the bottom of a five-way table:

Every arm was checked alive before any of this counted: 70 of 70 cells returned status: ok, the embedder was live on every repowise cell, and a known-perfect and a known-wrong prediction were graded first to confirm the grader could tell the difference. This was a real measurement, not a broken harness reporting a plausible zero.

The split by repository is where it gets uglier. On django we scored 0.312, bad but recognizably in the same family as everyone else. On the 20 Go instances we scored 0.025 while every competitor cleared 0.50. That's not "weaker." That's not working, and it sat inside the pooled 0.228 the whole time, invisible unless you split it out.

The diagnosis: it wasn't retrieval

The instinct at this point is to assume the ranking is bad, or the embeddings are bad, or the index is missing something. None of that was it.

We looked at the 19 instances where we did surface a gold file, and 14 of them had it at rank 1. Ranking, when it got the chance to run, was fine. The problem was upstream of ranking: a set of query-time gates, short-circuit checks that fire before the retrieval pipeline's main path ever runs, and return an answer built from something other than the actual candidate pool.

A failure taxonomy over the 29 get_answer misses on the development half found that 21 of them were exactly this: early-return gates firing and bypassing the ranking mainline entirely. The mainline they bypassed was healthy, hitting 33 of 37 instances that reached it. One gate was tuned to recognize "questions about a data shape" using trigger words like field, key, and schema, and those words show up incidentally in most bug reports about an ORM-backed codebase. It wasn't detecting data-shape questions, it was detecting database-shaped codebases, and it fired on 23 of 50 python instances and almost none of the Go ones, which is also most of why the Go score looked worse than django's: different gate exposure, not a deeper competence gap.

The most direct evidence this was a query-time problem, not a retrieval one: 18 of the 19 instances where CodeGraph won and our get_answer lost were files our own search_codebase tool had already found, on the exact same index, in the same run. Same bytes on disk, same embeddings, two tools pointed at them, and one tool's query path threw the candidate away before it reached ranking. That fact alone locates the entire competitive gap at query time, not in indexing or in the embedder.

The fix, and what it moved

The fix wasn't a new ranking model or a bigger context window. It was narrower: stop discarding the ranked candidate pool the early-return gates were sitting on top of, and tighten the trigger condition on the gate matching ORM vocabulary instead of data-shape questions. Both changes shipped as ordinary pull requests, the kind any user of the tool gets regardless of whether ContextBench exists.

Measured against the development half:

The middle row matters as much as the top one. search_codebase isn't touched by either fix, and it moved by exactly 0.000 on all 70 instances, in both languages. That rules out index drift, embedder drift, or grader drift as an alternate explanation for the get_answer jump: if something upstream had changed, the untouched arm would have moved too.

One instance regressed during this run, from a file coverage of 1.000 to 0.000, and we didn't chase it down cleanly. Its gold set is exactly one file, so a single file falling off the tail of a served list flips the score completely with nothing in between, and it read like ordinary tail churn rather than a systematic break. We queued a stability recheck of three independent rebuilds to confirm that reading. It never produced a result: every rebuild attempt in that batch failed for an unrelated operational reason before it could run. That regression is neither confirmed nor refuted, and it stays that way in the published data rather than getting rounded off to "probably fine."

Then we touched the sealed half, once

With both fixes shipped, we ran the 42 sealed instances for the first time. This was the run the whole protocol existed to protect: touch it once, whatever comes back gets published.

Here is the paragraph the whole exercise exists for. The sealed half scored higher than the half we tuned on. Overfitting has a signature, and it's the opposite of this: a model tuned against one slice of data gets worse on data it's never seen, because it learned the slice's quirks instead of the underlying thing. Our number went up on the half nobody optimized against.

The second row is the check on the check. CodeGraph is a fixed target: nobody on our side tuned anything against it, and nobody on its side knew our split existed. It scores 0.6093 on the development half and 0.6095 on the sealed half, three decimal places apart, which tells you the two halves of ContextBench are equally hard. If the sealed half had been the easier one, our jump would mean nothing. It wasn't, so the comparison is between tools rather than between question difficulty.

Head to head against CodeGraph on the sealed half, get_answer wins 19 wins, 1 loss, 22 ties, sign test n = 42 · p = 0.00004 · ContextBench, 42 sealed instances, Python and Go · 2026-08.

We also want to name the number we're deliberately not leading with. Pool both halves and the figure is 0.835, higher and easier to say. We don't quote it, because averaging the halves throws away the only distinction that makes this result mean anything: how the tool performs on instances it has genuinely never seen, versus instances it was developed against.

What this cost, and what it doesn't prove

Every arm in this benchmark builds its own index of every instance's repository, at that instance's own base commit, because a stale checkout is a wrong answer rather than a fast one. Nothing is shared or cached across arms or instances. Across the full matrix that's 748 index builds and Every arm builds its own index of every instance's repository at that instance's own base commit. Roughly 78 machine-hours for 1,129 graded cells.

A few things this result does not say, stated plainly rather than left for a reader to assume:

• This is retrieval, not task success. It says we find the right files, not that an agent using them writes better code. That's the subject of the agent-loop numbers on /benchmarks, not this post.
• We are still the slowest indexer in the field, by 22x against CodeGraph on the same repository, and nothing here touched that.
• One benchmark, one grading method. ContextBench is Python and Go only, and we won't imply a number for languages we haven't measured.

Why publishing the loss first is the actual point

In July 2026, JetBrains reran two published token-saving claims from tools in this category against real agent work. One advertised a 65% saving and measured out at 8.5%. The other advertised 60 to 90% cheaper and came back 7.6% more expensive at low reasoning effort. Separately, Greptile's advertised 82% became 45% under an independent rerun by Augment. Not edge cases: in this category, the number a company publishes and the number an outside party reproduces are routinely two different numbers.

A benchmark result is worth something to the extent it would survive someone else running it, not to the extent it's the biggest number available. Our recovery is credible, rather than another retuned headline, specifically because the loss came first, on the record, before any fix existed, checked against instances sealed before we knew what we'd need to fix. If we'd only ever shown the 0.876, there'd be no way to distinguish "genuine improvement" from "reran until it looked right." Showing 0.228 first is what makes 0.876 mean anything.

If you want to check this yourself, the raw data, every cell including the failures, is public. Our benchmarks page carries the full numbers with sample sizes and caveats attached, our docs describe how get_answer and search_codebase retrieve candidates, and /compare has the tool-by-tool breakdown against the rest of the field.

FAQ

Did you tune repowise against this benchmark before publishing the loss?

No. We fixed the split of 112 ContextBench instances into 70 development and 42 sealed before running rung 8 the first time, committed it, and queried the sealed 42 exactly once, after both fixes had already shipped based on the development half alone. If we'd tuned against the sealed set, CodeGraph's near-identical scores on both halves (0.6093 and 0.6095) wouldn't be possible unless the halves were equally hard, which is exactly the check that would catch us.

Why not just publish the pooled 112-instance number? It's higher.

Because it hides the thing we're trying to prove. Pooling both halves gives 0.835, easy to lead with, but it throws away the one distinction that makes the sealed-half score meaningful: performance on instances the improvement work never saw. A pooled figure can't tell you whether we generalized or got lucky. The split figure can.

Is this reproducible?

Grading is deterministic, with gold file spans and no LLM judge. The raw per-cell data, including every failed cell and invalidated run, is public in the benchmark repository linked from /benchmarks. Anyone can rerun the same instances against the same tool versions.

Does finding the right files mean an agent using repowise writes better code?

No. This measures retrieval only: whether the tool points at files a real fix actually touched. It says nothing about whether an agent that receives those files uses them well or writes a correct patch. That's a separate measurement, in the agent-loop section of /benchmarks, not this post.

What was actually broken, in one sentence?

Query-time gates were returning early with an answer built from something other than the ranked candidate pool, so most questions never reached the retrieval mainline, which was already fine when it ran.

Why does the sealed half score higher than the half you tuned against? Isn't that backwards?

It's backwards for overfitting, which is the point. A fix shaped around quirks in the questions you tuned against tends to score worse on unseen data, because those quirks don't generalize. Ours scored better on the half it never saw, and CodeGraph, an untuned baseline, scored within 0.0002 of itself on both halves, so the simplest explanation is that the fix generalizes rather than that we got a lucky draw.

Why are you still the slowest indexer in the field if this work went so well?

Because this work didn't touch indexing. Retrieval quality and index build time are separate costs, and fixing one doesn't move the other. We're still 22x slower than CodeGraph on the same repository: in the same pass where it builds a call graph, we also mine git history, generate documentation, extract decisions, and score code health. That tradeoff isn't resolved by anything in this post.