Does Code Health Predict Bugs? 21 Repos, 9 Languages, ROC AUC 0.74

Ranking files by repowise's code-health score predicts which files get bug-fixed over the next six months: a cross-project mean ROC AUC of 0.74 [95% CI 0.68-0.79] across 21 open-source repositories and 9 languages. On the 2,770 of those files that both tools scored, under a fixed review budget of 20% of lines, it surfaces 2.3x the defects that CodeScene's Code Health score does, and the lowest-health files carry 2.18x the defect density of the average file. Every marker is open source and scoring is deterministic, so you can reproduce the result on your own repository.

Teams put a single 0-to-10 number in dashboards and PR gates, so the question is fair: does the score point at the files that break, or is it a well-dressed line count? This post gives the method, the numbers, the controls built to break the result, and how to run the same check yourself. Results were measured in August 2026; the full test list is in BENCHMARKS.md.

What did we measure?

We tested one claim: does ordering a repository's files by code-health score put the files that actually receive bug-fixes near the top of the list? We score every file at a point in time, wait six months, label a file "defective" if a fix: commit lands on it, and measure how well the earlier score ranked those files. Prediction precedes the label.

Method

The corpus is 21 open-source repositories spanning all nine of repowise's full-tier languages: Python, TypeScript, JavaScript, Rust, Go, Java, Kotlin, C++ and C#. That comes to 2,826 source files carrying real, commit-derived defect labels, the set behind the ROC AUC. 2,770 of them were scored by both repowise and CodeScene, the set behind every head-to-head number below.

The scoring is deterministic. The code-health score is built from 23 markers with no model and no LLM in the loop, so identical inputs always produce an identical score.

The labels come from history, not opinion. A file is "defective" if a conventional-commit fix: touches it inside the measurement window, an SZZ-style attribution that ties the label to a real change a maintainer shipped.

Leakage is the trap this design avoids. Several strong markers are evolutionary: they read recent git history such as churn, author count and change entropy, and a bug-fix is itself a commit that bumps exactly those signals. So we score in the past: check out the last commit on or before a fixed date in a detached worktree, score that, and draw labels strictly from the window afterward. No future fix can reach back and inflate the score.

We report two metrics because they answer different questions. ROC AUC measures raw discrimination: the probability a random buggy file outscores a random clean one. Effort-aware recall and its summary, Popt, measure what a reviewer cares about: under a fixed inspection budget in lines of code, how many real defects do you catch?

Results: how many more defects does health-ordered review find?

Under the same budget, health-ordered review finds 0.173 of defects where CodeScene's ordering finds 0.074. The table compares the two on the same files, labels and budget.

That recall row is 2.3x more real defects under a fixed review budget (p = 0.003), attributable to this open 21-repo benchmark. The lowest-health files carry 2.18x the average defect density, against 0.56x for the tail of a size-ordered ranking.

Across all 21 repos, the resampled cross-project mean ROC AUC is 0.74 [95% CI 0.68-0.79], ranging from 0.55 to 0.86 across individual repos. We resample repositories rather than files, because the thing you generalize to is a new repo, not another file in one you have already seen.

Per-repository ROC AUC across the full 21-repo corpus

Each dot is one repo. The plot is from an earlier run, where zod reached 0.90: small, clean, structurally varied. axios at 0.55 is the other end, a micro-library where so much of the surface was touched in the window that little clean code remained to compare against. The spread is wide either way, which is why we quote no single repo.

Health versus the trivial baselines on AUC and Popt

On raw AUC the score ties a pure line-count baseline, 0.737 against 0.742, with a paired DeLong test at p = 0.92. Raw AUC rewards size, since big files genuinely carry more bugs. Under Popt, where reading big files costs what it costs, the score beats line count by +0.134 [0.080, 0.198].

Does the score survive the controls?

Yes. It holds against the three checks that usually sink defect predictors:

• It is not file size in a trench coat. After controlling for file size, a partial Spearman correlation of -0.16 still excludes zero.
• It out-discriminates recent churn by +0.10 AUC AUC and prior-defect history by +0.12 AUC, each with a paired DeLong test at p < 1e-9.
• The tail is concentrated where it should be. On a typical repo, 16 of the 20 lowest-health files had received a bug-fix in the prior six months, 3.3x the 24% base rate across all files.

Where is the score strongest?

The score reads structure, so we sliced results by file size to see where its signal concentrates.

ROC AUC computed strictly within file-size bands

Computing AUC inside a single size band holds the size advantage constant. On large files, the top two quartiles, the score discriminates at around 0.67. Small, tidy modules give it little structure to read, because every marker gate needs some size or activity to fire. For those files, prior-defect history is the better signal.

The slicing also produced a specific fix. Two markers, primitive_obsession and dry_violation, fire constantly on small modules where that shape is idiomatic, and in the 29-to-68-line band they pointed the wrong way. primitive_obsession now fires only in modules of 60 or more non-blank lines, which lifts that band back above random without regressing anything. We validated the change by re-scoring the cached findings.

Is the result stable across dates and labels?

Yes. A single measurement date could be a lucky window, so we re-ran the whole leakage-free pipeline at three rolling six-month start dates, re-indexing every repo at each one.

Cross-project mean AUC across three rolling measurement windows

The three came out at 0.771, 0.703 and 0.754, a mean of 0.743 that brackets the full-corpus figure. Individual repos move between windows (hono swings from 0.54 to 0.72 as its in-window bug set changes), which is why the headline is a cross-project mean over a diverse corpus and not any single cell.

We also re-ran everything under leakage-free SZZ labels, which git blame each fix back to the commit that introduced the buggy line and keep only files whose bug already existed at the measurement date. SZZ removes 17% of the keyword labels as noise, and accuracy barely moves (mean AUC 0.744 against 0.734) with every significance verdict reproducing. The score predicts where bugs originate about as well as where fixes land.

Does it hold on a dataset we did not build?

Yes. On the PROMISE/Jureczko jEdit benchmark, the CK-metrics-plus-post-release-bugs set used by hundreds of defect-prediction papers, the score reaches AUC 0.76 to 0.78 across two releases. A single-release snapshot has no git history, so only the structural half of the score runs, and it still lands within about 0.03 of the dataset's own cross-validated full-CK-metric model and beats line count on Popt. It is one project, so we treat it as an external check on generalization.

What does a ranking claim cover?

ROC AUC 0.74 is a ranking claim: the score orders files better than chance and better than the trivial baselines. It does not say any single file is doomed, and it does not replace a reviewer. Scoring in the past mitigates label leakage, and we anchor evolutionary windows to the real repo head so windowed markers cannot read the answer key.

Repositories were chosen by criteria fixed before scoring: they index inside a time budget, use Conventional Commits cleanly, and yield at least five defect-bearing files in the window.

Prior-defect history, "re-inspect whatever broke before", is a strong and nearly free heuristic: it leads on Popt, 0.609 against 0.524. The score adds discrimination plus an attributable explanation of which markers fired in a file. A rap sheet only says a file has broken before, so the two signals do different jobs. Use both: history tells you where bugs landed, and the score tells you what in the structure to change.

What this means for a team

• Spend review time by health. Order a fixed review budget by code-health score. In this corpus that finds 2.3x the defects CodeScene's ordering does.
• Look at the lowest tail first. The 20 lowest-health files are where defects concentrate: on a typical repo, 16 of them had a bug-fix in the prior six months.
• Fix from the findings. Each low score comes with the markers that fired, so the refactor has a target. repowise health --refactoring-targets turns them into plans.

How do I reproduce this on my own repository?

Every one of the 23 markers behind the score is open source under AGPL-3.0 (59 detectors across all three health pillars), and scoring needs no model or API key.

pip install repowise
repowise init --yes
repowise health

From an agent, get_health(include=["accuracy"]) reports how many of your 20 lowest-scoring files had a bug fix in the last 180 days against your base rate. It is an association on your own history, and it stays silent below 25 scored files or 5 fixed ones. For a full replication, score a historical commit, then count the fix: commits that touched each file in the following six months, as in the method above.

Related reading: the code health feature page, Code Health: The Complete Guide, which markers carry the prediction in why process metrics beat structural metrics, and the CodeScene alternative comparison. Last reviewed: October 2026.