What 6,000+ review runs taught us about useful feedback

Measure useful findings.

Every false positive asks a developer to spend time disproving a claim. Counting comments alone rewards volume. We care about findings worth acting on: concrete issues supported by the code, with a clear reason to fix them.

We used real PRs from our work on snaptale.com to test an internal version of HeyDeer in everyday development. We reviewed our internal PR activity across projects over the past six months, from March 19 to September 19, 2026. This window contains 6,132 recorded review attempts, including 5,879 successful runs. This includes retries, failures, cancellations, and different configurations; it is not a count of unique PRs or independently scored test cases.

The Elo analysis uses a fixed set of 1,024 assessed internal PR review rounds from September 5–14. Of these, 752 rounds produced eligible comparisons.

The model is one part of the review.

A model generates the analysis; its harness provides tools, repository access, prompts, and execution control. We identify both. In these runs, Grok 4.6 used Grok Build, Opus 5 used Claude Code, and GPT-5.6 Sol used Codex.

Each baseline received one review assignment. “Single pass” here does not mean one model response: the harness can make tool calls and perform multiple internal steps. HeyDeer’s deeper review workflow adds independent reviewers, then a lead reviewer verifies and deduplicates their findings before publication. These experiments with an internal version of HeyDeer helped shape that approach.

Observed reviewer ratings

These are relative Elo ratings for selected configurations within the archived comparison pool. Ratings use the full comparison pool, available in the downloadable results.

One round may generate several pairwise comparisons. Sample sizes differ, and Sol has no direct matchup with Grok in this archive. Shared opponents connect the ratings. These numbers do not establish a universal model ranking.

The HeyDeer row describes the multi-round architecture. Its system-level Elo is unmeasured: this archive scores individual reviewers and contains no independently judged entry for the completed workflow.

How a finding becomes a score

• Effective findings: issues judged valid and either published or fixed during the review. Repeated, low-value, and unresolved findings do not earn this credit.
• Exclusive findings: effective issues found by only one reviewer, with no independent discovery by the lead reviewer. Exclusivity is checked against the full round’s recorded contributors.
• False positives: findings explicitly judged incorrect. Missing feedback or an unpublished comment is not automatically a false positive.
• LLM feedback: the lead reviewer supplies structured judgments and reasons. This is internal model-assisted adjudication, not independent human ground truth.

The higher score wins the pair; a tie counts as half a win. Both reviewers must succeed on the same dispatched and final code revision, and the review must be published. Pairs with no effective findings and no false positives are skipped.

Every reviewer starts at 1,200. We replay rounds chronologically with K = 32 and a 400-point scale. All matches within a round use the ratings from before that round, then their updates are applied together.

Why we use multiple review stages

Independent reviewers can surface different issues. Verification then checks the evidence, reconciles overlapping findings, and filters unsupported claims before they reach the developer. Our implementation gives deeper reviews separate investigation and verification stages for this reason.

The archive measures individual reviewer contributions. Quantifying the improvement attributable to multiple stages requires a matched single-pass versus multi-round experiment, with independent judging and comparable budgets. That comparison remains outside the scope of these results.

Quality and cost need separate evidence.

Elo measures relative outcomes under this rubric; it does not include price, latency, or token use. A higher Elo therefore does not prove better cost-effectiveness. Our product lets teams choose review depth to suit the change. A cost comparison needs matched workloads and measured cost per useful finding.

These retrospective runs differ in repository, date, context, reasoning settings, and reviewer availability. Judgments may inherit the lead reviewer’s biases, and the archive does not label every possible bug. We do not infer recall, a universal accuracy rate, or statistical significance from this table.