Guides / Code review
How to evaluate AI code review on your own pull requests
Choose an AI code reviewer by running it on changes your team understands. Record which findings deserve a fix, which claims are wrong, and how much time the team spends checking them. A comment count alone cannot tell you whether a reviewer helped.
HeyDeer team ·
Download the evaluation worksheet (CSV)Here is a trial process you can use with HeyDeer or another review tool.
Pick changes that represent your work
Start with a small set of recent pull requests: a routine feature, a bug fix, a refactor, and a change to a sensitive path such as authorization or billing. Include changes that your team believes are correct. Those help reveal unsupported warnings.
Ten PRs can be a manageable starting point for a pilot. It is not enough to establish a universal accuracy ranking. Expand the sample before making claims beyond your own team’s experience.
For every PR, save the base and head commit hashes. If you compare tools, give them the same revision. A review of the original change and a review after fixes answer different questions.
Record the context and settings
Keep the repository instructions, review depth, included files, and available context alongside the results. Record the date and product configuration too. If one reviewer receives an explanation of an important business rule, give the other reviewer equivalent information where its configuration allows it. Document any difference you cannot remove.
Evaluate the workflow you would use after the trial. If your normal process includes a follow-up review after a fix, include that run in the time and cost records.
Judge each claim against the code
Have a developer familiar with the change classify each distinct finding. When a claim is disputed, record the evidence and ask another developer to check it.
| Classification | What it means |
|---|---|
| Valid and actionable | The issue exists, and the team would change the code because of it. |
| Valid, deferred | The issue exists, but fixing it is outside this change or not currently worth the cost. |
| False positive | The claimed issue does not exist under the relevant code and requirements. |
| Duplicate | Another finding already describes the same underlying issue. |
| Unresolved | The available evidence is insufficient to decide. |
A comment that the team ignores is not automatically false. It may be correct but low priority, already known, or hard to understand. Keep those reasons separate so that the configuration change addresses the actual problem.
Count repeated descriptions of one bug as one distinct issue. Retain the duplicate count separately to measure how much extra reading the reviewer creates.
Measure the time needed to use the feedback
For each PR, record the tool’s elapsed review time, the developer’s time checking findings, and the actual charge. Keep code-fixing time separate: fixing a useful discovery is different from spending ten minutes disproving a warning.
Open the CSV worksheet in your spreadsheet app. Use one row per tool and review run; keep follow-up runs on separate rows. Record costs in the same currency when comparing tools.
| Record | Include |
|---|---|
| PR and revision | PR URL, base commit, head commit, and review date. |
| Configuration | Tool and version, review depth, instructions, files, and available context. |
| Findings | Counts for each classification above, with evidence for disputed claims. |
| Time and cost | Elapsed review minutes, developer triage minutes, actual charge, and currency. |
Report the counts before reducing everything to a score. If you calculate precision, the share of assessed findings judged correct, define which findings enter its denominator. Keep unresolved findings visible. If no findings were reported, precision is undefined rather than 100%.
Recall, the share of real defects a reviewer finds, needs a known set of defects. Reviewing your ordinary PRs does not establish that every bug has been found. You can add a separate exercise using known historical defects, but label those results separately and withhold the later fix from the reviewer.
Compare seat fees and usage-based pricing
No seat fees means adding a developer does not add a per-person charge. A product can still have a subscription, usage charges, or both. Compare the total bill for the same workload, including follow-up reviews after a new push.
| Billing model | What to check |
|---|---|
| Per developer | Who counts as a paid seat, whether occasional contributors count, and whether reviews have separate usage limits or charges. |
| Shared workspace credits | The subscription, included credits, additional usage charges, expiry rules, and parallel review limits. |
| Per repository | Which repositories are billed and whether activity, repository size, or review volume affects the price. |
| Bring your own model key | Any platform fee plus the model provider’s bill. Self-hosted tools can also require infrastructure and maintenance. |
Shared credits can suit a team with many occasional contributors. A seat-based plan may cost less for a smaller team running frequent reviews, depending on its limits. Neither model is always cheaper. For example, 100 PRs with two review runs each means budgeting for 200 runs, not 100. This is an illustrative workload, not what a HeyDeer plan includes.
Use the actual charges in your trial worksheet to estimate a normal month and a busier month. Compare recurring prices over the same billing period, separate introductory discounts from renewal prices, and check whether tax is included. Keep developer time saved separate from the amount you pay for the tool.
How HeyDeer fits
HeyDeer reviews GitHub pull requests with a workspace plan and shared credits. Every plan includes unlimited developers and repositories, with no per-developer seat fees. Charges depend on the review work performed, so a plan does not promise a fixed number of PR reviews.
Each private repository review, including the first, uses credits for the work performed, even when it finds no issues. Reviewing another push or running another review on the same PR uses more credits. Free includes 1,000 credits in the first month from workspace creation, then 300 each month. Public repository reviews are free within fair-use limits. Monthly credits expire at the next refresh; add-on credits are valid for 12 months. Additional credits are purchased automatically only when a workspace admin turns on auto-reload.
The cost calculator estimates the number of review runs with a few repository checks. Count each new push and repeated review separately; more checks can increase usage.
Check the current plans and cost calculator, credit rules, and spending controls against your observed workload before choosing a plan.
Choose a rollout threshold before expanding
Decide what would make the trial useful for your team. That might be finding previously missed issues in a critical path while keeping triage time acceptable. A threshold should reflect your repository and reviewers, not a number borrowed from another company’s benchmark.
In HeyDeer, start with manual reviews on one repository. Use the same worksheet to check whether a change to instructions or feedback scope improves the results. Once the team trusts the feedback, expand automatic reviews gradually. The rollout guide describes those settings, and reducing noise maps common problems to specific adjustments.
Our internal evaluation explains how we assessed reviewer contributions during development. It does not independently rate the completed HeyDeer workflow or establish its recall. Use it as context for the evaluation method, then test the product against your own changes.
Compare the current plans with your observed trial charges. The credit guide explains what each review uses and where to see its final cost.
Try the method on your own pull requests
Explore an example review, or start using HeyDeer for free and record your findings in the worksheet. The quickstart walks you through connecting a repository and running your first review.