AI Code Review Benchmark: Which Tool Catches the Most Bugs?

Which AI code review tool catches the most bugs? We benchmarked five leading tools against 118 real production bugs from open-source repositories. Macroscope detected 48.3% (57/118), CodeRabbit detected 45.8% (54/118), Cursor Bugbot detected 42.4% (50/118), Greptile detected 23.6% (17/72), and Graphite Diamond detected 18.3% (21/115). The top three are close, and no tool catches most bugs, which is why human review stays essential. This benchmark is Macroscope's own evaluation; we publish the methodology so you can judge it alongside your own testing.

TL;DR

• Macroscope achieved the highest detection rate at 48.3% (57 of 118 bugs).
• CodeRabbit and Cursor Bugbot were close behind at 45.8% and 42.4% respectively.
• Greptile and Graphite Diamond trailed at 23.6% and 18.3%.
• Detection rate alone is not enough. Precision matters: a tool that comments on everything is noise, not signal.
• This is our benchmark. Treat it as one input and run tools side by side on your own PRs.

To benchmark the effectiveness of Macroscope's code review, we constructed a dataset of real-world production bugs drawn from open-source repositories. We then evaluated Macroscope and other AI code review tools (Graphite Diamond, Cursor Bugbot, Greptile and CodeRabbit) against that dataset. The results of our benchmark showed that Macroscope achieved the highest bug detection rate from our dataset of known bugs.

Dataset Methodology

We assembled our dataset with the following process:

1. We selected 45 popular open-source repositories across the 8 programming languages Macroscope supports (Go, Java, JavaScript, Kotlin, Python, Rust, Swift, TypeScript). We searched their commit logs to find commits labelled as bug fixes.
2. An LLM classified each commit as either a self-contained issue or a context-dependent issue, and either a runtime issue or a subjective code-quality issue (e.g. adding documentation, styling, linting). We kept only the self-contained runtime bugs, since Macroscope Code Review is designed specifically to detect this type of issue. Given that the other tools describe themselves as being capable of detecting self-contained runtime bugs, focusing on this type of issue allowed for an apples-to-apples evaluation.
3. For each bug-fix commit, an LLM analyzed the original commit message and patch to generate a description of the bug that was fixed.
4. We used git blame data to attempt to identify the commit that introduced the bug and used an LLM to try to verify whether the commit identified was indeed the source.
5. We manually reviewed a subset of the dataset and removed any samples where the dataset appeared to be incorrect.

This process ultimately yielded a dataset of 118 runtime bugs across the 45 repositories we sampled. Each sample in our dataset contains the commit hash that we believe introduced the bug, the commit hash that fixed the bug, the patch that fixed the bug, and an LLM-generated description of the bug. Here's a sample of one bug from the dataset:

We identified this bug-fix commit:

• Commit message: fixed overflow error in gdc computation
• Commit Hash: dabf3a5beb9ab697d570154b9961078a8586c787
• Repository: https://github.com/apache/commons-math

We identified this commit as the commit that introduced the bug:

• Commit Hash: 746892442f75845426e16f258d42498ad1de154b

A LLM generated this description of the bug that was fixed:

Description: MathUtils.gcd incorrectly detects zero operands by testing u * v == 0, which can overflow to zero for non-zero inputs (e.g., 65536 × 65536), causing the method to take the zero-case path and return |u| + |v| instead of the correct GCD.

Evaluation Methodology

Once we assembled the dataset of known bugs, we evaluated whether different code review tools detected those bugs. For each bug in our dataset:

1. We forked its repository and created two branches: one at the parent of the commit that introduced the bug (to simulate the state of the codebase before the bug was introduced) and another branch at the bug commit itself (to simulate the state of the codebase when the bug was introduced).
2. We opened a pull request with the base set to the parent commit branch and the head set to the branch with the bug commit. We did this so that the PR diff would be identical to the commit we believed introduced the bug.
3. Each tool ran with its default settings, and ran in isolation within independent GitHub PRs in order to avoid scenarios where one tool might skip reporting a correctly identified bug because another tool already flagged it. Whenever a PR review failed or was rate-limited, we attempted to re-run.
4. After the reviews were complete, an LLM evaluated whether any review comments from each tool matched the bug descriptions generated during dataset assembly.
5. We manually verified all of the matched bugs identified for each tool. We also randomly sampled and verified five non-matches per tool to ensure that the LLM was not misclassifying cases in either direction. Full manual QA of all results was impractical due to the sheer number of aggregate review comments left across all tools, so sampling was used instead.

Due to issues with rate limits and availability, we were not able to have a consistent sample size of bugs across all tools (see this table for the number of bugs evaluated per tool). Most notably, our access to Greptile's code review functionality was disabled midway through our evaluation. We did not include any results from Pull Requests where the tool did not explicitly indicate within the GitHub PR that the review was completed successfully.

Results

We analyzed our results across the following dimensions:

• Known Bug Detection Rate - The percentage of known bugs from the dataset that each code review tool correctly identified in a review comment.
• Price - The standard monthly per-seat price of each code review tool.
• Comment Volume - The average number of review comments left per pull request by each tool.

Known Bug Detection Rate

The results of our benchmark indicate Macroscope achieved the highest detection rate of bugs from our dataset (48%). CodeRabbit and Cursor Bugbot were close behind (46% and 42% respectively), followed by Greptile (24%) and Graphite Diamond (18%).

Detection rates varied across languages. Macroscope had the highest bug detection rates for Go (86%), Java (56%), Python (50%), and Swift (36%), while CodeRabbit had the highest bug detection rates for JavaScript (59%) and Rust (45%). For Kotlin, CodeRabbit and Macroscope tied for the highest detection rate (50%), and for TypeScript, CodeRabbit and Cursor Bugbot tied for highest (36%).

Comment Volume

To evaluate how "loud" or "quiet" each tool is during code reviews, we measured the number of comments left on each pull request we opened. We did not assess the quality, value or correctness of all of these comments. We simply quantified the average number of review comments left per opened PR, so that we could illustrate a normalized volume meter for how "loud" each tool was on its default settings.

Macroscope's average comment volume ranked in the middle of the five tools we evaluated (all on default settings), while CodeRabbit was significantly the "loudest" and Graphite Diamond was the "quietest".

We then narrowed our analysis to focus specifically on review comments that identified runtime issues from our dataset. By excluding style and documentation comments, we were able to do a closer like-for-like comparison of the volume of runtime-related comments across tools. Since each tool categorizes issues differently, we aligned categories as best as we could.

Within this subset of review comments that flagged runtime issues, CodeRabbit still generated the highest average volume of comments (i.e. the "loudest"), while Graphite Diamond generated the least (i.e. the "quietest"). Macroscope again fell between these extremes, generating fewer average runtime-related comments than CodeRabbit, but more than Greptile, Cursor Bugbot, and Graphite Diamond.

Pricing

For this benchmark, we used the minimum plans — both in functionality and term (e.g. monthly) — that allowed us to run the evaluation.

Comparing pricing across code review tools is challenging because the features and functionality offered by each tool aren't directly comparable. Some tools (e.g. Greptile, CodeRabbit) offer customizable rules and configuration options to detect issues beyond runtime issues. In Macroscope's case, we provide a suite of features beyond AI code review that deliver value for leaders and product stakeholders.

Evaluation Results

We plan to continue running internal benchmarking to make sure that we're delivering the best code review experience for our customers.

Limitations

As with any benchmark evaluation, this evaluation has limitations to consider when interpreting the results:

• We only evaluated detection of self contained runtime bugs, and did not assess detection of other types of issues such as style issues, compilation issues etc.
• All tools ran with their default settings (e.g. we did not utilize custom rules, which some tools support). For each tool, we used their minimum plan (i.e. in terms of functionality and term) that enabled us to run our benchmark evaluation.
• Due to issues with rate limits and availability, we were not able to have a consistent sample size of bugs across all tools (see this table for the number of bugs evaluated per tool). Most notably, our access to Greptile's code review functionality was disabled midway through our evaluation. We excluded from our evaluation any Pull Requests where the code review tool did not explicitly indicate within the GitHub PR that the review was completed successfully.
• It is possible that we were invisibly rate-limited by some of these tools, which could have impacted results. To mitigate this risk, for each PR that we included in our result set, we confirmed that each tool indicated a successful PR (e.g. a Check Run shown, or a footer generated which quantified a non-zero number of files reviewed).
• Macroscope has a hybrid pricing model to accommodate unusual circumstances where a customer's codebase activity exceeds typical volumes. In these cases, rather than rate-limit our customers, we allow "usage-based" overages which are charged per commit processed and per PR review. The quantity of usage simulated by this benchmark evaluation is considerably below the threshold that would have triggered usage-based overages on Macroscope's lowest-tier monthly plan.
• We have used the results of this benchmarking data to fix bugs and improve our code review pipeline (e.g. fix bugs in our review pipeline), however we have not encoded any of the bugs in our dataset into our pipelines, nor trained any models on this data. We acknowledge that the other tools we evaluated did not have the same opportunity to fix any issues that they would have encountered with this exact dataset.
• Due to the scale of our dataset, we did not manually validate every false result to ensure that the LLM classification was correct, so it's possible that some false detections (for every tool, including Macroscope) were incorrect. We did manually verify a random subset of 5 false detections per tool.
• Due to the approach we took generating our dataset, the distribution of bugs across languages is uneven (see Chart 1). This unevenness did not benefit the overall results in our favor relative to other tools. In fact, when examining the unweighted averages (i.e. averaging each tool's bug detection rate across their per-language bug detection rates), we observed that our relative overall ranking remained unchanged.
• This evaluation was conducted between August 25 2025 and September 14 2025. Because each tool is constantly changing, it's possible that the results would be different if the same benchmark were rerun today - including for Macroscope.

Frequently Asked Questions

Which AI code review tool catches the most bugs?

On this 118-bug benchmark, Macroscope detected the most at 48.3% (57/118), followed by CodeRabbit at 45.8% (54/118), Cursor Bugbot at 42.4% (50/118), Greptile at 23.6% (17/72, evaluated on 72 bugs), and Graphite Diamond at 18.3% (21/115). This is Macroscope's own benchmark, so treat it as one input and run tools side by side on your own PRs.

What is a good detection rate for AI code review?

No AI code review tool catches most bugs. The highest rate in this benchmark was 48.3%, meaning the majority of bugs still escaped detection. That is why human review stays essential. AI code review is a first pass that catches a meaningful fraction of bugs before humans look; it is not a replacement for human judgment.

How does AI code review differ from static analysis and linters?

Static analysis and linters apply fixed rules and catch known patterns (unused variables, null pointer patterns, style issues). AI code review reasons about behavior and catches novel bugs that no rule describes, like cross-file breakage when a signature changes. Most teams run all three: linters for style, static analyzers for known patterns, and AI code review for the bugs that fall through both. See AI code review vs static analysis for a detailed comparison.

Why does comment volume matter?

A tool that comments on every line will technically find more bugs, but it also generates noise that trains engineers to ignore it. Precision matters as much as recall. In this benchmark, CodeRabbit generated the highest comment volume; Macroscope fell in the middle. Macroscope's Code Review v3 separately reports higher review-comment precision, with 22% fewer comments overall.

How much does each tool cost?

Pricing models vary. Macroscope is usage-based at $0.05 per KB of diff reviewed with no seat fees. CodeRabbit starts at $24/developer/month billed annually. Greptile charges $1 per Base review credit, no seat fees. Cursor Bugbot is usage-based inside Cursor subscriptions. See AI code review pricing for detailed cost modeling.

Does Macroscope work with GitLab or Bitbucket?

Macroscope currently supports GitHub, with GitLab and Bitbucket on the roadmap. CodeRabbit supports GitHub, GitLab, and Bitbucket. If you are on GitLab or Bitbucket today, check each vendor's current platform support before evaluating.

What is codebase-aware AI code review?

Codebase-aware AI code review analyzes a pull request against a model of the whole repository rather than the diff alone. The reviewer knows who calls a changed function, which types flow through it, and what depends on it, so it can flag breakage outside the changed files. Macroscope does this by exploring the surrounding code in your repository on every review. See how AI code review catches bugs across files.

How do AI coding agents affect AI code review?

AI coding agents produce pull requests at higher volume than human developers alone. Those PRs need review just like human-written code. AI code review catches bugs in agent-written code the same way it catches bugs in human-written code. For teams running multiple agents in parallel, cloud AI coding agent orchestration coordinates the agents while AI code review validates their output.