Custom AI Code Review Agents: Automating the Checks Linters Can't

Custom AI code review agents let you write team-specific review rules in plain English and enforce them on every pull request. They run as GitHub check runs that can block merges, connecting to your codebase, git history, Slack, Sentry, and other tools. In Macroscope, these are called Check Run Agents: you create a markdown file in your repository's .macroscope/check-run-agents/ directory, describe what to check, and the agent investigates every PR.

This is what makes custom AI code review agents different from linters and static analysis: they can investigate. They browse your codebase, query related files, check Sentry for production errors, verify feature flags, and reason about what they find. For how this fits into the broader category, see what is agentic CI and how to enforce team coding standards with AI.

This is a practical, use-case-driven guide. Catching bugs and security flaws manually on every PR is nearly impossible, and generic automated tools only cover a narrow slice - bugs, vulnerabilities, runtime errors. Modern teams care about much more: coding conventions, architectural consistency, test coverage, migration safety. The challenge is that every team's priorities differ. What matters to a fintech startup is not what matters to a healthcare platform or an e-commerce company. One-size-fits-all tooling can't address that. Custom AI code review agents can.

TL;DR - Custom AI code review agents

• What: AI checks defined in plain English markdown, one file per check, that run on every PR
• Beyond bugs: Enforce coding principles, generate architecture diagrams, gate risky migrations, verify test coverage
• Tool access: Browse code and git history, plus Slack, Sentry, PostHog, LaunchDarkly, BigQuery, Jira, Linear, and any MCP server
• Enforcement: Each agent's conclusion is neutral (reports findings without failing) or failure (can fail the check, which blocks merging when the check is required)
• Selective: Scope each agent to the files or paths it should watch, so it stays quiet everywhere else
• Cost: Usage-based. $100 in free usage and 1,000 free agent credits every month per workspace

How Can You Write Custom AI Checks That Run on Every Pull Request?

You write a custom AI check as a plain-English markdown file in your repository, and Macroscope runs it as a Check Run Agent, a separate GitHub check run on every pull request open, push and rerun. In practice:

1. Install Macroscope on GitHub and connect any integrations the check needs (Slack, Sentry, PostHog and so on).
2. Add one markdown file per check in .macroscope/check-run-agents/, for example migration-safety.md. Macroscope reads definitions from the PR's latest commit, so the check runs on the PR that adds it.
3. Describe the rule in plain English: what to look for, which files it applies to, and what a finding should say.
4. Set conclusion in frontmatter (neutral by default; failure with input: full_diff to fail on a violation), and when it should block merges, make the check required in branch protection.
5. Cap the spend with maxRuns and maxBudgetPerRun per agent (how to control AI code review costs).

The do-it-yourself route is a GitHub Actions workflow that sends the diff to a model API and posts the result with the GitHub CLI or the Checks API, or running an agent such as Claude Code in Actions (Claude Code on GitHub). It works, but you maintain the prompts, keys, secrets, output formatting and model upgrades yourself. The comparison table below covers the trade-offs.

Beyond Basic Bug Detection

Traditional code review tools focus on a narrow set of problems: bugs, security vulnerabilities, and runtime errors. Those correctness issues are critical - and Macroscope's built-in Correctness check, with Fix It For Me, is built precisely for them. But modern development teams need more than correctness.

They need to enforce coding conventions, maintain architectural consistency, verify test coverage, and ensure database migrations follow best practices. These are judgment calls, not pattern matches. A linter can tell you a variable is unused. It cannot tell you whether a new API endpoint follows your naming convention, whether a migration has a rollback path, or whether a change to the payments flow might break event tracking.

The deeper problem is that every team's priorities are different. Generic tooling optimizes for the average team, which means it under-serves every specific one. Custom AI code review agents flip that: instead of adapting your standards to the tool, you encode your standards into the tool, in the same language you'd use to explain them to a senior engineer.

What Custom AI Code Review Agents Do Differently

The thing that separates a custom AI code review agent from a custom rule in a traditional tool is that the agent can investigate. It does not just run a pattern against the diff and pass or fail. It browses your codebase, reads related files, queries git blame, checks external systems, and reasons about what it finds - all within a single check run.

That investigation is what makes the following capabilities possible:

• Customizable prompts. Define exactly what each agent looks for, from a specific coding pattern to an architectural principle, in natural language.
• Selective execution. Scope an agent to run only when certain files or paths change, so it focuses attention where it matters and stays quiet everywhere else.
• Tool integration. Equip an agent with capabilities like Slack notifications, database queries, or diagram generation so it can do real analysis, not just text matching.
• Flexible enforcement. Set each check to fail and block the merge (conclusion: failure on a required check) or simply report findings (conclusion: neutral).

It runs on advanced reasoning models, so it catches nuanced violations that simple pattern matching would miss. And because the instructions live in a markdown file in your repo, anyone on the team can read, review, and change them - no YAML pipeline, no custom GitHub Action, no ESLint plugin to maintain.

Real-World Applications

The best way to understand custom AI code review agents is to see what teams actually build with them. Here are four high-leverage patterns.

Enforcing Coding Principles

A principles agent monitors for the common mistakes that slip past traditional linters. For example, you might configure an agent to catch developers accidentally mangling a primary key - a subtle but critical error that causes data-integrity issues no regex would flag. The agent reviews every pull request, understands your codebase context, and flags the violation before it reaches production. Because it runs on a reasoning model, it understands the intent of the principle, not just a literal pattern.

---
title: Coding Principles
model: claude-opus-4-6
effort: medium
tools:
  - browse_code
  - git_tools
  - modify_pr
---

Review this PR against our core data-integrity principles:

- Primary keys must never be reassigned, mutated, or reused. Flag any
  code that writes to a PK column after row creation.
- Money is stored in integer cents, never floats. Flag float arithmetic
  on currency values.
- Soft-deleted rows must stay excluded from default queries.

For each violation: file, line, the principle broken, and a fix.
If nothing is found, report that all principles passed.

Automated Architecture Diagrams

An architecture-visualization agent analyzes all the changes in a pull request and generates a diagram showing how the touched components interact. This helps human reviewers quickly grasp the scope and impact of a change, especially on a complex PR that touches multiple systems. The agent can post a summary to a Slack channel so the broader team stays informed about significant architectural changes - turning a wall of diff into a picture a reviewer can read in seconds.

Database Migration Safety

Database changes are high-risk. A migration-checklist agent verifies that the developer followed every required step before the PR can merge: a rollback procedure exists, the migration was tested against production-like data volumes, index performance was considered, and breaking changes are documented. Set this one to block the merge when a step is missing, and pair it with branch protection so a risky migration can't slip through on a Friday afternoon.

---
title: Migration Safety
model: claude-opus-4-6
reasoning: high
effort: high
input: full_diff
conclusion: failure
tools:
  - browse_code
  - git_tools
  - modify_pr
---

If this PR adds or changes a database migration, verify ALL of:

1. A rollback / down migration is included.
2. New columns on large tables are nullable or have a safe default
   (no blocking table rewrites).
3. New indexes are created concurrently where the engine supports it.
4. Any breaking schema change is documented in the PR description.

If any item is missing, the check MUST fail. List exactly what's missing.
If there is no migration in this PR, pass immediately.

Test Coverage Verification

A unit-test agent analyzes whether new code includes appropriate test coverage, checks that tests follow team conventions, and suggests additional cases based on the code's complexity and risk profile. Unlike a coverage percentage gate, it reasons about which new logic is risky and therefore which missing tests actually matter - so it nudges toward meaningful tests rather than gaming a number.

The Integration Ecosystem

Custom AI code review agents become genuinely powerful when connected to the tools your team already uses. Macroscope ships native integrations:

• Communication: Slack for notifications and team updates
• Project management: Linear and Jira for ticket tracking
• Feature flags: LaunchDarkly for deployment coordination
• Analytics: PostHog and Amplitude for product insights
• Monitoring: Sentry for error tracking
• Cloud and data: BigQuery and GCP cloud logging for production data analysis

Beyond the native set, MCP (Model Context Protocol) connectors let agents reach virtually any tool. Teams commonly wire in monitoring platforms, search tools, and on-call systems through MCP.

The key advantage is that you configure an integration once, then make it available to any agent. Each agent accesses exactly the tools it needs for its specific job. That is what lets a single agent check the diff, look up the last 90 days of Sentry errors for the modified file, verify the relevant LaunchDarkly flag, and post a summary to Slack - all in one check run. No glue scripts, no separate webhooks, no integration to maintain per tool.

Balancing Automation and Flexibility

Not every check should block a pull request. Some findings are informational; others are critical. In Macroscope's Check Run Agents, the conclusion frontmatter field sets the most severe result an agent may report. The Check Run Agents configuration docs (checked 5 November 2026) list two values:

conclusion	Behavior	Use it for
failure	The check fails when the agent finds a violation; a clean result still passes. Requires input: full_diff, code_object or pr_metadata, and blocks merging only when the check is required in branch protection	Security, migration safety, mandatory standards
neutral (default)	The agent reports findings in the check and as comments, without failing the check	Conventions, new agents you're still tuning, low-stakes nudges

There is no separate "warning" level: a neutral agent's findings are visible, and the developer can proceed. This supports a sensible rollout: start informational, gather data on the issues that actually recur, and raise enforcement as the patterns become clear. You earn trust in an agent before you let it block anyone.

How Custom AI Code Review Agents Compare to the Alternatives

Most tools that advertise "custom rules" offer configuration, not investigation. The difference shows up the moment a check needs to look beyond the diff.

	Custom AI Code Review Agents	CodeRabbit custom rules	Greptile	Custom GitHub Actions	Linters (ESLint/Semgrep)
Definition	Plain-English markdown	YAML config	Learns from PR comments	Code + YAML	Rule files
Investigates beyond the diff	Yes - browses code, git, external tools	No	Learned patterns only	Build it yourself	No
External integrations	Slack, Sentry, PostHog, LaunchDarkly, BigQuery, Jira, Linear, MCP	None from rules	None from rules	Build it yourself	None
Can block merges	Yes	Yes	Comments only	Yes	Yes
Maintenance	Edit a markdown file	Edit YAML	Auto (drifts)	Maintain code + prompts	Maintain rules
Pricing	Usage-based	Per-seat	$1 per Base review credit, no seat fees	Actions minutes + API	Varies

CodeRabbit's custom rules tell the tool "be stricter in the auth directory." A custom AI code review agent says "for every modified file in auth, look up the last 90 days of Sentry errors, check the relevant feature flag, and post a summary to #security." For teams evaluating CodeRabbit alternatives or Greptile alternatives, that gap - investigation and tool access versus configuration - is usually the deciding factor.

A caveat worth stating plainly: these agents are complementary to static analysis, not a replacement. Deterministic rules - formatting, import order, banned functions - are faster and cheaper with ESLint, Semgrep, or golangci-lint. Reach for an agent when the check requires judgment, context, or a query to another system.

Getting Started

Setting up custom AI code review agents is straightforward:

1. Install the GitHub app to connect your repositories. Every workspace gets $100 in free usage and 1,000 free agent credits each month.
2. Configure integrations in the Connections tab for any external tools your agents will use (Slack, Sentry, PostHog, and so on - optional, only if you reference them).
3. Create your first agent by adding a markdown file to the .macroscope/check-run-agents/ directory: define its purpose, scope it to the files it should watch, and write the prompt describing what it checks.
4. Choose its tools and conclusion - start with the default neutral while you tune it.
5. Test on a few PRs and refine. Agents improve as you sharpen the instructions.

Start simple. A PR-labeling agent or a changelog enforcer is a good first build before you graduate to a merge-blocking security or migration check. For the full configuration reference - every frontmatter field, the complete tool list, and more worked examples - see the Check Run Agents guide, and for the broader concept, what agentic CI is.

Why This Matters in the Age of AI-Generated Code

Custom AI code review agents represent a shift in how teams approach code quality. The old choice was between manual review - thorough but slow - and generic automated tools - fast but inflexible. Now teams can automate their specific standards and practices, in their own words.

The result is faster pull requests, fewer bugs reaching production, and more consistent adherence to team conventions. Developers spend less time on repetitive checks and more on real problems. Senior engineers can encode their hard-won expertise into agents that guide the whole team, so a principle a staff engineer cares about gets enforced on every PR whether or not they're the reviewer.

This matters more as AI coding agents write an increasing share of pull requests. When AI generates the code, you need review with the same depth and judgment a senior engineer would bring - not pattern matching, but investigation against the full context of your codebase, your production systems, and your team's standards. The blank-canvas approach means that as your team's needs evolve, your code review automation evolves with them.

Frequently Asked Questions

How can I write custom AI checks that run on every pull request?

Add a plain-English markdown file to .macroscope/check-run-agents/ in your repository, and Macroscope runs it as a Check Run Agent, a GitHub check run on every pull request. Optional frontmatter sets the model, effort, file scope, tools and conclusion, so a check can stay informational (neutral) or fail and block merges (failure, with input: full_diff and a required check). Agents can browse the codebase and git history and query Slack, Sentry, PostHog, Jira, Linear and MCP servers. Runs are billed in Agent credits ($0.01 each, 1,000 free per workspace per month). Full reference: the Check Run Agents guide.

What are custom AI code review agents?

Custom AI code review agents are AI-powered checks you define in plain English that run automatically on every pull request. In Macroscope they're called Check Run Agents: you write what to check in a markdown file in your repo's .macroscope/check-run-agents/ directory, give the agent access to your codebase and connected tools, and it runs as a GitHub check run. Unlike generic tools that only catch bugs and vulnerabilities, custom agents enforce team-specific standards like coding conventions, architectural consistency, migration safety, and test coverage.

How are custom AI agents different from linters and custom rules?

Linters and custom rules match patterns; AI agents investigate. A linter checks the diff against a fixed pattern and passes or fails. A custom AI code review agent browses your codebase, reads related files, queries git history, checks external systems like Sentry and LaunchDarkly, and reasons about what it finds. CodeRabbit's custom rules, for instance, adjust review behavior via YAML configuration. A custom AI agent can look up the last 90 days of production errors for a modified file and post a summary to Slack - investigation and tool access that configuration alone can't match.

What can custom AI code review agents check beyond bugs?

They can enforce coding principles (like never mutating a primary key), generate architecture diagrams from a PR's changes, gate risky database migrations on a safety checklist, verify meaningful test coverage, enforce API naming conventions, auto-label PRs, and check that production systems are healthy before merge. Anything you could explain to a senior engineer as a review standard, you can encode as an agent - provided it requires judgment or context rather than a simple deterministic pattern.

Can a custom AI code review agent block a pull request from merging?

Yes. Each agent's conclusion is either neutral (the default: findings are reported without failing the check) or failure (the check fails on a violation; this needs input: full_diff, code_object or pr_metadata). Set a migration-safety or security agent to conclusion: failure, mark its check required in GitHub branch protection, and a PR with a missing rollback or a hardcoded secret can't merge until it's fixed. A good rollout starts agents at neutral, then switches to failure once the patterns are clear.

What tools and integrations can custom AI code review agents use?

By default they can browse code, read git history, and read GitHub PR metadata, and update the PR. Optional integrations include Slack, Sentry, PostHog, Amplitude, LaunchDarkly, BigQuery, GCP cloud logging, and issue trackers like Jira and Linear. Beyond the native set, MCP connectors let agents reach virtually any tool, including monitoring platforms, search tools, and on-call systems. You configure an integration once and make it available to any agent.

How much do custom AI code review agents cost?

Pricing is usage-based, not per-seat. Custom agents bill under Macroscope's Agent credits at $0.01 per credit, and every workspace gets 1,000 free agent credits each month. A run's credits are its raw LLM cost plus a 5% markup, divided by $0.01, so cost depends on the model, effort level and how much the agent reads. New workspaces also start with $100 in free usage. You control spend by scoping agents tightly with file patterns, choosing an appropriate model and effort level for each check, and setting maxRuns and maxBudgetPerRun (cost controls guide).

Are custom AI agents a replacement for static analysis like ESLint or Semgrep?

No - they're complementary. Static analysis tools are fast and deterministic, ideal for pattern-based rules like import ordering, formatting, and banned functions. Custom AI code review agents handle what static analysis can't: business-logic validation, cross-system verification (checking Sentry errors or feature-flag states), contextual judgment, and natural-language standards that resist formalization as regex or AST rules. The strongest setups use both - linters for deterministic rules, agents for investigative checks.

How do I create my first custom AI code review agent?

Install the Macroscope GitHub app, then add a markdown file to the .macroscope/check-run-agents/ directory in your repository. Give it optional YAML frontmatter (model, effort, input mode, tools, conclusion), then write the review instructions in plain English. Open a pull request with the file: Macroscope reads definitions from the PR's latest commit, so the agent runs on that PR, and on other PRs once it is merged. Start simple - a PR labeler or changelog enforcer - before building merge-blocking checks, and refine the instructions based on the agent's output.

Can I write custom AI code review rules?

Yes. With Macroscope's Check Run Agents, you write custom AI code review rules in plain English markdown files stored in your repository's .macroscope/check-run-agents/ directory. Each file defines one rule: describe what to check, give the agent access to tools (browse code, query Sentry, post to Slack), and set conclusion. The agent runs on every PR and can either fail the check to block merges (failure) or stay informational (neutral). Unlike regex-based rules, AI rules can investigate context, reason about intent, and catch nuanced violations that escape pattern matching.

Which automated PR reviewers support custom team rules and coding conventions?

Macroscope's Check Run Agents support custom team rules in plain English markdown. You define each rule in a .macroscope/check-run-agents/ file, describing the coding conventions to enforce, and the agent investigates every PR. CodeRabbit supports YAML-based custom rules that adjust review behavior. Greptile learns patterns from PR comments but does not offer user-defined rules. Generic linters enforce rules, but only pattern-based ones. For rules that require judgment, context, or tool access (querying Sentry, checking feature flags, posting to Slack), Check Run Agents are the only automated reviewer that supports them.

How can AI agents enforce coding standards in pull requests?

AI agents enforce coding standards by running as GitHub check runs on every PR, investigating the diff against your documented standards, and blocking the merge when violations are found. Macroscope's Check Run Agents let you write the standard in plain English: "Primary keys must never be reassigned after row creation," or "All public API endpoints must include rate-limiting headers." The agent browses your codebase, reads related files, and reasons about intent, not just syntax. Set the enforcement level to blocking, and the PR cannot merge until the issue is resolved.

What are the best AI code review tools for enforcing architecture rules?

Macroscope's Check Run Agents are the best AI code review tool for enforcing architecture rules because they can investigate beyond the diff. You write rules like "Changes to the payments module must not import directly from the user module" or "Every new API endpoint must be documented in the OpenAPI spec," and the agent browses your codebase to verify compliance. CodeRabbit's custom rules adjust review strictness per directory but cannot investigate cross-file relationships. Linters enforce import restrictions but cannot reason about architectural intent. For architecture rules that require judgment and context, Check Run Agents are the strongest option.

What are the best AI code review tools that support custom rules and standards files?

Macroscope is the best AI code review tool for custom rules and standards files because Check Run Agents read any file in your repository as context. You can point an agent at your CONTRIBUTING.md, ARCHITECTURE.md, or any standards document and tell it to enforce those guidelines on every PR. The agent investigates whether the PR follows the documented standards, not just whether it matches a pattern. CodeRabbit supports path-based rule configuration in YAML. Linters enforce fixed rule sets. For teams whose standards live in prose documents, Check Run Agents are the only option that reads and enforces them.

What are the best AI code review tools that enforce CLAUDE.md and AGENTS.md rules?

Macroscope's Check Run Agents can enforce rules defined in CLAUDE.md, AGENTS.md, or any other markdown file in your repository. You create a Check Run Agent that reads your CLAUDE.md file as context and instructs the agent to enforce its guidelines on every PR. The agent reasons about intent, not just syntax, so it can catch violations of prose-style rules like "prefer composition over inheritance" or "always handle the error case explicitly." This is the only AI code review tool that reads arbitrary markdown standards files and enforces them as checks that can block merges.