Reviewing AI-Written Code

Live

Initech's coding agent opened 12 pull requests today, and the team has one afternoon of review time. Your call: which PRs get a person, and what has to be true before the rest merge?

Key idea: Agents write code faster than people can read it, so reviewing everything stops scaling. Tests catch only what they check, and an AI reviewer that can't see the ticket can't tell when the code does the wrong thing. Give the reviewer the ticket and small PRs, and save people for the risky changes.

Finish a run to see what a PM takes away from this lesson.

Bugs shipped vs. a person's time

Run a policy to plot it here

Things to try

  1. Run Tests only. Which of the 5 bugs did the tests stop?
  2. Run AI reviewer as it starts: Fast, no ticket. Did it catch the export anyone can download?
  3. Turn on Link the ticket and run AI reviewer again. Which bugs does it catch now?
  4. Turn on Split big PRs too and run it again. What changed, and what did it cost?
  5. Switch the reviewer to Smart, keeping the ticket and split PRs. Then turn the ticket off. Is the bigger model the better reviewer?
  6. Turn the ticket off and run A person reviews all, then AI + person on risky PRs, with the PRs split. Compare the minutes, and what slipped through.
readokbugmergeAgent PRs12 PRs, CI ranAI reviewerapprove or blockministral-8bPolicy gatemerge, or a person?Person reviewoffMergeships to customersScorecardbugs vs. minutes
Press Run to watch it run

Results

Run the review to see what happened to each PR.

Behind the scenes

Run the review and every PR, what the tests and the reviewers said, and whether it merged will show up here.