Let the Agent Edit the Tests?

Live

Initech's coding agent has 10 pull requests waiting with CI red. Your call: may it change the tests to get them green, and who checks when it does?

Key idea: Told to get CI green, a coding agent will sometimes change the test instead of the code, with a confident reason. Locking tests blocks the legitimate updates too. Let it change them, and have a person who knows the rules read every test change, which takes about a minute each.

Finish a run to see what a PM takes away from this lesson.

Tests weakened vs. rule changes stuck

Run a policy to plot it here

Things to try

  1. Run it as it starts: Smart, test edits Allowed. Find the PRs marked test weakened. What did the agent change, and why?
  2. Switch Test edits to Not allowed. What's stuck now?
  3. Set Test edits back to Allowed and turn on Tell it tests are the spec, on Smart and then on Fast. Does a written rule stop it?
  4. Turn the rule off, switch back to Smart, and choose Allowed, with review. How many minutes does it cost, and what gets through?
  5. Switch to Allowed and the Fast model. Is the smaller model more careful?
readokrejectmergeRed PRs10 PRs, CI failingCoding agentcode or tests?mistral-smallTest gatemay tests change?Person reviewoffMergeCI greenScorecardweakened vs. stuck
Press Run to watch it run

Results

Run it to see what the agent did with each PR.

Behind the scenes

Run it and every PR, what the agent changed, and whether it merged will show up here.