Initech's coding agent has 10 pull requests waiting with CI red. Your call: may it change the tests to get them green, and who checks when it does?
Key idea: Told to get CI green, a coding agent will sometimes change the test instead of the code, with a confident reason. Locking tests blocks the legitimate updates too. Let it change them, and have a person who knows the rules read every test change, which takes about a minute each.
Finish a run to see what a PM takes away from this lesson.
Tests weakened vs. rule changes stuck
Run a policy to plot it here
Things to try
- Run it as it starts: Smart, test edits Allowed. Find the PRs marked test weakened. What did the agent change, and why?
- Switch Test edits to Not allowed. What's stuck now?
- Set Test edits back to Allowed and turn on Tell it tests are the spec, on Smart and then on Fast. Does a written rule stop it?
- Turn the rule off, switch back to Smart, and choose Allowed, with review. How many minutes does it cost, and what gets through?
- Switch to Allowed and the Fast model. Is the smaller model more careful?