Initech's coding agent will build 8 billing and account features this sprint. Your call: how much do you write down before it starts, and is a one-line ticket ever enough?
Key idea: A coding agent told to keep going builds what the ticket says and fills every gap with a guess, and nothing in its work marks which parts were guesses. The edge cases you write down are the ones it gets right, so with agents the PM's spec becomes the product.
Finish a run to see what a PM takes away from this lesson.
Edge cases right vs. words in the tickets
Build the features to plot the run here
Things to try
- Run it as it starts: Fast, one-line ticket. How many of the 32 edge cases did the agent get right, and can you tell which answers were guesses?
- Switch to Acceptance criteria and run it again. Which features improved, and which rows say "in the ticket, ignored"?
- Switch to Full spec. What's still wrong, and why?
- Go back to Acceptance criteria and turn on Let the agent ask first. Read its questions in the log. Do they help?
- Turn off Let the agent ask first, switch the agent to Smart, and run the one-line ticket. Does a bigger model guess better?