Spec-Driven Development

Live

Initech's coding agent will build 8 billing and account features this sprint. Your call: how much do you write down before it starts, and is a one-line ticket ever enough?

Key idea: A coding agent told to keep going builds what the ticket says and fills every gap with a guess, and nothing in its work marks which parts were guesses. The edge cases you write down are the ones it gets right, so with agents the PM's spec becomes the product.

Finish a run to see what a PM takes away from this lesson.

Edge cases right vs. words in the tickets

Build the features to plot the run here

Things to try

  1. Run it as it starts: Fast, one-line ticket. How many of the 32 edge cases did the agent get right, and can you tell which answers were guesses?
  2. Switch to Acceptance criteria and run it again. Which features improved, and which rows say "in the ticket, ignored"?
  3. Switch to Full spec. What's still wrong, and why?
  4. Go back to Acceptance criteria and turn on Let the agent ask first. Read its questions in the log. Do they help?
  5. Turn off Let the agent ask first, switch the agent to Smart, and run the one-line ticket. Does a bigger model guess better?
asksTicket8 featuresPMoffCoding agentbuilds each featureministral-8bEdge cases32 checksScorecardright vs. guessed
Press Run to watch it run

Results

Build the features to see what the agent did on each edge case.

Behind the scenes

Build the features and every ticket, question and edge case will show up here.