The Bug Report Is the Prompt

Live

Initech's support team files bugs straight into the coding agent's queue, most of them a single line saying something is broken. Your call: do reporters fill in steps, the error and the expected result before the agent starts, or does the agent ask?

Key idea: Told to keep going, a coding agent given a vague bug report picks the likeliest fix and ships it, and that fix sometimes hides the symptom instead of fixing the cause. Steps, the error message and the expected result make the cause findable, so the bug report is the agent's spec.

Finish a run to see what a PM takes away from this lesson.

Bugs fixed vs. words in the reports

Fix the bugs to plot the run here

Things to try

  1. Run it as it starts: Smart, It's broken. How many of the 8 bugs did the agent fix, and how many of its fixes only hide what the reporter saw?
  2. Switch to Steps to reproduce and run it again. Which bugs did the steps settle?
  3. Switch to Full report (steps, the error, and what should have happened). What changed?
  4. Go back to It's broken and turn on Let the agent ask first. Read its questions in the log. Which ones paid off?
  5. Turn off Let the agent ask first, switch the agent to Fast and run It's broken. Does the smaller model guess better?
asksBug report8 bugsReporteroffCoding agentpicks a causemistral-smallFixcause + fileScorecardcause vs. symptom
Press Run to watch it run

Results

Fix the bugs to see what the agent picked for each one.

Behind the scenes

Fix the bugs and every report, question and pick will show up here.