You're about to change the prompt behind Initech's ticket triage. Your call: how will you know the new version is better, and that it didn't break something else?
Key idea: An eval is a fixed test set plus graders, giving a score you can compare across changes.
Prompt under test
Graders and model
Finish a run to see what a PM takes away from this lesson.
Things to try
- Run v1, then v2. Watch the format pass rate jump when the prompt spells out the output.
- Run v2, then v3, and compare the angry ticket and the "ignore all instructions" ticket.
- Switch to the Smart model and rerun v1. Does a bigger model fix a vague prompt?
- Switch back to Fast, run v3, then turn off the LLM judge and run it again. The run is faster and cheaper, but what does it miss?
- Edit the prompt yourself and try to get 8/8.