Initech's ticket router passed all 8 of its tests and shipped. Your call: how will you know it's working on real customers' messages?
Key idea: An offline test set only checks what you thought of. Watch real traffic, add what fails to the test set, and fix it so both scores agree.
Finish a run to see what a PM takes away from this lesson.
Offline score vs. live score
Run v1 to plot it here
Things to try
- Run v1. The offline test set says 8/8. What does the live week say?
- Look at the live misses in the results. What do they have in common?
- Turn on Add this week's failures to the test set and run v1 again. Does the offline score finally show the problem?
- Switch to v2 and run. Do the offline and live scores agree now?
- Which day looked worst on the dashboard? How long would you have shipped broken routing without it?