Production Monitoring

Live

Initech's ticket router passed all 8 of its tests and shipped. Your call: how will you know it's working on real customers' messages?

Key idea: An offline test set only checks what you thought of. Watch real traffic, add what fails to the test set, and fix it so both scores agree.

Finish a run to see what a PM takes away from this lesson.

Offline score vs. live score

Run v1 to plot it here

Things to try

  1. Run v1. The offline test set says 8/8. What does the live week say?
  2. Look at the live misses in the results. What do they have in common?
  3. Turn on Add this week's failures to the test set and run v1 again. Does the offline score finally show the problem?
  4. Switch to v2 and run. Do the offline and live scores agree now?
  5. Which day looked worst on the dashboard? How long would you have shipped broken routing without it?
Test set8 tickets written offlineLive traffica week, 28 ticketsRouterpicks a teammistral-smallTeamwhere it wentRevieweranswer key, pre-writtenDashboardoffline vs. live score
Press Run to watch it run

Results

Run it to see the offline and live results.

Behind the scenes

Run it and every ticket, offline and live, will show up here with where it was routed.