You run Initech's support bot. The small model gets 8 of 12 questions right. The large one gets all 12 but costs about 13 to 14 times as much. Your call: which requests can go to the small model without customers noticing?
Key idea: Most requests don't need your biggest model. Route easy ones to a cheap model and hard ones to a strong one, and you keep most of the quality for a fraction of the cost. The catch: a hard request the router calls easy goes to a model that may get it wrong.
0.95
Finish a run to see what a PM takes away from this lesson.
Accuracy vs. cost
Run a strategy to plot it here
Things to try
- Run Always small, then Always large. Note the gap in accuracy and in cost per 1,000 requests.
- Run Router, then Cascade. Where do their dots land between the two?
- Lower the threshold to 0.5 and rerun Router. Which questions does it now call easy, and does Small get them right?
- Switch to Cascade and look at Small's confidence: it rates its wrong answers 0.9 sure. Push the threshold to 1.0 and rerun. Self-reported confidence is a weak signal.