You run Initech's support bot. The large model gets every question right but costs about 20 times as much as the small one. Your call: which requests can go to the small model without customers noticing?
Key idea: Most requests don't need your biggest model. Route easy ones to a cheap model and hard ones to a strong one, and you keep most of the quality for a fraction of the cost. The catch: hard requests that look easy.
0.95
Finish a run to see what a PM takes away from this lesson.
Accuracy vs. cost
Run a strategy to plot it here
Things to try
- Run Always small, then Always large. Note the gap in accuracy and in cost per 1,000 requests.
- Run Router, then Cascade. Where do their dots land between the two?
- Lower the threshold to 0.5 and rerun Router. Find the short question it now calls easy: short questions can hide hard math.
- Look at Cascade's results: Small rates itself 0.9 or higher even when it's wrong. Push the threshold to 1.0 and rerun. Self-reported confidence is a weak signal.