Key idea: Most requests don't need your biggest model. Route easy ones to a cheap model and hard ones to a strong one, and you keep most of the quality for a fraction of the cost. The catch: hard requests that look easy.
0.95
Accuracy vs. cost
Always smallAlways largeRouterCascade
- Run Always small, then Always large. Note the gap in accuracy and in cost per 1,000 requests.
- Run Router, then Cascade. Where do their dots land between the two?
- Lower the threshold to 0.5 and rerun Router. Find the short question it now calls easy: short questions can hide hard math.
- Look at Cascade's results: Small says it's sure even when it's wrong, so little escalates. Push the threshold to 1.0 and rerun. Self-reported confidence is a weak signal.
- Try a few thresholds and watch each strategy's dot slide between cheap and accurate.