Model Routing

Live

You run Initech's support bot. The small model gets 8 of 12 questions right. The large one gets all 12 but costs about 13 to 14 times as much. Your call: which requests can go to the small model without customers noticing?

Key idea: Most requests don't need your biggest model. Route easy ones to a cheap model and hard ones to a strong one, and you keep most of the quality for a fraction of the cost. The catch: a hard request the router calls easy goes to a model that may get it wrong.

0.95

Finish a run to see what a PM takes away from this lesson.

Accuracy vs. cost

Run a strategy to plot it here

Things to try

  1. Run Always small, then Always large. Note the gap in accuracy and in cost per 1,000 requests.
  2. Run Router, then Cascade. Where do their dots land between the two?
  3. Lower the threshold to 0.5 and rerun Router. Which questions does it now call easy, and does Small get them right?
  4. Switch to Cascade and look at Small's confidence: it rates its wrong answers 0.9 sure. Push the threshold to 1.0 and rerun. Self-reported confidence is a weak signal.
escalateRequests12 support questionsRouteroffSmall modelcheap, fastgpt-4o-miniLarge modeloffGraderchecks the answerScorecardaccuracy and cost
Press Run to watch it run

Results

Run a strategy to see how each request was handled.

Behind the scenes

Run the requests and every routing decision will show up here.