Reasoning Models

Live

Initech's assistant answers billing questions, some of them multi-step math. A reasoning model can think before it answers, but it costs more and makes customers wait. Your call: how much thinking is worth paying for?

Key idea: Reasoning models think before they answer. In our run a little thinking fixed most multi-step problems and Medium was the only level to get them all; High cost more time and money without doing better, and easy questions never needed it.

Finish a run to see what a PM takes away from this lesson.

Questions right vs. time

Run an effort level to plot it here

Things to try

  1. Run Minimal. Which hard questions does it miss, and how fast is it?
  2. Run Low. How many more does it get right, and what did that cost in time and tokens?
  3. Run High. Did all that extra thinking buy any accuracy over Low?
  4. Look at the easy questions across your runs. Did thinking ever help them?
thinkplanQuestions8 hard, 4 easyModelreads the questiongpt-5-nanoThinkingoffAnswerfinal line onlyGraderright or wrong?Scorecardaccuracy, time, cost
Press Run to watch it run

Results

Run the questions to see each answer.

Behind the scenes

Run the questions and every answer, with how long the model took, will show up here.