Initech's assistant answers billing questions, some of them multi-step math. A reasoning model can think before it answers, but it costs more and makes customers wait. Your call: how much thinking is worth paying for?
Key idea: Reasoning models think before they answer. In our run a little thinking fixed most multi-step problems and Medium was the only level to get them all; High cost more time and money without doing better, and easy questions never needed it.
Finish a run to see what a PM takes away from this lesson.
Questions right vs. time
Run an effort level to plot it here
Things to try
- Run Minimal. Which hard questions does it miss, and how fast is it?
- Run Low. How many more does it get right, and what did that cost in time and tokens?
- Run High. Did all that extra thinking buy any accuracy over Low?
- Look at the easy questions across your runs. Did thinking ever help them?