AI PM Lab

Challenges

Played on recorded real runs, so everyone gets the same results. Beat the goal, then share your best.

The capstone
Ship It

Launch a support bot that clears every bar, for the lowest cost.

Start the launch

Read the models

Call what a real model did, and catch it when it fails. Ten rounds each.

Predict the outcome
Call It

Predict what real models did in ten recorded runs, then see if you called it.

Review the output
Spot the Failure

Review ten real model outputs: ship it, or flag what went wrong.

Compare the setups
Which Wins?

Two real setups, one setting apart. Pick the one that did better, ten times.

Find the best setup

Change a lesson's settings until you hit the goal at the best score.

Hallucination & Grounding
Nothing Made Up

Get the model to make up nothing, while handling as many of the 12 questions well as you can.

Human-in-the-Loop
Safe Without the Busywork

Let no harmful action through, with as little human review time as you can.

Tool Design & MCP
Lean Tool Kit

Get all 12 tool picks right while sending the fewest tokens per request.

Reasoning Models
Just Enough Thinking

Get all 8 hard questions right at the lowest cost.

Model Routing
The Cheapest Perfect Router

Get all 12 requests right at the lowest cost per 1,000 requests.

Evaluating AI Changes
Pass the Full Eval

Pass as many of the 8 tickets as you can with all three graders on.