# AI PM Lab > Lessons that build AI product sense, one run at a time. Run real models, step inside them in 3D, push them until they break, then take on the challenges. Each lesson is an interactive page: pick a few settings, run a real model, and watch every step drawn as a diagram. The immersive pages are full-screen 3D tours of one idea, replaying real runs. ## Foundations How models behave: tokens, prompts, cost, and when they make things up. - [How an LLM Answers a Prompt](https://aipmlab.io/lessons/how-llms-work): Follow a prompt through tokenization, the context window, and sampling, and see why temperature and limits matter. Key idea: An LLM predicts one token at a time inside a fixed context window. The decision it informs: Initech is adding an AI assistant to its CRM. Your call: which model settings do you fix before launch, and what will users run into if you get them wrong? - [Writing Good Prompts](https://aipmlab.io/lessons/writing-good-prompts): Build a prompt from role, categories, format, examples, and rules, score it on 10 labeled items, and see what moves the score. Key idea: A prompt is a set of ingredients, and you can measure each one. Spelling out the format, defining your categories, and covering edge cases usually matter far more than telling the model who it is. The decision it informs: Initech wants customer feedback sorted by theme and sentiment automatically. Your call: what has to be in the prompt before the results are good enough to act on? - [Hallucination & Grounding](https://aipmlab.io/lessons/hallucination-and-grounding): Ask 12 questions, some nobody can answer, and count how often the model makes something up. Then see which fixes work. Key idea: A model would rather answer than admit it doesn't know, and a made-up answer sounds as confident as a real one. Give it permission to say "I don't know" and the facts it needs, and check where every answer came from. The decision it informs: Initech's help bot answers customers' questions, and some of them have no answer anywhere. Your call: what should it do when it doesn't know? - [Embeddings & Semantic Search](https://aipmlab.io/lessons/embeddings-and-semantic-search): Search 25 support tickets by keyword and by meaning, and watch your query land on a map where similar tickets sit together. Key idea: An embedding turns text into numbers so that similar meanings land close together. That's how search finds the right ticket even when no words match. The decision it informs: Initech's support team wants to find past tickets like a new one, however the customer words it. Your call: search by matching words, by meaning, or both? - [Prompting vs. RAG vs. Fine-tuning](https://aipmlab.io/lessons/prompting-vs-rag-vs-fine-tuning): Run 12 support jobs with a better prompt, with retrieval, and with a fine-tuning stand-in. Each fixes a different kind of problem. Key idea: A better prompt fixes rules you can write down, RAG fixes facts the model can't know, and fine-tuning teaches patterns from examples but bakes in old facts. Most real products combine them. The decision it informs: Initech's support bot quotes wrong prices, ignores the house reply style, and sends tickets to the wrong squad. Your call: a better prompt, retrieval (RAG) or fine-tuning, and in what order? - [Prompt Caching & Cost](https://aipmlab.io/lessons/prompt-caching-and-cost): Chat about a long contract and watch where every token and dollar goes, and how caching makes repeat requests cheaper. Key idea: You pay for every token in and out. Caching makes a repeated prompt start much cheaper, if you build prompts for it. The decision it informs: Initech's contract assistant answers questions about a long customer agreement, and every question resends the whole document. Your call: how do you keep the bill down as usage grows? ## Building Putting models to work in a product: tools, retrieval, agents. - [Structured Outputs](https://aipmlab.io/lessons/structured-outputs): Turn 12 messy sales emails into orders. Ask for JSON four ways and see which replies your code can read, and which are right. Key idea: A model can return the right information in a shape your code can't read. Enforce a schema so every reply parses, and describe the tricky fields so the values come back right. The decision it informs: Orders arrive at Initech as sales emails, written every which way. Your call: how do you get them into the billing system without someone retyping each one? - [How an AI Agent Works](https://aipmlab.io/lessons/how-ai-agents-work): Watch the agent loop: the model decides to call tools, reads their results, and keeps going until it can answer. Key idea: Agents = an LLM in a loop with tools, memory, and guardrails. The decision it informs: Initech wants an assistant that can look things up and do math for customers. Your call: which tools does it get, and what stops it from looping forever or doing harm? - [Retrieval-Augmented Generation](https://aipmlab.io/lessons/retrieval-augmented-generation): Ask questions about a fictional product the model has never seen. Turn retrieval on and off to see grounding vs. guessing. Key idea: RAG = search your own data first, then let the model answer from it. The decision it informs: Customers ask Initech's help bot about plans and policies the model was never trained on. Your call: how does it answer from Initech's own documents instead of guessing? - [Tool Design & MCP](https://aipmlab.io/lessons/tool-design-and-mcp): Give a model 12 requests and a set of tools. Blur the names or add look-alikes from other MCP servers, and watch what it picks. Key idea: A model chooses a tool only from its name and description, and every tool you connect is sent with every request. Clear descriptions get the right tool called; extra tools cost tokens even when they don't help. The decision it informs: Initech's support assistant calls tools, and the team wants to plug in three more MCP servers. Your call: what has to be true about those tools before you ship? - [Multi-Agent Systems](https://aipmlab.io/lessons/multi-agent-systems): A lead agent splits a product brief across specialists with their own data. Compare it to one agent on quality, time, and cost. Key idea: Multi-agent means splitting work across specialized model calls. It can help when each agent brings its own data, but every handoff costs time and money. The decision it informs: Initech wants AI to draft product briefs from customer interviews, analytics and platform limits. Your call: one agent or a team of specialists, and is the difference worth what it costs? - [Workflows vs. Agents](https://aipmlab.io/lessons/workflows-vs-agents): Run 10 support jobs through fixed steps, an agent, and both. See which handles surprises, which stays reliable, and what each costs. Key idea: A workflow runs fixed steps: cheap and predictable, but it quietly mishandles jobs it wasn't built for. An agent adapts, but costs more and can wander. Most products route routine work to workflows and the rest to an agent. The decision it informs: Initech wants to automate support jobs like refunds, 2FA resets and plan changes. Your call: fixed steps, a free agent, or both? ## Shipping Making it measurable, safe and affordable. - [Evaluating AI Changes](https://aipmlab.io/lessons/ai-evals): Run a prompt against a labeled test set and let graders score it. Change the prompt or model and see if the score goes up. Key idea: An eval is a fixed test set plus graders, giving a score you can compare across changes. The decision it informs: You're about to change the prompt behind Initech's ticket triage. Your call: how will you know the new version is better, and that it didn't break something else? - [Reasoning Models](https://aipmlab.io/lessons/reasoning-models): Run 12 questions at four thinking levels. See where a little thinking fixes multi-step problems, and where more just costs. Key idea: Reasoning models think before they answer. A little thinking fixes multi-step problems; more costs time and money without helping, and easy questions never needed it. The decision it informs: Initech's assistant answers billing questions, some of them multi-step math. A reasoning model can think before it answers, but it costs more and makes customers wait. Your call: how much thinking is worth paying for? - [Production Monitoring](https://aipmlab.io/lessons/production-monitoring): A ticket router aces its test set, then meets a real week of messages. Find what the tests missed, add it, and fix it. Key idea: An offline test set only checks what you thought of. Watch real traffic, add what fails to the test set, and fix it so both scores agree. The decision it informs: Initech's ticket router passed all 8 of its tests and shipped. Your call: how will you know it's working on real customers' messages? - [Prompt Injection & Guardrails](https://aipmlab.io/lessons/prompt-injection): An email assistant reads a poisoned newsletter and leaks internal numbers. Turn on defenses one at a time and see which ones stop it. Key idea: Prompt injection hides instructions inside content the model reads. Check what comes back from tools, not just what the user types, and limit what the agent is allowed to do. The decision it informs: Initech's email assistant reads the inbox and can send email on the user's behalf. Your call: which defenses do you ship, knowing each one costs something? - [Model Routing](https://aipmlab.io/lessons/model-routing): Run 12 support questions through a small model, a large one, a router, and a cascade, and find the sweet spot on accuracy vs. cost. Key idea: Most requests don't need your biggest model. Route easy ones to a cheap model and hard ones to a strong one, and you keep most of the quality for a fraction of the cost. The catch: hard requests that look easy. The decision it informs: You run Initech's support bot. The large model gets every question right but costs about 20 times as much as the small one. Your call: which requests can go to the small model without customers noticing? - [Human-in-the-Loop](https://aipmlab.io/lessons/human-in-the-loop): A support agent takes real actions on 12 requests. Choose which wait for a person, and trade bad actions against review time. Key idea: Letting an agent act on its own is fast until it refunds the wrong customer. Asking a person every time is safe but slow. Route only risky actions (money, anything irreversible) to a person, and keep the policy in the agent's prompt so less reaches them at all. The decision it informs: Initech's support agent can refund, change plans, delete accounts and email customers. Your call: which actions wait for a person, and how much review time can the team afford? ## Immersive 3D pages - [Inside an Embedding Model](https://aipmlab.io/explore/embeddings): Watch a search become tokens, then 1,024 numbers, then a point in meaning space, and find its nearest neighbors. - [Inside Next-Token Prediction](https://aipmlab.io/explore/next-token): Watch a model write: every candidate for the next token, the roll of the dice, and what temperature does to the odds. - [Inside an Agent Loop](https://aipmlab.io/explore/agent-loop): Ride along as an agent re-reads everything, calls tools, and goes around again, and see what each lap costs. - [Inside RAG](https://aipmlab.io/explore/rag): Watch a question find its nearest docs, see them pasted into the prompt, and trace each citation back to its source. - [Inside an Eval](https://aipmlab.io/explore/evals): Watch 8 tickets graded three ways, read a failure, and see a bigger model score worse. - [Inside a Prompt Injection](https://aipmlab.io/explore/injection): Watch a hidden instruction in an email hijack an inbox agent, then try the defenses and see which ones stop it. ## Today's Call - [Today's Call](https://aipmlab.io/challenges/today): one real model run a day, the same for everyone, from the challenges' recorded rounds; call what the model did, say how sure you are, then see what really happened and how other PMs called it. A new round at midnight US Eastern. ## Blind Spots - [Blind Spots](https://aipmlab.io/challenges/blind-spots): the real AI runs product managers most often call wrong in the site's challenges, from anonymous answer counts, by decision area, with how players answered and what the run really did. ## Glossary - [AI terms for product managers](https://aipmlab.io/glossary): 76 concepts in plain words (Tokens, Context Window, Temperature, Max Tokens, Prompt Design, Few-Shot Examples, Output Format, Ablation, Hallucination, Grounding, Abstaining, Embeddings, Semantic Search, Keyword Search, Similarity, Prompting, RAG, Fine-tuning, Trade-offs, Pricing, Prompt Caching, Unit Economics, Structured Output, JSON Schema, Parsing, Reliability, Tool Use, Agent Loop, Guardrails, Memory, Vector Search, Top-K, Reranking, MCP, Tool Descriptions, Token Cost, Orchestration, Specialists, Handoffs, Workflows, Agents, Cost, Test Sets, LLM-as-Judge, Pass Rate, Regressions, Reasoning, Thinking Effort, Latency, Monitoring, Live Traffic, Prompt Injection, Untrusted Data, Least Privilege, Model Choice, Routing, Cascades, Approvals, Autonomy, Risk Policy, LLM, Inference, System Prompt, Tokenizer, Sampling, Streaming, Context Engineering, Chunking, Evals, Benchmark, Frontier Model, Small Model, Open-Weight Model, Jailbreak, Red Teaming, Human Review), each linked to the lesson that teaches it. ## MCP server - [Use AI PM Lab in your assistant](https://aipmlab.io/mcp): a free remote MCP server at https://aipmlab.io/api/mcp (Streamable HTTP, no authentication) with tools to search lessons, define concepts, count tokens, estimate model costs, replay recorded experiments, and guides to review AI feature specs, prepare planning meetings and design eval suites.