How an LLM Answers a Prompt

Live

Initech is adding an AI assistant to its CRM. Your call: which model settings do you fix before launch, and what will users run into if you get them wrong?

Key idea: An LLM predicts one token at a time inside a fixed context window.

Things to try

  1. Send the tagline prompt at temperature 0 three times, then at 1. Compare how much the answers vary.
  2. Set the temperature back to 0.7 and Max output tokens to 40, then ask for the list of 10 fruits. Watch it get cut off.
  3. Set Max output tokens back to 200, and set the context window to Tiny before your first message. Send the tagline prompt, then the token prompt, then ask what you asked first. Then start a new chat on Large and do the same.
  4. Type in the box and watch the token chips update. Try emoji, code, or a non-English word.
Try a prompt
Enter to send · Shift+Enter for a new line

Change how it works, then send again

0.7
probabilitiesrepeatPromptyour textTokenizertext → token idsContext windowwhat the model seesChat historyearlier turnsSystem promptinstructionsModelscores next tokensministral-8bSamplerpicks one tokenOutputstreamed reply
Send a message to watch it run

Tokens · last prompt

Start typing to see how text is split into tokens.

Approximate (OpenAI cl100k tokenizer). Each model family splits text a little differently.

Behind the scenes

Send a message and every step the system takes will show up here.