Prompt Caching & Cost

Live

Initech's contract assistant answers questions about a long customer agreement, and every question resends the whole document. Your call: how do you keep the bill down as usage grows?

Key idea: You pay for every token in and out. Caching makes a repeated prompt start much cheaper, if you build prompts for it.

Things to try

  1. Ask the uptime question, then the refund question, and check the Prefix cache step on each request.
  2. Start a new chat, turn off Cache-friendly prompt and ask the uptime question again. A timestamp at the top breaks the cache every time.
  3. Turn Cache-friendly prompt back on, turn off the document, and ask the uptime question in a new chat. The prompt drops below about 1,024 tokens, OpenAI's minimum for caching, so it can't be cached at all.
  4. Turn the document back on, switch the answer length to Detailed, and ask in a new chat. On Small, output tokens cost 5 times as much as input, so watch the Answer slice grow.
  5. Set the answer length back to Short, then compare Small and Large models on the uptime or refund question, a new chat for each, and check the cost per 1,000 requests.
  6. Set the model back to Small. Set Chat history to All, then Last 2, then None, starting a new chat each time. Ask the three suggested questions in order, then the uptime question again.
Try a prompt
Enter to send · Shift+Enter for a new line

Change how it works, then send again

usageSystem promptinstructionsDocumentcustomer agreementChat historyearlier turnsQuestionyour messagePromptall input tokensPrefix cacheseen this start?Modelreads, then writesgpt-6-lunaAnsweroutput tokensBilltokens × price
Send a message to watch it run

Cost

Ask a question to see where every token and dollar of the request goes.

Behind the scenes

Send a message and every step the system takes will show up here.