Initech's contract assistant answers questions about a long customer agreement, and every question resends the whole document. Your call: how do you keep the bill down as usage grows?
Key idea: You pay for every token in and out. Caching makes a repeated prompt start much cheaper, if you build prompts for it.
Things to try
- Ask the uptime question, then the refund question, and check the Prefix cache step on each request.
- Start a new chat, turn off Cache-friendly prompt and ask the uptime question again. A timestamp at the top breaks the cache every time.
- Turn Cache-friendly prompt back on, turn off the document, and ask the uptime question in a new chat. The prompt drops below about 1,024 tokens, OpenAI's minimum for caching, so it can't be cached at all.
- Turn the document back on, switch the answer length to Detailed, and ask in a new chat. On Small, output tokens cost 5 times as much as input, so watch the Answer slice grow.
- Set the answer length back to Short, then compare Small and Large models on the uptime or refund question, a new chat for each, and check the cost per 1,000 requests.
- Set the model back to Small. Set Chat history to All, then Last 2, then None, starting a new chat each time. Ask the three suggested questions in order, then the uptime question again.
Try a prompt