prompt caching
Reusing a provider's stored work on an identical prompt prefix, so repeated calls cost less and start answering sooner.
In more detail
Anthropic charges 1.25 times the input price to write a five-minute cache entry and a tenth of it to read one; OpenAI caches long prefixes automatically. Only an exact prefix matches, so put tools, instructions and reference text first and per-request details such as the time, user and question last.
Related terms
- usage block — The part of a model reply that reports how many tokens the call read and wrote, split by type, which is what you are billed for.
- time to first token — The delay between sending a request and receiving the first token of the reply, the latency a streaming user feels first.
- batch API — An API for sending many requests to be processed asynchronously, usually within 24 hours, at about half the normal price.
Taught in
- Tokens And The Context Window — AI Engineering: LLM Apps, RAG & Agents
- When The Call Fails — AI Engineering: LLM Apps, RAG & Agents
- Cost, Latency And Caching — AI Engineering: LLM Apps, RAG & Agents
- Anatomy Of A Prompt — AI Engineering: LLM Apps, RAG & Agents
- Zero-Shot And Few-Shot — AI Engineering: LLM Apps, RAG & Agents
- Memory And Compaction — AI Engineering: LLM Apps, RAG & Agents
- How RAG Works And When To Use It — AI Engineering: LLM Apps, RAG & Agents
- What MCP Is — AI Engineering: LLM Apps, RAG & Agents
- Build An MCP Server — AI Engineering: LLM Apps, RAG & Agents
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents