batch API
An API for sending many requests to be processed asynchronously, usually within 24 hours, at about half the normal price.
In more detail
Batching suits work with no user waiting, such as nightly classification, evals or backfills. Anthropic's Message Batches and OpenAI's Batch API both discount by half, and the discount stacks with prompt caching.
Related terms
- prompt caching — Reusing a provider's stored work on an identical prompt prefix, so repeated calls cost less and start answering sooner.
- usage block — The part of a model reply that reports how many tokens the call read and wrote, split by type, which is what you are billed for.
Taught in
- Cost, Latency And Caching — AI Engineering: LLM Apps, RAG & Agents
- Agent SDKs And Human Approval — AI Engineering: LLM Apps, RAG & Agents
- Excessive Agency — AI Engineering: LLM Apps, RAG & Agents