time to first token
The delay between sending a request and receiving the first token of the reply, the latency a streaming user feels first.
In more detail
Measure it separately from output speed in tokens per second: prompt caching and shorter prompts improve the first, while model size and load govern the second. OpenTelemetry's GenAI metrics define a time-to-first-chunk histogram for it.
Related terms
- streaming — Sending a response in pieces as they become available instead of building the whole thing first.
- latency — The time between asking for something and getting the answer back.
- prompt caching — Reusing a provider's stored work on an identical prompt prefix, so repeated calls cost less and start answering sooner.
Taught in
- How The Next Token Is Chosen — AI Engineering: LLM Apps, RAG & Agents
- When The Call Fails — AI Engineering: LLM Apps, RAG & Agents
- Cost, Latency And Caching — AI Engineering: LLM Apps, RAG & Agents
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents