time to first token

The delay between sending a request and receiving the first token of the reply, the latency a streaming user feels first.

In more detail

Measure it separately from output speed in tokens per second: prompt caching and shorter prompts improve the first, while model size and load govern the second. OpenTelemetry's GenAI metrics define a time-to-first-chunk histogram for it.

Related terms

Taught in

References

Certification courses