inference
Running a trained model to produce output for new input, as opposed to training it.
In more detail
Every API call to a model is an inference request, billed by the tokens it reads and writes. Inference speed shows up as time to first token and output tokens per second, and it depends on the model size, the hardware and the provider's load.
Related terms
- model training — The process that sets a model's weights by adjusting them, over many examples, until its predictions fit the data.
- time to first token — The delay between sending a request and receiving the first token of the reply, the latency a streaming user feels first.
Taught in
- Choosing A Model — AI Engineering: LLM Apps, RAG & Agents