inference

Running a trained model to produce output for new input, as opposed to training it.

In more detail

Every API call to a model is an inference request, billed by the tokens it reads and writes. Inference speed shows up as time to first token and output tokens per second, and it depends on the model size, the hardware and the provider's load.

Related terms

Taught in

References

Certification courses