sampling (decoding)

Choosing each next token at random from the model's probability distribution, shaped by settings such as temperature, top k and top p.

In more detail

Sampling is why the same prompt can give different replies. Lower temperature and tighter cut-offs make output more repeatable but never fully deterministic on hosted models, so tests should check properties of a reply rather than its exact text.

Related terms

Taught in

References

Certification courses