sampling (decoding)
Choosing each next token at random from the model's probability distribution, shaped by settings such as temperature, top k and top p.
In more detail
Sampling is why the same prompt can give different replies. Lower temperature and tighter cut-offs make output more repeatable but never fully deterministic on hosted models, so tests should check properties of a reply rather than its exact text.
Related terms
- temperature — The sampling setting that divides the logits before softmax: values below one sharpen the distribution towards the likeliest token, values above one flatten it.
- top k sampling — A sampling rule that keeps only the k most likely next tokens and samples among them.
- top p sampling — A sampling rule that keeps the smallest set of likeliest tokens whose probabilities add up to p, and samples among them.
- greedy decoding — Always choosing the single most likely next token, with no randomness.
Taught in
- How The Next Token Is Chosen — AI Engineering: LLM Apps, RAG & Agents
- The Request And The Response — AI Engineering: LLM Apps, RAG & Agents
- Zero-Shot And Few-Shot — AI Engineering: LLM Apps, RAG & Agents
- Reasoning Techniques — AI Engineering: LLM Apps, RAG & Agents
- Structured Output — AI Engineering: LLM Apps, RAG & Agents
- What MCP Is — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents
References
- Holtzman et al. (2019): The Curious Case of Neural Text Degeneration (nucleus sampling)
- Anthropic: Messages API reference (temperature, top_k, top_p, stop_sequences, stop_reason)