top p sampling
A sampling rule that keeps the smallest set of likeliest tokens whose probabilities add up to p, and samples among them.
In more detail
Holtzman and colleagues called it nucleus sampling in 2019: when the model is confident the set is small, and when it is unsure the set grows. A top p of 0.9 drops the long tail of unlikely tokens that cause incoherent text.
Related terms
- top k sampling — A sampling rule that keeps only the k most likely next tokens and samples among them.
- temperature — The sampling setting that divides the logits before softmax: values below one sharpen the distribution towards the likeliest token, values above one flatten it.
Taught in
- How The Next Token Is Chosen — AI Engineering: LLM Apps, RAG & Agents
References
- Holtzman et al. (2019): The Curious Case of Neural Text Degeneration (nucleus sampling)
- Anthropic: Messages API reference (temperature, top_k, top_p, stop_sequences, stop_reason)