top k sampling
A sampling rule that keeps only the k most likely next tokens and samples among them.
In more detail
Introduced for story generation by Fan, Lewis and Dauphin in 2018, top k cuts off the unlikely tail with a fixed count, which can be too many candidates when the model is sure and too few when it is not. Top p adapts the cut-off instead.
Related terms
- top p sampling — A sampling rule that keeps the smallest set of likeliest tokens whose probabilities add up to p, and samples among them.
- sampling (decoding) — Choosing each next token at random from the model's probability distribution, shaped by settings such as temperature, top k and top p.
Taught in
- How The Next Token Is Chosen — AI Engineering: LLM Apps, RAG & Agents
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents
- How RAG Works And When To Use It — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
References
- Fan, Lewis and Dauphin (2018): Hierarchical Neural Story Generation (top-k sampling)
- Anthropic: Messages API reference (temperature, top_k, top_p, stop_sequences, stop_reason)