top p sampling

A sampling rule that keeps the smallest set of likeliest tokens whose probabilities add up to p, and samples among them.

In more detail

Holtzman and colleagues called it nucleus sampling in 2019: when the model is confident the set is small, and when it is unsure the set grows. A top p of 0.9 drops the long tail of unlikely tokens that cause incoherent text.

Related terms

Taught in

References

Certification courses