top k sampling

A sampling rule that keeps only the k most likely next tokens and samples among them.

In more detail

Introduced for story generation by Fan, Lewis and Dauphin in 2018, top k cuts off the unlikely tail with a fixed count, which can be too many candidates when the model is sure and too few when it is not. Top p adapts the cut-off instead.

Related terms

Taught in

References

Certification courses