softmax

The function that turns a list of scores into probabilities by exponentiating each one and dividing by the total.

In more detail

Applied to a model's logits, softmax gives the probability of every possible next token. Large scores dominate after exponentiation, which is why dividing the logits by a temperature first changes how peaked the distribution is.

Related terms

Taught in

References

Certification courses