LLM token

The unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence from the tokenizer's vocabulary.

In more detail

Context windows, rate limits and prices are all measured in tokens, not words or characters. English averages roughly four characters a token, but code, numbers and many other languages take more. Count with the provider's own tokenizer or token-counting endpoint before relying on a figure, because each model family splits text differently.

Related terms

Taught in

References

Certification courses