tokenizer
The component that splits text into a model's tokens from a fixed vocabulary learned before training, and turns tokens back into text.
In more detail
Each model family has its own tokenizer, so the same text costs different numbers of tokens on different models. Common words in well-represented languages become single tokens, while rare words, other scripts and code split into many, which makes the same request dearer in some languages.
Related terms
- LLM token — The unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence from the tokenizer's vocabulary.
- byte pair encoding — A tokenizer training method that starts from single bytes or characters and repeatedly merges the most frequent adjacent pair into a new token.
Taught in
- Tokens And The Context Window — AI Engineering: LLM Apps, RAG & Agents
- How The Next Token Is Chosen — AI Engineering: LLM Apps, RAG & Agents
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents
- Images, Audio And Speech — AI Engineering: LLM Apps, RAG & Agents
References
- OpenAI: Token counting guide
- Petrov et al. (2023): Language Model Tokenizers Introduce Unfairness Between Languages