byte pair encoding
A tokenizer training method that starts from single bytes or characters and repeatedly merges the most frequent adjacent pair into a new token.
In more detail
Byte pair encoding was adapted for neural translation by Sennrich, Haddow and Birch in 2016, and most language model tokenizers use it or a close relative. The merge list learned from the training text decides which strings become single tokens, so frequent words cost one token and rare ones several.
Related terms
- tokenizer — The component that splits text into a model's tokens from a fixed vocabulary learned before training, and turns tokens back into text.
- LLM token — The unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence from the tokenizer's vocabulary.
Taught in
- Tokens And The Context Window — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents
- Working With AI Coding Assistants — AI Engineering: LLM Apps, RAG & Agents
References
- Sennrich, Haddow and Birch (2016): Neural Machine Translation of Rare Words with Subword Units
- OpenAI Help Center: Understanding and counting tokens