byte pair encoding

A tokenizer training method that starts from single bytes or characters and repeatedly merges the most frequent adjacent pair into a new token.

In more detail

Byte pair encoding was adapted for neural translation by Sennrich, Haddow and Birch in 2016, and most language model tokenizers use it or a close relative. The merge list learned from the training text decides which strings become single tokens, so frequent words cost one token and rare ones several.

Related terms

Taught in

References

Certification courses