tokenizer

The component that splits text into a model's tokens from a fixed vocabulary learned before training, and turns tokens back into text.

In more detail

Each model family has its own tokenizer, so the same text costs different numbers of tokens on different models. Common words in well-represented languages become single tokens, while rare words, other scripts and code split into many, which makes the same request dearer in some languages.

Related terms

Taught in

References

Certification courses