tokenisation

In search, splitting text into the separate words or pieces that queries are matched against.

In more detail

Whether design-review is one token or two, and whether case and punctuation count, changes what a search finds, so the rules belong in the contract. Search engines also normalise tokens, for example by lower-casing them or reducing words to their stems.

Example

A tokenizer decides where one token ends: at spaces, dots, slashes or other punctuation.

Related terms

Taught in

References