tokenisation
In search, splitting text into the separate words or pieces that queries are matched against.
In more detail
Whether design-review is one token or two, and whether case and punctuation count, changes what a search finds, so the rules belong in the contract. Search engines also normalise tokens, for example by lower-casing them or reducing words to their stems.
Example
A tokenizer decides where one token ends: at spaces, dots, slashes or other punctuation.Related terms
- text search — Finding items whose text matches a query, such as titles containing design, rather than fields equal to an exact value.
- full-text search — Searching for words anywhere in stored text, using an index of every token rather than named fields.
- index — A separate, sorted structure a database keeps alongside a table so it can find matching rows without reading them all.
Taught in
- Filtering And Searching — Modern API Architecture: REST, GraphQL & gRPC
- SIEM Architecture: Ingestion Pipelines & Indexing — Security Operations & Threat Hunting (SOC)