stop word
A very common word, such as "the" or "of", that keyword search drops because it says little about relevance.
In more detail
Removing stop words shrinks the index, but a careless list can drop words that matter, such as "not" in a question about what a product cannot do. BM25's rarity weighting already gives common words little weight, so many systems keep them.
Related terms
- BM25 — The standard keyword ranking function, which scores a document on the query words it contains, weighted by rarity, saturated for repeats and normalised for length.
- inverse document frequency — A weight that makes rare words count more than common ones in keyword search, based on how few documents contain each word.
Taught in
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
References
- Manning, Raghavan and Schütze: Introduction to Information Retrieval, Evaluation of ranked retrieval results
- Elasticsearch reference: similarity settings (BM25 k1 and b defaults)