inverse document frequency
A weight that makes rare words count more than common ones in keyword search, based on how few documents contain each word.
In more detail
A word in every document, such as "the", tells you nothing and gets almost no weight; a product code in two documents gets a lot. BM25 multiplies each matched word's term frequency score by its IDF.
Related terms
- BM25 — The standard keyword ranking function, which scores a document on the query words it contains, weighted by rarity, saturated for repeats and normalised for length.
- stop word — A very common word, such as "the" or "of", that keyword search drops because it says little about relevance.
Taught in
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
References
- Elasticsearch reference: similarity settings (BM25 k1 and b defaults)
- Manning, Raghavan and Schütze: Introduction to Information Retrieval, Evaluation of ranked retrieval results