BM25
The standard keyword ranking function, which scores a document on the query words it contains, weighted by rarity, saturated for repeats and normalised for length.
In more detail
Its two settings are k1, how quickly repeats stop adding score (1.2 by default in Elasticsearch), and b, how strongly long documents are penalised (0.75). It remains a strong baseline that semantic search often fails to beat on names and codes.
Example
score = Σ IDF(t) · tf·(k1+1) / (tf + k1·(1 − b + b·len/avglen))Related terms
- inverse document frequency — A weight that makes rare words count more than common ones in keyword search, based on how few documents contain each word.
- keyword search — Searching by the words a query shares with each document, rather than by meaning.
- hybrid search — Combining keyword and semantic search results, so exact names and paraphrases are both found.
Taught in
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents
- Chunking And Ingestion — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents
References
- Elasticsearch reference: similarity settings (BM25 k1 and b defaults)
- Anthropic: Introducing Contextual Retrieval