hybrid search
Combining keyword and semantic search results, so exact names and paraphrases are both found.
In more detail
Run both searches, merge the two rankings, commonly with reciprocal rank fusion, and optionally rerank the merged shortlist. Anthropic's contextual retrieval results showed the combination beating either alone.
Related terms
- BM25 — The standard keyword ranking function, which scores a document on the query words it contains, weighted by rarity, saturated for repeats and normalised for length.
- semantic search — Searching by meaning: embedding the query and returning the stored items whose vectors are closest.
- reciprocal rank fusion — A way to merge several rankings by summing 1 divided by (k plus the rank) for each document, using ranks alone and ignoring raw scores.
- reranker — A model that reorders a shortlist of retrieved passages by reading each one together with the query.
Taught in
- From Prompts To Context — AI Engineering: LLM Apps, RAG & Agents
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents
- How RAG Works And When To Use It — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents
References
- Anthropic: Introducing Contextual Retrieval
- Cormack, Clarke and Büttcher: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009)