labelled query set
A set of real questions, each with the passages judged relevant, often on a graded scale, used to score retrieval.
In more detail
Build it by pooling the top results of several systems and having people grade them. It turns opinions about retrieval into numbers; keep it growing, and distrust small differences while it is small.
Related terms
- recall at k — The share of the relevant passages for a query that appear in the top k results.
- mean reciprocal rank — The average, over queries, of 1 divided by the rank of the first relevant result, so 1 means the answer always came first.
- normalised discounted cumulative gain — A ranking score that uses graded relevance, discounts results lower down the list, and divides by the best possible score so 1 is a perfect ranking.
- golden set — A curated set of inputs with known good outcomes, used to check that an AI feature still behaves after every change.
Taught in
- Embedding Models — AI Engineering: LLM Apps, RAG & Agents
- Keyword And Hybrid Search — AI Engineering: LLM Apps, RAG & Agents
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents
References
- Manning, Raghavan and Schütze: Introduction to Information Retrieval, Evaluation of ranked retrieval results
- Järvelin and Kekäläinen: Cumulated gain-based evaluation of IR techniques (ACM TOIS, 2002)