judgement pooling
Building relevance labels by taking the top results from several retrieval systems and judging only that combined pool.
In more detail
Pooling keeps labelling affordable while giving every system's best candidates a fair chance to be marked relevant. Passages no system found remain unjudged, so a new retriever can be underrated.
Related terms
- labelled query set — A set of real questions, each with the passages judged relevant, often on a graded scale, used to score retrieval.
Taught in
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents