rubric
The written criteria a grader applies to decide whether an output passes, ideally specific, observable and with a way out for "unknown".
In more detail
Vague rubrics such as "is it helpful?" give noisy grades. Read the disagreements between the judge and people: they show where the rubric is unclear.
Related terms
- LLM-as-judge — Using a language model to grade another model's output against a rubric, for qualities that code cannot check.
- Cohen's kappa — A measure of agreement between two raters that corrects for the agreement expected by chance, from 1 for perfect to 0 for chance level.
Taught in
- Model-Graded Evals — AI Engineering: LLM Apps, RAG & Agents
- Evaluating RAG And Agents — AI Engineering: LLM Apps, RAG & Agents