Cohen's kappa
A measure of agreement between two raters that corrects for the agreement expected by chance, from 1 for perfect to 0 for chance level.
In more detail
Raw agreement flatters a judge that passes nearly everything when most cases pass; kappa does not. McHugh suggests that values below about 0.6 are too weak for important decisions.
Example
kappa = (p_o − p_e) / (1 − p_e)Related terms
- LLM-as-judge — Using a language model to grade another model's output against a rubric, for qualities that code cannot check.
- rubric — The written criteria a grader applies to decide whether an output passes, ideally specific, observable and with a way out for "unknown".
Taught in
- Model-Graded Evals — AI Engineering: LLM Apps, RAG & Agents
References
- McHugh (2012): Interrater reliability: the kappa statistic, Biochemia Medica
- scikit-learn: cohen_kappa_score (definition of kappa)