LLM-as-judge

Using a language model to grade another model's output against a rubric, for qualities that code cannot check.

In more detail

Zheng and colleagues showed in 2023 that strong judges agree with people about as often as people agree with each other, but have biases. Give the judge a specific rubric, ask it to reason before ruling and allow "unknown", and measure it against human labels with kappa before trusting it.

Related terms

Taught in

References

Certification courses