judge bias

Systematic errors of model graders, such as favouring the first answer shown, longer answers or their own model's style.

In more detail

Zheng and colleagues documented position, verbosity and self-enhancement bias. Swap the order of pairs, cap length, use a different model family as judge and check against human labels.

Related terms

Taught in

References

Certification courses