eval harness
The code that runs every eval case through the system, applies the checks and reports scores per case and overall.
In more detail
Checks are plain functions that return pass or fail with a reason, such as "cites a source" or "uses the refusal sentence". Prefer them to model grading wherever a rule can decide, because they are fast, free and repeatable.
Related terms
- AI eval — A repeatable test of an AI feature: a set of inputs, a way to grade each output, and a score you track across changes.
- golden set — A curated set of inputs with known good outcomes, used to check that an AI feature still behaves after every change.
- LLM-as-judge — Using a language model to grade another model's output against a rubric, for qualities that code cannot check.
Taught in
- Eval-Driven Development — AI Engineering: LLM Apps, RAG & Agents
- Model-Graded Evals — AI Engineering: LLM Apps, RAG & Agents