AI eval
A repeatable test of an AI feature: a set of inputs, a way to grade each output, and a score you track across changes.
In more detail
Evals replace "it looked fine" with numbers. Grade with code where you can, with a model judge where you must, and compare versions case by case, since a higher average can hide a regression. Without a way to judge answers there is no eval, and the feature cannot be shown to work.
Related terms
- golden set — A curated set of inputs with known good outcomes, used to check that an AI feature still behaves after every change.
- eval harness — The code that runs every eval case through the system, applies the checks and reports scores per case and overall.
- LLM-as-judge — Using a language model to grade another model's output against a rubric, for qualities that code cannot check.
- eval-driven development — Deciding what a good output looks like and writing evals for it before changing an AI feature, then running them on every change.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- Choosing A Model — AI Engineering: LLM Apps, RAG & Agents
- When The Call Fails — AI Engineering: LLM Apps, RAG & Agents
- Cost, Latency And Caching — AI Engineering: LLM Apps, RAG & Agents
- Zero-Shot And Few-Shot — AI Engineering: LLM Apps, RAG & Agents
- Reasoning Techniques — AI Engineering: LLM Apps, RAG & Agents
- Structured Output — AI Engineering: LLM Apps, RAG & Agents
- Prompts As Code — AI Engineering: LLM Apps, RAG & Agents
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents
- Grounded Answers With Citations — AI Engineering: LLM Apps, RAG & Agents
- Workflows Before Agents — AI Engineering: LLM Apps, RAG & Agents
- Excessive Agency — AI Engineering: LLM Apps, RAG & Agents
- Guardrails, Privacy And Governance — AI Engineering: LLM Apps, RAG & Agents
- Eval-Driven Development — AI Engineering: LLM Apps, RAG & Agents
- Evaluating RAG And Agents — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents
- Images, Audio And Speech — AI Engineering: LLM Apps, RAG & Agents
- Working With AI Coding Assistants — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents