eval gate
An eval suite run in CI that fails the build when scores fall below thresholds, so a regression cannot ship unnoticed.
In more detail
A gate is only useful if a red build means a real regression. Keep pull request builds deterministic with recorded replies, re-record deliberately when the prompt or model changes, and run live evals on a schedule.
Related terms
- trials — Running each eval case several times to measure how much scores vary between runs of the same system.
- continuous integration — Automatically building and testing every change as soon as it is pushed, so problems are caught early.
- record and replay — Testing code that calls a model by recording real replies once and serving them from a file afterwards, matched by request.
Taught in
- Measuring Retrieval — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents