trials
Running each eval case several times to measure how much scores vary between runs of the same system.
In more detail
Model output varies, so one run cannot tell a regression from noise. Measure the current release over several trials, set each threshold below the baseline by the noise you saw, and require critical cases to pass every trial.
Related terms
- eval gate — An eval suite run in CI that fails the build when scores fall below thresholds, so a regression cannot ship unnoticed.
- flaky test — A test that sometimes passes and sometimes fails on the same code, because it depends on something it does not control.
- threshold — A chosen limit that a value is compared against to decide whether something should happen, such as an alert firing.
Taught in
- Prompts As Code — AI Engineering: LLM Apps, RAG & Agents
- When Context Fails — AI Engineering: LLM Apps, RAG & Agents
- Eval-Driven Development — AI Engineering: LLM Apps, RAG & Agents
- Evaluating RAG And Agents — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents