record and replay
Testing code that calls a model by recording real replies once and serving them from a file afterwards, matched by request.
In more detail
A replay makes tests fast, free and deterministic, and a strict replay fails when your code sends a request that was never recorded, printing the difference. Re-record deliberately when the prompt or model changes, and keep live evals for real behaviour.
Related terms
- AI eval — A repeatable test of an AI feature: a set of inputs, a way to grade each output, and a score you track across changes.
- deterministic — Always producing the same result from the same inputs, with nothing left to chance or timing.
- test fixture — Fixed, known data or setup that tests use so their results are predictable, such as a sample file, a seeded database or a started application.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- The Request And The Response — AI Engineering: LLM Apps, RAG & Agents
- Streaming Responses — AI Engineering: LLM Apps, RAG & Agents
- When The Call Fails — AI Engineering: LLM Apps, RAG & Agents
- Anatomy Of A Prompt — AI Engineering: LLM Apps, RAG & Agents
- Zero-Shot And Few-Shot — AI Engineering: LLM Apps, RAG & Agents
- Reasoning Techniques — AI Engineering: LLM Apps, RAG & Agents
- Structured Output — AI Engineering: LLM Apps, RAG & Agents
- Prompts As Code — AI Engineering: LLM Apps, RAG & Agents
- Memory And Compaction — AI Engineering: LLM Apps, RAG & Agents
- When Context Fails — AI Engineering: LLM Apps, RAG & Agents
- How RAG Works And When To Use It — AI Engineering: LLM Apps, RAG & Agents
- Retrieval That Finds The Right Thing — AI Engineering: LLM Apps, RAG & Agents
- Grounded Answers With Citations — AI Engineering: LLM Apps, RAG & Agents
- Function Calling — AI Engineering: LLM Apps, RAG & Agents
- Workflows Before Agents — AI Engineering: LLM Apps, RAG & Agents
- Eval-Driven Development — AI Engineering: LLM Apps, RAG & Agents
- Model-Graded Evals — AI Engineering: LLM Apps, RAG & Agents
- Regression Tests In CI — AI Engineering: LLM Apps, RAG & Agents
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents