production monitoring
Watching a live AI feature's latency, cost, error rate and user feedback, and alerting when they move.
In more detail
Evals catch problems before release; monitoring shows real behaviour after it, at the price of being reactive and noisy. Alert on what customers feel, sample traces to learn why numbers moved, and feed failures back into the golden set.
Related terms
- GenAI span — A trace span for one model call, named after the operation and model, such as "chat claude-sonnet", carrying the GenAI convention attributes.
- observability — How well you can understand what a system is doing, and why, from the data it emits: mainly logs, metrics and traces.
- percentile — The value below which a given percentage of measurements fall; p99 latency is the time 99 in 100 requests beat.
- user feedback — Ratings or comments from users on model answers, recorded and linked to the call that produced them.
Taught in
- Tracing And Monitoring — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents