production monitoring

Watching a live AI feature's latency, cost, error rate and user feedback, and alerting when they move.

In more detail

Evals catch problems before release; monitoring shows real behaviour after it, at the price of being reactive and noisy. Alert on what customers feel, sample traces to learn why numbers moved, and feed failures back into the golden set.

Related terms

Taught in

References

Certification courses