AI Evaluation Doesn't End at Deployment
In my previous article, I discussed AI evaluation as a new stage in CI/CD—a quality gate that can help determine whether an AI system is ready to move from staging into production. But passing that gate only tells us that the system behaved acceptably at the time it was evaluated. Once deployed, the environment changes, user behavior changes, data changes, and AI systems can behave differently over time. For production AI applications, evaluation therefore shouldn't be a one-time release activity. It should become part of a continuous feedback loop.
A practical architecture can treat evaluation as two connected stages. Pre-deployment evaluation runs against a staging version and checks metrics such as retrieval quality, grounding, answer quality, tool-call behavior, latency, and regression against a validated baseline. If the results meet the required thresholds, the release can proceed. Continuous evaluation then operates after deployment, using appropriately sampled production interactions, curated evaluation datasets, or both to identify changes in system behavior. The purpose is different: the first stage asks "Is this version safe to release?" while the second asks "Is the system still behaving as expected?"
This is where observability and evaluation complement each other. Observability provides the evidence about what happened inside the system—retrieval steps, model calls, tool usage, latency, errors, and traces. Evaluation adds another layer of interpretation by determining whether that behavior meets the expected quality criteria. For example, a production Agentic RAG system might show stable latency and successful requests while retrieval quality or grounding gradually deteriorates. Observability can expose the change; continuous evaluation can help identify it as a quality regression.
The results of continuous evaluation can then become another input into the engineering lifecycle. A detected regression might trigger an investigation, create a new evaluation case, lead to a prompt or retrieval change, or ultimately result in a new code change. That change moves through the same CI/CD process again, including pre-deployment evaluation before reaching production. This changes the traditional CI/CD model from a linear pipeline into a continuous learning loop: Code → CI → AI Evaluation → Deploy → Observability → Continuous Evaluation → Code.
As AI systems become more dynamic and increasingly agentic, I believe this feedback loop will become an important part of production engineering. Traditional monitoring tells us whether the system is available and what happened during execution. Evaluation helps us understand whether the system's behavior remains acceptable. Together, they create a mechanism not only to release AI systems with greater confidence, but also to continuously learn from their behavior in production. The goal isn't simply to evaluate an AI system before deployment. The goal is to keep evaluating whether it remains worthy of trust after deployment.