AI Evaluation as a New Stage in CI/CD
A release can pass every CI/CD check—and still make your AI system worse. Traditional CI/CD pipelines are built to validate whether software works as expected... Unit tests, integration tests, security scans, and deployment checks help ensure that a new release is technically sound. But AI applications introduce a different challenge: a system can pass every traditional software test and still produce inaccurate, ungrounded, or inconsistent results. The code can be correct, the deployment can succeed, and the API can respond—while the AI system still behaves worse than the previous version. That makes to believe that AI evaluation should become a dedicated stage in the CI/CD pipeline, rather than remaining a separate activity performed only during development.
A practical approach is to deploy the new version to a staging environment and then run an AI Evaluation Quality Gate before promoting it to production. For an Agentic RAG application, this suite could evaluate retrieval quality, grounding, answer quality, tool-call behavior, latency, hallucination indicators, and regression against a validated baseline. The results can be compared against predefined thresholds or a previous version. If a change to an embedding model, chunking strategy, prompt, or agent workflow causes a significant regression, the pipeline can flag the release for investigation or prevent it from moving forward. The goal is not to reduce AI quality to a single score, but to create a repeatable process that provides evidence about how a change affects system behavior.
In my own implementation, I use CircleCI to orchestrate the CI/CD workflow. The process begins when a reviewed and approved pull request is merged into the main branch. The pipeline runs unit and integration tests, performs security checks, builds the Docker image, and deploys the application to a staging environment. The AI Evaluation Suite runs immediately after deployment to staging, evaluating the new release before it becomes eligible for production. Depending on the outcome, the release can continue toward production or be stopped for further investigation.
This creates an additional quality gate for AI-enabled software delivery. Traditional CI/CD helps answer "Does the software work?" AI evaluation adds another question: "Does the AI system behave as expected?" For Agentic RAG systems, this distinction is important because seemingly small changes can affect retrieval quality, grounding, tool selection, or response behavior in ways that traditional tests may not detect. Observability helps us understand what happened inside the system, evaluations help us determine whether the behavior was acceptable, and CI/CD provides the mechanism to integrate those checks into the software delivery process.
As AI applications move from prototypes into production, I believe this evaluation layer will become an increasingly important part of the engineering lifecycle. As AI systems become increasingly probabilistic, release confidence depends not only on whether the application works, but also on whether its behavior remains acceptable after every change. Traditional CI/CD validates the software. AI-enabled CI/CD validates the software—and its behavior.