AI Evaluation Doesn't End at Deployment

In my previous article, I discussed AI evaluation as a new stage in CI/CD—a quality gate that can help determine whether an AI system is ready to move from staging into production. But passing that gate only tells us that the system behaved acceptably at the time it was evaluated. Once deployed, the environment changes, user behavior changes, data changes, and AI systems can behave differently over time. For production AI applications, evaluation therefore shouldn't be a one-time release activity. It should become part of a continuous feedback loop.

A practical architecture can treat evaluation as two connected stages. Pre-deployment evaluation runs against a staging version and checks metrics such as retrieval quality, grounding, answer quality, tool-call behavior, latency, and regression against a validated baseline. If the results meet the required thresholds, the release can proceed. Continuous evaluation then operates after deployment, using appropriately sampled production interactions, curated evaluation datasets, or both to identify changes in system behavior. The purpose is different: the first stage asks "Is this version safe to release?" while the second asks "Is the system still behaving as expected?"

This is where observability and evaluation complement each other. Observability provides the evidence about what happened inside the system—retrieval steps, model calls, tool usage, latency, errors, and traces. Evaluation adds another layer of interpretation by determining whether that behavior meets the expected quality criteria. For example, a production Agentic RAG system might show stable latency and successful requests while retrieval quality or grounding gradually deteriorates. Observability can expose the change; continuous evaluation can help identify it as a quality regression.

The results of continuous evaluation can then become another input into the engineering lifecycle. A detected regression might trigger an investigation, create a new evaluation case, lead to a prompt or retrieval change, or ultimately result in a new code change. That change moves through the same CI/CD process again, including pre-deployment evaluation before reaching production. This changes the traditional CI/CD model from a linear pipeline into a continuous learning loop: Code → CI → AI Evaluation → Deploy → Observability → Continuous Evaluation → Code.

As AI systems become more dynamic and increasingly agentic, I believe this feedback loop will become an important part of production engineering. Traditional monitoring tells us whether the system is available and what happened during execution. Evaluation helps us understand whether the system's behavior remains acceptable. Together, they create a mechanism not only to release AI systems with greater confidence, but also to continuously learn from their behavior in production. The goal isn't simply to evaluate an AI system before deployment. The goal is to keep evaluating whether it remains worthy of trust after deployment.

AI Evaluation as a New Stage in CI/CD

A release can pass every CI/CD check—and still make your AI system worse. Traditional CI/CD pipelines are built to validate whether software works as expected... Unit tests, integration tests, security scans, and deployment checks help ensure that a new release is technically sound. But AI applications introduce a different challenge: a system can pass every traditional software test and still produce inaccurate, ungrounded, or inconsistent results. The code can be correct, the deployment can succeed, and the API can respond—while the AI system still behaves worse than the previous version. That makes to believe that AI evaluation should become a dedicated stage in the CI/CD pipeline, rather than remaining a separate activity performed only during development.

A practical approach is to deploy the new version to a staging environment and then run an AI Evaluation Quality Gate before promoting it to production. For an Agentic RAG application, this suite could evaluate retrieval quality, grounding, answer quality, tool-call behavior, latency, hallucination indicators, and regression against a validated baseline. The results can be compared against predefined thresholds or a previous version. If a change to an embedding model, chunking strategy, prompt, or agent workflow causes a significant regression, the pipeline can flag the release for investigation or prevent it from moving forward. The goal is not to reduce AI quality to a single score, but to create a repeatable process that provides evidence about how a change affects system behavior.

In my own implementation, I use CircleCI to orchestrate the CI/CD workflow. The process begins when a reviewed and approved pull request is merged into the main branch. The pipeline runs unit and integration tests, performs security checks, builds the Docker image, and deploys the application to a staging environment. The AI Evaluation Suite runs immediately after deployment to staging, evaluating the new release before it becomes eligible for production. Depending on the outcome, the release can continue toward production or be stopped for further investigation.

This creates an additional quality gate for AI-enabled software delivery. Traditional CI/CD helps answer "Does the software work?" AI evaluation adds another question: "Does the AI system behave as expected?" For Agentic RAG systems, this distinction is important because seemingly small changes can affect retrieval quality, grounding, tool selection, or response behavior in ways that traditional tests may not detect. Observability helps us understand what happened inside the system, evaluations help us determine whether the behavior was acceptable, and CI/CD provides the mechanism to integrate those checks into the software delivery process.

As AI applications move from prototypes into production, I believe this evaluation layer will become an increasingly important part of the engineering lifecycle. As AI systems become increasingly probabilistic, release confidence depends not only on whether the application works, but also on whether its behavior remains acceptable after every change. Traditional CI/CD validates the software. AI-enabled CI/CD validates the software—and its behavior.

Using AI to Optimize an Agentic RAG Platform: Why Local Validation Still Matters

In my previous articles, I explored how observability supports AI security, governance, and compliance. While continuing development of my Agentic RAG platform, I found myself applying many of those same principles during development. This time, however, the focus wasn't on production observability—it was on improving the engineering feedback loop itself.

The feature I was working on was my Evaluation pipeline. As I expanded evaluation capabilities, latency became the biggest obstacle to rapid iteration. Faster evaluations meant faster experimentation, shorter development cycles, and more opportunities to validate improvements before moving on to the next feature.

Rather than treating AI as an automatic code generator, I approached it as an engineering collaborator. I asked it to analyze bottlenecks, suggest targeted optimizations, explain the reasoning behind each recommendation, and propose alternatives where appropriate. The quality of the suggestions was impressive. Several changes significantly reduced latency, simplified portions of the implementation, and improved the overall responsiveness of the evaluation workflow. Two images below show latency before and after improvements.

The biggest lesson, however, wasn't about AI's ability to optimize code—it was about the importance of validating every optimization locally.

At a recent Codex Meetup in San Francisco, I had an interesting conversation with an engineer from Reddit. During our discussion, he mentioned that he didn't have a local environment for validating AI-generated changes before committing them. That conversation stayed with me because my experience had been quite different. Several optimizations suggested by AI appeared perfectly reasonable during review, but only local execution revealed whether they actually delivered the expected improvements. In a few cases, I discovered regressions, subtle behavioral differences, or optimizations that simply didn't produce meaningful performance gains. Those findings would have been difficult to identify through code review alone.

That experience gradually evolved into a repeatable engineering workflow. Every optimization began with measuring the existing bottleneck, followed by targeted AI-assisted suggestions, incremental implementation, local validation, latency benchmarking, and a full evaluation run before any code was committed. The AI dramatically accelerated the optimization process, but measurement and validation remained essential parts of the engineering discipline.

Human-Validated AI Development Cycle

One unexpected realization was how closely this mirrors the principles behind AI observability. In production, we rely on traces, evaluations, and monitoring to verify how AI systems behave under real workloads. During development, local validation serves a similar purpose—it provides objective evidence that a proposed change actually improves the system rather than simply appearing to do so. Whether validating an LLM response or an AI-generated code change, evidence consistently proves more valuable than assumptions.

This experience reinforced an important lesson for me: AI coding assistants are becoming exceptional implementation partners, but they don't eliminate the need for engineering rigor. They generate ideas, accelerate experimentation, and often uncover optimizations that might otherwise be overlooked. It is still the responsibility of the engineer to measure, validate, benchmark, and confirm that those improvements hold up in practice.

As AI becomes a regular part of software development, I believe the most effective teams won't be those that rely on AI the most—they'll be the ones that build disciplined workflows around it. In my experience, combining AI-assisted development with local validation, repeatable measurements, and continuous evaluation produces far better outcomes than either approach alone.

When Observability Stops Being a Tool Choice and Becomes an Architecture Choice

In my previous article, When AI Observability Becomes Compliance Infrastructure, I argued that observability is evolving from a debugging capability into a foundational requirement for governance, auditability, and trust.

Recently, I instrumented the same Agentic RAG application with both LangSmith and Langfuse to better understand what observability looks like in practice.

The goal was not to determine which platform is "better."

The goal was to understand how different observability approaches shape the way we operate AI systems.

What I discovered is that the most important difference between observability platforms is not their dashboards, user interfaces, or individual features.

The most important difference is what role observability plays in the overall architecture.

The Same System, Two Observability Models

The application itself remained unchanged.

The LangGraph node has a "stateful" badge to highlight its role in agentic orchestration

The same workflows executed.

The same prompts were submitted.

The same retrieval pipeline operated.

The same model generated responses.

Both platforms observed the same underlying system.

Yet they encouraged slightly different ways of thinking about observability.

One emphasized developer productivity and rapid debugging.

The other emphasized ownership, flexibility, and operational control.

Both perspectives proved valuable.

What LangSmith Does Exceptionally Well

The first thing that stood out was how quickly LangSmith provided useful visibility into the system.

Key strengths include:

• Fast setup

• Excellent workflow visualization

• Strong LangChain and LangGraph integration

• Minimal configuration

• Intuitive debugging experience

The trace views consistently made it easier to understand what happened inside a workflow and where failures occurred.

When investigating retrieval quality issues, prompt behavior, or agent execution paths, the ability to inspect individual steps significantly reduced troubleshooting time.

The overall experience feels optimized for helping developers understand system behavior as quickly as possible.

That is particularly valuable when building and iterating on complex AI workflows.

What Langfuse Does Differently

While Langfuse provides many of the same observability capabilities, the platform feels different in practice.

Several characteristics stood out:

• Open-source architecture

• Self-hosting capabilities

• Data ownership

• Export flexibility

• Vendor independence

One observation that emerged during implementation was that Langfuse felt less like a service being consumed and more like infrastructure being operated.

That distinction may seem subtle initially, but it becomes increasingly relevant as AI systems move beyond experimentation and into environments with governance, compliance, or data residency requirements.

Questions begin to emerge that extend beyond debugging:

• Where are observability records stored?

• Who owns those records?

• How long are they retained?

• Can they be exported?

• Can they remain within a particular jurisdiction?

Those questions are architectural rather than operational.

The Trade-Off That is Less Discussed

Many product comparisons focus on features.

My experience suggested a different comparison.

The most important trade-off is not interface versus interface.

It is convenience versus control.

Neither approach is inherently better.

The choice depends on what problem an organization is trying to solve.

Teams focused on rapid experimentation may prioritize simplicity and ease of adoption.

Organizations operating under stricter governance requirements may place greater value on ownership and deployment flexibility.

Both represent valid architectural choices.

What Changed My Thinking

Before implementing both platforms, I primarily viewed observability as a developer tool.

The objective was straightforward:

• Understand failures faster

• Improve debugging efficiency

• Increase development velocity

After instrumenting the same system with both LangSmith and Langfuse, my perspective changed.

I began viewing observability as an architectural layer.

The question is no longer:

"Which interface do I prefer?"

The more interesting question is:

"Who owns the operational data generated by my AI system?"

That single question influences decisions around governance, compliance, security, retention, portability, and long-term operational strategy.

It also changes how observability fits into the overall system design.

Conclusion

One of the most interesting outcomes of implementing both platforms was realizing that observability is no longer a single capability.

Tracing remains important.

Debugging remains important.

Developer productivity remains important.

But as AI systems become more deeply integrated into business processes, observability begins serving additional purposes: governance, accountability, operational control, and trust.

LangSmith and Langfuse both provide valuable visibility into AI workflows.

What differs is not simply how they present information, but how they position observability within the architecture.

My initial goal was to compare two observability platforms.

The larger lesson was that observability is increasingly becoming an architectural decision.

As AI systems mature, organizations may find themselves spending less time asking which tool they prefer and more time asking what level of ownership and control they need over the operational data their systems generate.

When AI Observability Becomes Compliance Infrastructure

AI Trust Governance Layers

In many AI projects, observability starts as a developer tool. Teams implement tracing to debug prompts, evaluate retrieval quality, or understand why an agent made a particular decision. But as AI systems move into production, observability becomes something much bigger: part of the organization's compliance and governance architecture.

The reason is simple. AI traces often contain more than technical telemetry. They may include prompts, retrieved documents, tool invocations, customer data, business context, and decision paths. From a security and compliance perspective, these traces can become just as sensitive as the underlying application data.

This creates a new set of questions. Where does trace data live? Who controls retention policies? Can traces remain within a regulated environment? What happens if auditors request evidence of model behavior, prompt changes, or incident investigations? At that point, observability is no longer just an engineering concern—it becomes a governance concern.

For organizations operating in financial services, healthcare, government, defense, or other regulated industries, vendor independence and data sovereignty can become important architectural requirements. Self-hosted and open-source observability platforms such as Langfuse or Arize Phoenix are gaining attention not simply because they provide tracing, but because they allow organizations to maintain greater control over where operational AI data resides and how it is managed.

This feels similar to the evolution of traditional security logging. What began as troubleshooting eventually became a critical component of audit readiness, incident response, and regulatory compliance. AI observability appears to be following a similar path.

As organizations continue building AI-powered applications, the question may no longer be whether tracing is needed. The more interesting question may be: who owns the observability layer, and does it align with your security, compliance, and data sovereignty requirements?