Designing an Agentic RAG API: What the Response Actually Tells You - Part Two: From Model Responses to Document-Grounded Answers: Extending an Agentic RAG API
In Part One, we looked at what an Agentic RAG API response tells a client—and what it does not. There is another important distinction: making documents available on a platform is not the same as ensuring that a particular API request uses them to answer a question. For applications that rely on internal knowledge, policies, technical documentation, or customer records, the connection between a request and its supporting evidence is an essential part of the API design.
API flow design consideration for Agent RAG
A natural next step is to make document selection explicit. Rather than relying only on a model ID and a knowledge-base ID, an API could allow a client to reference specific documents or a managed collection of documents. Those documents might be stored in an object store such as Amazon S3, while the platform manages their ingestion, indexing, and retrieval. The important part is that the API defines how document references become searchable knowledge—not simply where the files reside.
For example, a future API design might allow a client to identify a document collection alongside its question. The platform could then retrieve relevant passages and provide them as context to the selected model. This would make the intended knowledge source explicit and create a clearer path from the original documents to the generated answer. This is a proposed extension, not a capability demonstrated by the current /v1/chat endpoint.
There are several implementation details behind this seemingly simple change. Documents need stable identifiers, ingestion and indexing status, access controls, and a way to handle updates or removal. The retrieval stage must select relevant material, and the application should ideally be able to identify which sources supported the answer. A reference to a document alone does not guarantee that it was retrieved, used by the model, or sufficient to answer the question.
The broader design principle is that model access and knowledge access are separate responsibilities. Selecting a model determines which model generates a response; connecting a request to a knowledge source determines what evidence can inform it. An Agentic RAG API should make both responsibilities explicit and predictable. That is a practical step toward applications that are not only capable of generating answers, but can also explain where those answers came from.
Designing an Agentic RAG API: What the Response Actually Tells You - Part One
An API response can look simple, but each field in the contract matters. In deepVerified, the /v1/chat endpoint accepts a model ID, a knowledge-base ID, and a user message. The response contains three fields: id, model, and content. The id identifies the API response, while model echoes the requested public model ID. Neither should be confused with an execution trace or the underlying runtime model.
API call samples using two different models
Model selection also matters. When we submitted the same question about China's 2026 GDP growth forecast using two different models, qwen returned a forecast of 5.0%, while deepseek-r1 said the available context did not contain a forecast. These are examples of different model outputs, not verification of the forecast itself. The API provides a consistent response structure, but the content can vary with the selected model.
How can a client discover what a public model ID actually represents? DeepVerified's /v1/models endpoint provides the available model IDs, their providers, and the underlying runtime models currently mapped to them. For example, a client can request qwen without needing to specify the Ollama runtime model directly. This separates the API's public interface from the model implementation behind it.
Observability is another distinct concern. The API response ID identifies the returned response; it is not a LangSmith trace ID. deepVerified uses LangSmith separately to inspect execution traces and investigate application behavior. Keeping the response contract focused on client needs while maintaining a separate path for developer diagnostics avoids exposing internal execution details unnecessarily.
There is one more question worth exploring: how does an API request connect the model to the documents that should support its answer? A platform may support document uploads, but that does not by itself mean every chat request retrieves and uses those documents. In Part 2, we'll look at how document references and retrieval could become an explicit part of an Agentic RAG API, moving from model-generated responses toward answers grounded in supporting evidence.
AI Observability in Practice: When Evaluation Traces Go Missing
AI observability is most useful when it gives us visibility into how an AI system actually executed—not just the final answer. But implementing that consistently can be more difficult than it first appears. While working on an AI evaluation pipeline, I encountered a case where traces were recorded correctly when an evaluation was triggered through the UI, but disappeared when the same evaluation was executed through the command line or an API. The evaluation itself completed successfully in both cases, making the problem easy to overlook.
The first assumption was that the tracing system was not working correctly. The more useful question, however, was whether the two execution paths were actually initializing the application in the same way. The UI path was responsible for initializing parts of the observability configuration and execution context that were not necessarily present when the evaluation was launched programmatically. As a result, the evaluation logic was shared, but the observability lifecycle was not. This is an important distinction for AI applications: two paths can execute the same evaluation logic while producing different observability results.
The solution was to move observability initialization out of the UI-specific execution path and into the shared evaluation/application layer. That way, the UI, CLI, and API could all use the same initialization and tracing lifecycle before executing an evaluation. Conceptually, the architecture changed from UI → initialize tracing → evaluation to UI / CLI / API → shared evaluation layer → initialize tracing → evaluation → trace. This also made the behavior easier to test: an evaluation should produce the expected trace regardless of how it was triggered.
This matters beyond a single missing trace. AI evaluation is increasingly becoming part of automated workflows such as CI/CD, scheduled regression runs, and production monitoring. If observability only works when an engineer launches an evaluation manually through a UI, important execution evidence can disappear precisely when evaluations become automated. Evaluation tells us whether a system met a defined criterion; tracing helps us understand how the system reached that result. Both need to remain available across the execution paths that an engineering system actually uses.
This experience reinforced a design principle behind deepVerified: AI evaluation and AI observability should be execution-path independent. An evaluation should be traceable whether it is triggered interactively, from a command line, through an API, or as part of an automated pipeline. As AI systems move further into production, observability cannot be treated as a UI feature—it needs to be part of the underlying evaluation infrastructure.
What Should an AI Evaluation Actually Test?
An AI system can produce a convincing answer and still have significant problems underneath. The system may retrieve the wrong documents, generate an answer that isn't supported by its context, choose an inappropriate tool, or become slower and more expensive after a change. This makes AI evaluation fundamentally different from simply checking whether an output looks correct. An effective evaluation suite needs to test the behavior of the system across its entire execution path, not just the final response.
For a RAG or agentic AI system, that means evaluating several different dimensions:
Retrieval — Did we retrieve the right information?
Grounding — Is the answer supported by retrieved/contextual data?
Answer quality — Is the response accurate and useful?
Agent behavior — Did the system choose the right tool or action?
Safety / policy — Did it stay within defined boundaries?
Performance — Did latency or cost degrade?
Regression — Did a model, prompt, retrieval, or code change make existing behavior worse?
These checks also shouldn't necessarily run at the same stage of the development lifecycle. Fast and deterministic checks can run earlier, while broader behavioral evaluations can run in staging before a release. Production systems can then use continuous evaluation to identify changes in behavior over time. The appropriate evaluation strategy depends on the architecture and the risks of the particular application.
This also changes how AI releases should be viewed. A model upgrade, prompt change, embedding change, retrieval modification, or agent workflow change may leave the application's API and traditional software tests completely intact while changing how the system behaves. A successful build does not necessarily mean a successful AI release. Evaluation provides another layer of validation focused specifically on behavior.
The goal, therefore, isn't to create one universal AI score. It's to build an evaluation suite that reflects the actual failure modes of the system and provides meaningful evidence before and after deployment. As AI systems become more complex, evaluation becomes less about asking “Did the model give a good answer?” and more about asking “Did the system behave as expected?”
AI Evaluation Doesn't End at Deployment
In my previous article, I discussed AI evaluation as a new stage in CI/CD—a quality gate that can help determine whether an AI system is ready to move from staging into production. But passing that gate only tells us that the system behaved acceptably at the time it was evaluated. Once deployed, the environment changes, user behavior changes, data changes, and AI systems can behave differently over time. For production AI applications, evaluation therefore shouldn't be a one-time release activity. It should become part of a continuous feedback loop.
A practical architecture can treat evaluation as two connected stages. Pre-deployment evaluation runs against a staging version and checks metrics such as retrieval quality, grounding, answer quality, tool-call behavior, latency, and regression against a validated baseline. If the results meet the required thresholds, the release can proceed. Continuous evaluation then operates after deployment, using appropriately sampled production interactions, curated evaluation datasets, or both to identify changes in system behavior. The purpose is different: the first stage asks "Is this version safe to release?" while the second asks "Is the system still behaving as expected?"
This is where observability and evaluation complement each other. Observability provides the evidence about what happened inside the system—retrieval steps, model calls, tool usage, latency, errors, and traces. Evaluation adds another layer of interpretation by determining whether that behavior meets the expected quality criteria. For example, a production Agentic RAG system might show stable latency and successful requests while retrieval quality or grounding gradually deteriorates. Observability can expose the change; continuous evaluation can help identify it as a quality regression.
The results of continuous evaluation can then become another input into the engineering lifecycle. A detected regression might trigger an investigation, create a new evaluation case, lead to a prompt or retrieval change, or ultimately result in a new code change. That change moves through the same CI/CD process again, including pre-deployment evaluation before reaching production. This changes the traditional CI/CD model from a linear pipeline into a continuous learning loop: Code → CI → AI Evaluation → Deploy → Observability → Continuous Evaluation → Code.
As AI systems become more dynamic and increasingly agentic, I believe this feedback loop will become an important part of production engineering. Traditional monitoring tells us whether the system is available and what happened during execution. Evaluation helps us understand whether the system's behavior remains acceptable. Together, they create a mechanism not only to release AI systems with greater confidence, but also to continuously learn from their behavior in production. The goal isn't simply to evaluate an AI system before deployment. The goal is to keep evaluating whether it remains worthy of trust after deployment.