What Should an AI Evaluation Actually Test?
An AI system can produce a convincing answer and still have significant problems underneath. The system may retrieve the wrong documents, generate an answer that isn't supported by its context, choose an inappropriate tool, or become slower and more expensive after a change. This makes AI evaluation fundamentally different from simply checking whether an output looks correct. An effective evaluation suite needs to test the behavior of the system across its entire execution path, not just the final response.
For a RAG or agentic AI system, that means evaluating several different dimensions:
Retrieval — Did we retrieve the right information?
Grounding — Is the answer supported by retrieved/contextual data?
Answer quality — Is the response accurate and useful?
Agent behavior — Did the system choose the right tool or action?
Safety / policy — Did it stay within defined boundaries?
Performance — Did latency or cost degrade?
Regression — Did a model, prompt, retrieval, or code change make existing behavior worse?
These checks also shouldn't necessarily run at the same stage of the development lifecycle. Fast and deterministic checks can run earlier, while broader behavioral evaluations can run in staging before a release. Production systems can then use continuous evaluation to identify changes in behavior over time. The appropriate evaluation strategy depends on the architecture and the risks of the particular application.
This also changes how AI releases should be viewed. A model upgrade, prompt change, embedding change, retrieval modification, or agent workflow change may leave the application's API and traditional software tests completely intact while changing how the system behaves. A successful build does not necessarily mean a successful AI release. Evaluation provides another layer of validation focused specifically on behavior.
The goal, therefore, isn't to create one universal AI score. It's to build an evaluation suite that reflects the actual failure modes of the system and provides meaningful evidence before and after deployment. As AI systems become more complex, evaluation becomes less about asking “Did the model give a good answer?” and more about asking “Did the system behave as expected?”