AI Observability in Practice: When Evaluation Traces Go Missing
AI observability is most useful when it gives us visibility into how an AI system actually executed—not just the final answer. But implementing that consistently can be more difficult than it first appears. While working on an AI evaluation pipeline, I encountered a case where traces were recorded correctly when an evaluation was triggered through the UI, but disappeared when the same evaluation was executed through the command line or an API. The evaluation itself completed successfully in both cases, making the problem easy to overlook.
The first assumption was that the tracing system was not working correctly. The more useful question, however, was whether the two execution paths were actually initializing the application in the same way. The UI path was responsible for initializing parts of the observability configuration and execution context that were not necessarily present when the evaluation was launched programmatically. As a result, the evaluation logic was shared, but the observability lifecycle was not. This is an important distinction for AI applications: two paths can execute the same evaluation logic while producing different observability results.
The solution was to move observability initialization out of the UI-specific execution path and into the shared evaluation/application layer. That way, the UI, CLI, and API could all use the same initialization and tracing lifecycle before executing an evaluation. Conceptually, the architecture changed from UI → initialize tracing → evaluation to UI / CLI / API → shared evaluation layer → initialize tracing → evaluation → trace. This also made the behavior easier to test: an evaluation should produce the expected trace regardless of how it was triggered.
This matters beyond a single missing trace. AI evaluation is increasingly becoming part of automated workflows such as CI/CD, scheduled regression runs, and production monitoring. If observability only works when an engineer launches an evaluation manually through a UI, important execution evidence can disappear precisely when evaluations become automated. Evaluation tells us whether a system met a defined criterion; tracing helps us understand how the system reached that result. Both need to remain available across the execution paths that an engineering system actually uses.
This experience reinforced a design principle behind deepVerified: AI evaluation and AI observability should be execution-path independent. An evaluation should be traceable whether it is triggered interactively, from a command line, through an API, or as part of an automated pipeline. As AI systems move further into production, observability cannot be treated as a UI feature—it needs to be part of the underlying evaluation infrastructure.