For a long time, the question asked about an AI system was: “Does it give the right answer?” That question makes sense when a system is a single model call — one prompt in, one response out. It no longer makes sense for most systems being built today. The gap between what is measured and what can go wrong is the biggest sources of unreliability in production AI.
How We Got Here?
The architecture of AI applications has gone through several distinct generations in 3 years, each one forced by a step-change in what models could actually do.
In 2023, systems were single-call pipelines: send a prompt, receive a completion, check the answer. The eval problem was reasonable: measure final answer accuracy, catch hallucinations, and handle knowledge cutoff issues.
Then came RAG — the approach of pulling relevant documents (chunks rather) from a database and injecting them into the prompt before generating a response. Suddenly there were additional intermediate steps to get wrong: parsing, retrieval ranking, context assembly. Evals had to become more complex to cover those steps, to avoid that a broken retrieval stage silently poisoned quality upstream.
Then came the ReAct loop — a pattern where the model reasons about a task, calls a tool, observes the result, and repeats until the task is done. In mid-to-late 2023, tool selection was unreliable, arguments came out malformed, and long reasoning chains collapsed. Teams often found themselves debugging failures that their evals weren’t designed to catch because the evals still targeted final answers, not the trajectory that produced them.
The pragmatic response was workflow graphs: encoding the orchestration logic explicitly into a state machine, letting the model operate only at specific decision nodes. More predictable, yes — but brittle outside the paths the designers anticipated, and carrying an entirely new eval surface: node-by-node correctness, branch consistency, inter-node contracts, retry behavior.
Then, in mid-to-late 2025, the underlying models changed dramatically. Tool calling became reliable. Planning improved. Self-correction emerged as a genuine capability. Teams using graph-based systems found themselves sitting on architecture designed to compensate for limitations that no longer existed. The natural response was to return to the ReAct loop — but now in a world where capable models produce high-variance trajectories. The same input can generate very different sequences of tool calls and still arrive at a correct answer.
This variance is what broke evals that weren’t designed for it.
New Metrics
When trajectories are variable, a simple pass/fail eval on a single run is no longer meaningful. Two metrics have emerged to fill the gap.
Pass-at-k asks: across k independent runs of this task, does the system succeed at least once? This measures whether the capability exists at all — the system can, in principle, do the thing.
Pass-wedge-k (sometimes written as pass∧k) asks: across those same k runs, how often does it succeed? This measures reliability — whether that capability is consistent enough to be useful in production, where you typically get one shot.
A system that scores well on pass-at-k but poorly on pass-wedge-k is capable but not reliable. Both numbers matter, and neither one is redundant.
The Cycle Problem
In principle, the idea of a production loop where real usage shows new failure modes, those become new evals, and the system improves, is acceptable In practice, most teams don’t actually run this loop. The evals are static: defined once at launch, updated rarely, and increasingly blind as the system evolves.
Static evals can catch the failures you anticipated. They cannot catch the failures not anticipated, the unknowns unknowns. A production system that has been running for six months will have encountered user behaviors, edge cases, and interaction patterns that nobody wrote a test for. Without a mechanism to surface those patterns and convert them into evals, the coverage gap compounds.
The mechanism required is cluster analysis on production data — grouping real interactions to reveal failure categories that weren’t on anyone’s radar at launch. This turns the unknown unknowns into known ones that can be measured and addressed.
What This Means Going Forward?
Models are not done evolving. The pattern of capability upgrades forcing architectural rearchitecture is likely to continue. So practically no eval suite is finished — it has a shelf life tied to the architecture it was written for.
The teams that will navigate this most effectively are those who treat evals as a living system: one that grows alongside the architecture, stays connected to production data, and is explicitly designed to surface novel failures, not just the ones that were anticipated at build time.
The measurement discipline is, in the long run, at least as important as the architecture itself.
Text summarized and optimized using Anthropic’s models and reviewed by a human.