The problem

Most teams evaluating LLM agents default to one eval pattern — usually checking individual model calls for clean input/output pairs — and stop there. That gives a false sense of coverage. A span-level eval can look perfectly clean while the agent calls tool B before tool A even though B depends on A, or while two agents in a pipeline are silently passing corrupted context to each other, or while a multi-turn conversation technically “completes” while leaving the user frustrated and unresolved. Each of these is invisible to the eval scope that isn’t built to catch it.

The same blind spot shows up at the enterprise level in a more expensive form. Companies pour spend into LLM-as-judge evals for everything — because it feels like the “AI-native” thing to do — when a big share of what they’re checking (valid JSON, required fields non-null, schema conformance) is a deterministic check that costs nothing and is more reliable than any model judge. And even where LLM-as-judge is the right tool, almost nobody calibrates whether the judge actually agrees with a domain expert before trusting it at scale. Both mistakes are cheap to make and expensive to discover in production.

The approach

This is a curated study, not a running app — a structured knowledge base built with my own research pipeline (CogitOS), which pulls in Substack articles, YouTube talks, and other source material on a topic and synthesizes each into a discrete, sourced idea with a link back to the original. For this project that pipeline processed roughly two dozen sources — talks from AI Engineer conferences, breakdowns of specific model releases, postmortems of agent failures — and organized the output around one core framework plus a set of tightly related sub-frameworks.

The core framework is the four-scope eval hierarchy: span evals check a single LLM call’s input/output; multi-span evals check whether components are correctly passing context to each other; trajectory evals check whether a multi-step agent executed things in the right order (the concrete example that anchors it: a trajectory eval catching a tool called before its dependency, a bug that looked completely clean at the individual-call level); session evals check whether a full multi-turn conversation actually got resolved, not just whether it technically ended. The practical takeaway the study builds toward is that you pick eval scope based on the failure mode you’re worried about, not based on what’s easiest to instrument — and that running only span evals, which is what most teams do because it’s the easiest to bolt on, leaves the other three classes of failure completely uncovered.

Layered on top of that is a cost-discipline framework: five eval signal types (deterministic checks, LLM-as-judge, human feedback, golden datasets, business metrics) with very different cost profiles, and the rule is to use the cheapest one that reliably answers the question before reaching for a more expensive one — deterministic checks run on every trace, LLM judges get reserved for genuinely semantic questions, human review is for the ambiguous high-stakes residue. And the golden-dataset piece specifically addresses a question most people skip: golden datasets aren’t just “ground truth for evals,” their real job is calibrating whether your LLM judge is trustworthy as a proxy for a domain expert in the first place — you run both on the same sample, check agreement, and only scale the judge once it holds up. The rest of the corpus fans out from there into adjacent, concretely evidenced material: six recurring agent failure patterns (context degradation, spec drift, sycophantic confirmation, tool selection errors, cascade failures, silent failure), why completion rate and acceptance rate are different metrics that most dashboards conflate, why verification steps should be audited on interception rate rather than existence, and why user corrections in production are pre-labeled eval data that most teams throw away instead of feeding back into the suite.

What I learned

The clearest lesson is that eval coverage is not a single number you can improve by adding more of the same kind of check — it’s a set of independent dimensions, and being strong on one (say, per-call correctness) tells you nothing about the others (sequencing, cross-component handoffs, conversation-level resolution). The second lesson, which came up independently across several unrelated sources in the corpus, is that silent failure — plausible-looking wrong output with no error signal — is the failure mode every other technique in the framework exists to defend against, and it’s the one aggregate accuracy dashboards are worst at surfacing, because they hide exactly where the failures concentrate.

Where this could go

The natural extension is turning the framework into something runnable: a small eval harness that actually implements the four scopes against a real agent pipeline, with the deterministic-first cost layering built in as a triage step rather than a slide. The golden-dataset calibration piece is the part I’d build first, since it’s the one most teams skip entirely and the one that determines whether everything layered on top of an LLM judge is trustworthy.

The enterprise version of this problem is bigger than eval tooling. Several sources in the corpus converge on the same point from different angles: writing a good eval is the same skill that makes a senior person valuable in the first place — knowing what “right” looks like in a specific organizational context, and where an agent will go wrong in ways it can’t anticipate for itself. Companies that hand eval-writing to junior staff because it looks like a chore are handing the highest-leverage part of agent deployment to the people with the least context to do it well. And this generalizes past engineering — a legal team can write evals that check for conflicts with informal relationship terms, a finance team can write evals that catch projections contradicting board commitments, none of it requiring code. The organizations that build this discipline early — logging failures, converting them into permanent regression checks, calibrating judges against experts instead of trusting them by default — are building a compounding advantage that’s hard to buy off the shelf, the same way a company that’s spent years encoding domain-specific rejections into its systems ends up with judgment no competitor using the same base models can replicate.