For a while, the standard way to check whether an AI agent was working was to run it a few times, watch the outputs, and decide it looked reasonable. That was fine when agents were assistants you talked to. It stops being fine the moment they act autonomously — calling tools, writing reports, making decisions — in environments where their failures are expensive or hard to detect.
The field has been slowly building a more systematic answer. What that looks like in practice, end to end, is worth understanding.
Three kinds of evaluation, and why you need all of them
The most durable framing treats agent evals the same way a security team treats controls: no single layer catches everything, so you stack them.
Code evaluators are pure functions — given a trace (the complete log of what the agent did, including every tool call and response), they check deterministic properties: does the output mention the required stock ticker? Is the JSON parseable? Does the response stay under a length limit? They run in milliseconds and cost nothing. They will miss anything that requires understanding.
LLM-as-judge evaluators use a more capable model to score dimensions that require reasoning: is the report faithful to the source material? Does it make a specific recommendation rather than hedging? These catch what code cannot, but they are slower, cost money, and carry a small error rate. The judge model should be more capable than the agent model being evaluated — otherwise it is too easy to fool.
Human evaluations are the ground-truth layer. A domain expert reads a sample of traces, labels them, and flags failure modes the automated evals are not yet looking for. This is slow and expensive, but it is the only way to discover failure categories you have not yet thought to measure.
The three layers are complementary in the way that overlapping security controls are complementary: each has blind spots, but the overlap between them leaves fewer gaps.
Read the traces before writing any eval
The most common mistake is automating evaluation before understanding what failure looks like. The better order is: run the agent, collect 10 to 20 traces, and read them end to end before writing a single evaluator.
What comes back is usually surprising. In a financial-analyst agent built as a demo, the most common failure was not hallucination or wrong answers — it was harness leakage: the agent trying to write files to disk even though it had been told not to. That failure appeared in most of the agent’s writing turns. Automated evals written before this was noticed would have missed it entirely.
The discipline here is sometimes called open coding in qualitative research: write down what you observe in plain language, without forcing it into categories in advance. Then group the observations into failure categories (axial coding). Only once those categories exist should you write evaluators designed to detect them.
The god-evaluator trap
Once teams start writing LLM-as-judge evaluators, the instinct is to build a comprehensive one: a single prompt that checks accuracy, tone, completeness, format, and compliance all at once. This is the wrong approach.
When a composite evaluator returns a low score, there is no way to know which dimension failed. A report that is accurate but badly formatted and a report that is well-formatted but fabricated will both get poor scores, but they require entirely different fixes.
The approach worth keeping is one evaluator per dimension. A faithfulness evaluator that checks whether the report sticks to what the source material actually says. A separate actionability evaluator that checks whether the report makes a specific recommendation. A separate compliance evaluator for anything that touches regulated content. Each produces a label and an explanation. When the agent regresses, you know exactly where.
Structure a judge prompt properly
A well-written judge prompt has five components: a role that tells the model what expertise to simulate; explicit, concrete criteria (not “good and helpful” but “makes a specific buy/sell/hold recommendation with named supporting evidence”); data wrapped in XML tags to prevent the instruction and the content from bleeding together; a small number of labeled examples showing what passing and failing look like; and a constrained output via tool definitions that forces the model to return one of a fixed set of labels rather than free text.
Examples in the rubric can cause overfitting — a set of examples focused on one failure pattern will sometimes cause the judge to miss different-but-valid outputs. Testing with and without examples before finalizing is worth the overhead.
Where this is heading
The natural end state for this kind of work is evaluation as a CI gate (continuous integration — automated checks that must pass before any code change goes live). A new prompt version, a changed tool description, a different model — any of these can be tested against a curated dataset of real agent traces and scored against defined thresholds before reaching production. The evidence trail replaces “pushed on vibes” with something reproducible.
The investment is real: curating a good golden dataset takes time, and writing evaluators that actually agree with human judgment requires calibration. But the alternative — discovering failures only when users report them — gets worse the more autonomous the agent becomes. The sooner that evaluation infrastructure exists, the less expensive it is to maintain as the agent evolves.
Text summarized and optimized using Anthropic’s models and reviewed by a human.