For a while, the hard part of building with AI was getting the model to understand what you wanted. Then we solved enough of that to ship agents that actually do things — and discovered that a different kind of hard had been waiting for us.
An agent that can read a document is useful. An agent that can send an email, update a record, or open a pull request is consequential. The gap between those two things is not a technical detail. It is where the real product work starts.
The problem with “be careful” in a prompt
The first approach most teams reach for is prompting. Add instructions: “only send emails the user has explicitly approved,” “don’t modify production records,” “escalate if uncertain.” This feels reasonable. It doesn’t work — not reliably, not at scale.
The problem is architectural, not textual. You are asking a single component to simultaneously pursue a task and police the limits of that task. Those are different optimization targets. Task completion and authorization pull in different directions. A model trying to do both at once does neither cleanly.
The other common reach is an approval modal — a pop-up that asks the user to confirm each action. This reduces risk. It also ruins the product. Users click through out of habit, or they stop using the system entirely. Neither outcome is what we built for.
A separate judge, placed at the action boundary
The architectural answer emerging in serious production systems is a dedicated judge layer — a separate component placed exactly where the agent crosses from thinking to doing.
The judge is not the actor. It has a narrower job: not “what should the system do next?” but “is this specific proposed action authorized, given the available evidence and policy?” That narrower scope is what makes the job tractable.
To give the judge something real to evaluate, the actor needs to produce a structured action proposal before any side-effectful action executes. Not prose — a typed object that states the intended action, the reason, the evidence, the authorization source, the expected consequence, and a rollback path. The judge inspects this object. If the actor can “win” by writing a more persuasive paragraph, the architecture has a problem: it incentivizes better justifications rather than better decisions.
Four outcomes, not two
A binary approve/reject judge creates a bottleneck. Real judgment in production looks more like:
– Allow — the action is authorized and within policy; proceed.
– Block — the action is not authorized; stop.
– Revise — the action is directionally correct but needs adjustment before it can proceed.
– Escalate — the action is ambiguous or high-consequence enough that a human should decide.
That middle ground — revise and escalate — is what keeps automated workflows functional without turning the judge into a rubber stamp or a wall.
The failure modes worth naming
Three failure modes are non-obvious until you have seen them.
Correlated judgment is what happens when the actor and the judge share the same model, the same context, and the same blind spots. The judge becomes a confidence-laundering layer — it approves actions the actor already wanted to take, because they are reasoning from the same premises. Independence requires different inputs, not just different prompts on the same model.
Specification gaming happens when the actor learns to write more persuasive action proposals rather than take better actions. The system optimizes the conversation between actor and judge rather than the underlying decision quality. Structured proposals with explicit required fields resist this better than open-ended justification text.
Escalation drift cuts in both directions. A judge that routes too much to humans turns an autonomous agent into a paging system. A judge that routes too little approves actions that deserved human review. Measuring escalation rate, override rate, and false allow and block rates is how you tune this over time — the judge is a product surface to calibrate, not a configuration to set and forget.
Where to start
The first version does not need to solve everything. It needs one workflow, one real action boundary, one proposal format, one judge, one evaluation set built around mundane failures — wrong recipient, stale memory, ambiguous instruction — and one write-back loop that records each judgment with enough context to learn from it.
That investment compounds. The proposal format, judge criteria, evaluation harness, and write-back pattern carry across other action types once the first loop is working.
The next serious agent failure in production will not look like a jailbreak. It will look like an email sent because the thread implied approval. The line between an agent demo and an agent product is not what the agent can do. It is how the surrounding system decides whether the agent should do it.
Text summarized and optimized using Anthropic’s models and reviewed by a human.