For a while, the mental model of an AI engineer looked like a slightly upgraded software engineer: you’d call an API, wire a prompt into the product, maybe fine-tune a model. The job was to add AI to a system, more or less the same way you’d add a new endpoint or a new database table. That model is breaking down, and what’s replacing it is more interesting — and more demanding.
From writing features to building the loop
The shift worth paying attention to is not about which models to use or how to write better prompts. It’s about what the engineer’s primary job actually is.
The old picture: an AI engineer writes the AI feature. They sit in the diff, tune the prompt, ship the change. The new picture: the AI engineer builds the loop — the system that writes, evaluates, ships, monitors, and improves the feature. Agents do most of the direct work. The engineer’s leverage comes from designing the loop well.
This is a meaningful change in altitude. An engineer who builds a good loop isn’t just faster than one who does each step by hand — they’re operating at a different level of abstraction entirely.
What a production loop actually looks like
As a concrete example, consider an always-on agent that reads the telemetry (the structured logs and traces that record how an AI system is behaving in production) from a live LLM-powered product. Every few hours — or when a Slack message flags a complaint — the agent wakes up, sweeps through recent traces, identifies patterns that look like problems, saves what it found to persistent memory (so the next run can build on it rather than starting fresh), and if the issue is significant enough, opens a pull request with a proposed fix.
A few architectural details matter here. The agent runs inside an isolated sandbox — a temporary, contained environment — so that credentials and repository access are scoped tightly to the run and revoked when it ends. A single proxy handles all traffic in and out, which means the agent’s own behavior is fully traceable. The state that persists between runs lives outside the sandbox, so if something crashes mid-run, the agent can resume from the last saved checkpoint rather than restarting blind.
This is not a toy demo. It’s what it looks like to build infrastructure that continuously improves itself.
Evals are the skill most teams are underinvesting in
Running an agent in a loop like this creates an immediate question: how do you know the agent’s output is any good? A traditional test suite doesn’t help much here — AI outputs are non-deterministic (meaning the same input can produce different outputs each time), and “correctness” is often a matter of judgment rather than a pass/fail check.
The answer is evaluations, or evals — automated checks that assess whether an agent’s output meets a quality bar. An eval has three components: a dataset of representative inputs, a prompt that defines the quality criteria, and expected outputs to test against. Teams build evals offline to catch regressions before shipping, and run them online in production to monitor live behavior continuously.
The part that teams underestimate is how much work it takes to make evals trustworthy. An eval is only useful if it agrees with human judgment. That means building ground-truth datasets — curated examples where a human has verified what a good output looks like — running the eval against them, and checking that the eval’s verdicts match. Then doing that again for the evaluator itself. Evaluating the evaluators is not a one-time task; it’s an ongoing investment.
The teams running the most reliable agent loops tend to treat evals as a core engineering discipline, not an afterthought. That’s a calibration shift for most organizations.
The full cycle
When these pieces fit together, the feedback loop closes:
– Collect telemetry from production
– Run evals on that telemetry to surface quality signals
– Run agents on top of those signals to identify issues and propose fixes
– Ship the fixes, observe the new telemetry
– Repeat
What makes this more than a pipeline is that the loop feeds itself: production traces become the next round of input data for both evals and agents. Each cycle, the system gets slightly better information about where it’s failing.
What this asks of engineers
Three practices tend to separate teams that have this working from those that don’t. The first is tracing everything — including the coding agents and tools that built the system in the first place, not only the features they produced. The second is building and running evals as a real engineering function, aligned with human judgment rather than substituting for it. The third is designing for the loop from the start, rather than building features manually and hoping to automate them later.
The underlying principle is not complicated: instead of doing each step of the improvement cycle, build the system that does each step. But getting there requires a different set of skills than most engineering teams have yet developed — and that gap is probably where the most interesting work is happening right now.
Text summarized and optimized using Anthropic’s models and reviewed by a human.