There was a moment when deploying better AI meant waiting for the next model release. More parameters, better benchmarks, shinier outputs — that was the frame. What the field has gradually figured out is that the bigger opportunity sits in a layer most practitioners barely touched: the scaffolding around the model, not the model itself.
That scaffolding has a name: the agent harness — the combination of memory, tools, context assembly rules, and the loop that drives the whole thing. The same frozen language model performs markedly differently depending on how its harness is configured. That is both a constraint and an opportunity.
The harness is what we actually control
Frontier models are shipped and maintained by a small number of labs. Practitioners choose among them, but they do not shape them. The harness is different: it belongs entirely to the team building the application. Memory structure, what context gets assembled before each reasoning step, which tools are available and when — all of that is open design space.
This reframing matters because it changes what is worth optimizing. Fine-tuning (adjusting the model’s internal weights directly) is expensive and brittle. A well-designed harness can be iterated on in an afternoon.
Memory is not one thing
The most immediately practical piece of harness design is memory architecture. There are two levels to work with.
Short-term memory is the context window — everything the model can see in a single session. It is fast, zero-latency, and completely transient: when the session ends, it is gone.
Long-term memory splits further into three kinds:
– Episodic — logs of past conversations, summaries, tool call histories
– Procedural — stored skills and workflows the agent can retrieve and execute
– Semantic — known facts, user preferences, standing assumptions about the world the agent operates in
A common mistake is to treat short-term memory as the only memory. The result is an agent that starts every session cold, rediscovers things it already knew, and produces inconsistent outputs across runs. Adding even a simple long-term store changes the character of the system substantially.
Bigger context is not better context
Here is something that cuts against intuition: larger context windows tend to produce worse reasoning, not better. A model working at near-zero context occupancy solves problems more accurately than the same model loaded to 256,000 tokens of fill. Keeping the context window small is an active design goal, not just a cost concern.
This makes the toolbox problem tractable. A realistic agent skillset might include hundreds of tools; loading them all simultaneously would consume most of the available context before the agent has reasoned about anything. The solution is to precompute vector embeddings (mathematical representations that capture meaning) of every tool and skill, then pull only the two or three most relevant ones at runtime, using similarity search against the current task. Combining vector similarity with keyword matching retrieves better candidates than either approach alone.
Roughly 80 percent of a typical API request ends up being tools and retrieved documents. Treating that budget deliberately is the difference between a harness that fits and one that suffocates itself.
Skills that accumulate without fine-tuning
One of the more elegant patterns that falls out of this architecture is skill distillation: taking a successful past run and extracting it into a reusable skill file. That file goes into the skillbox, where the agent loop can discover it via similarity search in future sessions.
The frozen model does not change. But the skillbox grows. An agent that solves a complex compliance check today can rediscover that solution path automatically next quarter, without any fine-tuning and without any human remembering to wire it in. That is a meaningful form of organizational memory in a system otherwise built around stateless inference.
What this looks like in a real domain
Applied to fraud detection — specifically anti-money laundering compliance — these patterns translate cleanly. A suspicious-transaction flagging system runs around the clock, encounters repeating pattern types, and must produce consistent output (a Suspicious Activity Report is a regulatory document with a fixed structure). A memory-aware harness handles this well: episodic memory retains case context across sessions, procedural memory stores the reporting workflow, and pre-defined query templates — rather than dynamically generated SQL — keep database interaction predictable and audit-safe.
The discipline is building a narrow harness for a specific domain rather than rebuilding a general-purpose environment from scratch. Narrow harnesses are faster, more predictable, and easier to audit.
Where to start
The practical entry point is deliberately modest: begin with just two memory structures — one short-term, one long-term — before thinking about subcategories. Add a recall step inside the agent loop between observation and reasoning. Sophistication can accumulate from there.
The agents that hold up in production tend to have one thing in common: someone thought carefully about what the system should be able to remember, and designed for it from the start. The memory layer is not an afterthought to wire in later — it is the architecture.
Text summarized and optimized using Anthropic’s models and reviewed by a human.