When Keeping Everything Turns Out to Be Cheaper Than Summarising It
In early LLM applications, context management felt like an obvious problem: when the conversation window fills up, summarise it and move on. Fewer tokens means lower cost, lower latency, a tidier system. The pattern spread fast, but was it stress-tested against alternative?
A production experiment run on a real AI tutoring system — handling debugging sessions with AI engineering students — has put that intuition to the test. The results are worth analyzing because they shift the default assumption.
The Compaction Reflex, Revisited
The tutor application faced two genuine pressures: a finite context window and stateless models that forget everything between sessions. The team focused on the first — in-session context management — and ran eleven different presets against real student questions and synthetic multi-turn scenarios.
The presets ranged from truncating long tool outputs, to sliding windows keeping only the last N turns, to rolling summarisation, to full-reset summarisation (where the entire history is collapsed into a single summary before continuing). Each approach has a plausible rationale. Several are in common use.
The finding that ran counter to practitioners’ instincts: full history — no compaction at all — won on recall, cost, and speed across the board.
Memory recall in multi-turn sessions was 95 percent with full history. With summarised context, it dropped to 32 percent. That is not a marginal difference; it is the kind of gap that makes a tutoring session either coherent or frustrating.
Caching Flips the Economics
The mechanism behind this is prompt caching — a feature now available from several providers that stores the computed state (the “KV cache,” short for key-value cache, the internal representation a model builds as it reads tokens) for repeated sequences. When the next request reuses the same prefix, the provider can skip recomputing it, which can reduce costs dramatically. DeepSeek V4 Flash, for example, offers up to fifty times cheaper pricing for cached tokens; Gemini Flash offers around fifteen times cheaper.
Here is where the reflex breaks down: compaction destroys the cache. A summarised or truncated context is a new sequence — a cache miss — so the system pays full price for it. To break even against a cached full history, compaction would need to reduce the context by more than fifty times while maintaining quality. That threshold is very hard to clear.
The experiment proved this. With DeepSeek V4 Flash and full history, roughly 97 percent of tokens were cached. Clearing tool outputs (a common compaction strategy) forced the agent to re-retrieve information it already had, generating more tool calls, more tokens, and higher latency. The “savings” from compaction became a cost addition.
Local Models Are Different
This is not a universal verdict against compaction. The same experiments on a MacBook with a 32K-token context cap produced a different result. Once a single lesson exceeds the window, caching stops helping and compaction becomes necessary. With local models, full history recalled specific facts only around 33 percent of the time — comparable to the worst cloud-based summarisation results.
Document retrieval locally was different: RAG achieved 100% accuracy. Stuffing the entire document into the window beyond its capacity: 340 seconds to produce a single-token output. The constraint is real, and the fix is targeted.
One finding regardless of environment: dense vector search alone degraded to zero percent recall when facts were buried in the middle of a 400K-token context. BM25 keyword search maintained 100%. Hybrid retrieval — semantic plus keyword — is not a hedge; it is load-bearing.
The Principle
The central lesson from this body of work is not “never summarise” but “do not compact by default”. Identify the actual constraint first — is it the context window ceiling? Caching unavailability? Local hardware limits? — and then apply the most targeted fix for that specific problem.
The compaction reflex is easy to automate and hard to audit. Systems get built with summarisation baked in because it sounds prudent, and no one runs the counter-experiment. The result is systems that pay more, recall less, and feel less coherent to users — while appearing to be well-engineered.
As caching becomes a standard feature across providers and local hardware pushes context window sizes upward, the economics will keep shifting. The durable practice is not a particular technique but the habit of measuring before deciding — and treating “keep everything” as a serious hypothesis rather than the naive option.
Text summarized and optimized using Anthropic’s models and reviewed by a human.