Over time, production prompts tend to put on weight. A paragraph for the edge case that came up last month. A worked example to stop the model from over-formatting. A three-sentence anti-hallucination plea that no one remembers adding but everyone is afraid to remove. Before long, a simple classification task is running on five thousand tokens of accumulated scar tissue — and at scale, that becomes a significant, self-inflicted cost.
The discipline of trimming that weight without breaking anything is sometimes called token optimization. What makes it hard is not the trimming itself — it is knowing when “different” actually means “worse.”
The Ceiling Test: Know What You Are Comparing Against
The most underappreciated step in any cost-reduction project is measuring how consistent the current system already is — before touching anything.
Here is why it matters: if an expensive production model is run three times on the same inputs, it will disagree with itself around seven percent of the time. That is just inherent model variance — the natural unpredictability baked into how large language models generate text. When a cheaper candidate model is tested against the production model, it might disagree eighteen percent of the time — but seven of those percentage points are noise the production model generates on its own, not actual regression from switching models.
Without this baseline, teams hold the cheap model to a standard the expensive one cannot meet either. The ceiling test — measuring how often the current system agrees with itself — sets the mathematical floor for what “different” actually means versus what “worse” actually means. It is a simple idea, and it changes the entire framing of the optimization project.
Three Levers, in Order
Once the baseline is established, there are three places to pull:
Cache aggressively first. Structuring prompts so that repeated prefixes — the instructions, system context, and stable parts of the input — are reused across calls (a technique called prefix caching) is the cheapest win available. It requires no quality trade-off and delivers immediate cost reduction.
Then shrink the prompt itself. Rather than hand-editing instructions and hoping for the best, treating the prompt as a search problem produces more reliable results. Tools like DSPy — a framework that lets practitioners define inputs, outputs, and a scoring function, then automatically searches for better prompt formulations — can cut instruction-token counts by fifty percent or more while maintaining the same output contract. Worked examples and multi-paragraph explanations get replaced by compact rules. The result is often leaner and more robust at the same time.
Then consider downsizing the model. Two approaches are worth knowing. Decomposition means using cheaper, faster models for routing and extraction while reserving frontier models — the largest, most capable models from leading providers — for tasks that genuinely need heavy reasoning. Distillation means training a smaller model on the outputs of the larger one, using the same automated search loop, to teach it to match large-model performance on a specific task distribution. Both come with caveats: smaller models introduce new failure modes, and not every downsizing experiment produces something shippable.
Validating the Result
This is where many cost-reduction projects go wrong — either by shipping on intuition, or by treating any disagreement between old and new as a regression.
A shadow deployment runs both systems on identical inputs and measures three disagreement rates: the current system against itself, the candidate against itself, and the two against each other. The net drift — the disagreement between systems, minus each system’s internal variance — is the true signal. Crucially, disagreements need human review: “different” is not the same as “wrong,” and some differences are improvements.
The decision rule worth following: compute a confidence interval around the net gain (a statistical range that accounts for uncertainty in the measurement) and ship only if the lower bound of that interval is above zero. Run this calculation separately for each critical subcategory in the data — a strong global average can easily mask a failing slice.
Where This Is Heading
As LLM deployments mature, teams running them at scale are moving toward treating prompt optimization as an engineering discipline rather than an art form. The tools to do it systematically — automated search loops, shadow deployments, statistical validation — already exist. The gap is mostly organizational: building the labeling infrastructure, the telemetry, and the habits to use them.
The cost savings are real. The harder shift is accepting that getting there requires the same rigor applied to any other production system change: measure first, optimize second, and ship only when the numbers support it.
Text summarized and optimized using Anthropic’s models and reviewed by a human.