The problem

Most teams building AI systems optimize for output quality and stop there. Nobody prices the compute. That works fine at demo scale and breaks at production scale: a workflow that looks impressive at ten runs a day can be quietly unprofitable at ten thousand. Uber is the public cautionary tale — near-universal internal AI adoption, real agentic workflows, measurable output in commits, and still a blown budget, because per-seat SaaS budgeting has no meter for a workload that plans, retries, and runs for hours. Cursor is the other one: it hit $200M ARR faster than almost any dev tool in history while sending nearly all of that revenue to its model supplier, and a single mid-2025 caching price change broke its unit economics badly enough to force a plan restructuring. Any company shipping agents without a cost model is one supplier pricing change away from the same problem.

The trap compounds because “cheaper model” and “cheaper task” are not the same thing. A model priced at half the rate per token often needs twice the tokens to get to a usable answer, so the sticker price is actively misleading. Without a discipline for measuring this, teams either torch cash on frontier models for everything, or chase token-price discounts that don’t actually lower the bill.

The approach

I built this as a structured research study inside my own pipeline (CogitOS), not a piece of software — the deliverable is a synthesized body of evidence and a decision framework, assembled from close reading of roughly 30 primary sources: engineering talks, Substack deep-dives, and case studies (Cloudflare’s code-review system, Uber’s AI budget, Anthropic’s Opus pricing changes, Sora’s unit economics, Devin’s orchestrator/executor split, and more). Each source got distilled into a standalone, sourced idea — not a link dump, a claim with its mechanism and its numbers attached.

The core framework that came out of it is a three-level ladder: Level 1 is spending tokens (trivial, valueless alone). Level 2 is making that spend produce something useful. Level 3 is capturing more value than the tokens cost, proven, before you scale it. Most “always-on agent” deployments — the study’s estimate, drawn from one of the source talks, is 90% of agent cron jobs running today — never clear level 3, they’re just burning cash on autopilot. The practical output is a set of concrete, benchmarkable tactics that follow from the framework: route by task rather than by seniority (frontier model for planning/spec-writing, cheap model for high-volume execution, since output tokens cost far more than input tokens and execution is output-heavy); measure cost-per-completed-task on your own workload rather than trusting a rate card, because a “cheaper” model that needs 2x the tokens is not cheaper; scope each agent’s context to only what it needs instead of dumping the full task into every call (Cloudflare’s reviewer system does this explicitly and reviews merge requests at roughly $1 each); and insert cheap proxy-validation before expensive generation steps so you’re not paying full price to find out an idea was bad.

This is deliberately a thinking tool, not a dashboard — the artifact is the argument and the worked numbers, meant to be applied by hand to a specific team’s workflow, the way a cost-accounting framework gets applied rather than run.

What I learned

The single biggest surprise across the sources was how consistently “cheap model” claims fell apart under actual measurement. Kimi K3 at half the per-token price of GPT-5.6 ends up costing almost the same per completed task because it needs roughly double the tokens to get there. DeepSeek V4 Pro was the cheapest per-call option in one benchmark and the most expensive by the time ten correction rounds were counted, while the pricier frontier model finished in two passes and came out ahead. The pattern repeats enough that it stopped being a curiosity and became the study’s central rule: price-per-token tells you almost nothing, and price-per-completed-task is the only number worth trusting — and you have to measure it on your own workload, because it doesn’t transfer cleanly between task types or even between one release of a model and the next.

Where this could go

The natural next step is turning the framework into an actual cost-instrumentation layer — logging tokens-per-completed-task per workflow step, not just tokens-per-call, so the level-1/2/3 arbitrage test stops being a manual exercise and becomes a number you can check in a dashboard the way you’d check error rate or latency. For a team or an org this is the more urgent version of the same problem: whoever owns AI infrastructure spend needs a cost-per-task metric sitting next to cost-per-seat in the finance model before the agent workload is big enough to break the budget the way it broke Uber’s. The same three-level ladder scales directly to a portfolio view — which of the org’s twenty agent deployments have actually proven arbitrage at level 3, and which are quietly running on autopilot at level 1, is exactly the audit a platform or FinOps team should be able to run.