Most people and most teams run every AI task through whatever model they happen to have open — the same model drafts code, debugs a stack trace, writes a research summary, and reviews a PR. That’s expensive and it’s also worse: models have converged toward similar peak capability but diverged hard in what they’re actually good at and what they cost per token. A frontier model is overkill (and wasteful) for classification or summarization; a cheap model quietly fails at debugging and multi-step reasoning. At company scale this is the same problem as a procurement/FinOps question: teams paying frontier-model prices for every seat because nobody built a policy for routing cheap work to cheap models, then wondering why the AI line item outpaces the value delivered.

I started by mining 57 sourced ideas (from articles and YouTube transcripts on model routing, cost, and evals) and cutting them three separate ways — by mechanism (planning/implementation split, right-sizing per sub-task, multi-agent economics), by decision moment (“which model for this task?”, “how much should I spend?”, “can I trust this benchmark?”), and by action type (set up / build / pick / measure / avoid / anticipate). That triple-pass structure is itself the methodology: the same 57 ideas read differently depending on what question you’re standing in front of.

From there I turned the “Build” theme into five committed architecture decisions (ADRs) and actually built the thing: msbt-router, a provider-agnostic CLI auto-router. It has real commands — `msbt run “<task description>”` classifies free-form text and dispatches it to the right model automatically; `–provider`/`–model` let you override the routing outright; `msbt policy show` prints the current tier-to-model table; `msbt policy sync` rebuilds the SQLite mirror from the git-versioned policy.yaml; `msbt providers` lists which provider API keys are configured; `msbt log` tails recent routing decisions with cost and latency; `msbt init` bootstraps the local state directory. Under the hood: a two-tier classifier (cheap-model classify, escalate to a mid-tier model on low confidence), a closed vocabulary of ~17 task types (debugging, code_review, research, summarization, translation, orchestration, etc.), a five-tier cost policy (cheap/mid/advanced/frontier/frontier+) mapped per provider through LiteLLM’s OpenAI-compatible wire format, and every decision logged to a JSONL file plus SQLite for later analysis. The policy itself lives in git as the source of truth — a human edits policy.yaml, an eval pipeline can propose changes but never applies them directly.

Concretely, what this gives someone: point a task at the router instead of a specific model, and it tells you what type of work it thinks this is, what tier it picked, which model/provider it dispatched to, whether it had to escalate, and what it cost — before you’ve had to make that call yourself.

The clearest lesson is that “which model should I use” is the wrong question — the right one is “which axis of difficulty is this task on,” and building that map is a skill you get better at over time, not a one-time config. The second lesson, which the build stage surfaced hard, is that a policy hand-tuned from reasoning is not the same as a policy backed by evidence: I shipped a v1 routing table explicitly labeled as a guess, with unverified model IDs for several providers, specifically so it wouldn’t get mistaken for something validated. The golden-set eval track (grounded in real bugs mined from my own repos, not invented test cases) exists to close that gap but isn’t built yet — which is itself the honest status, not a gloss-over.

The unfinished pieces are concrete: wire real per-provider model IDs and confirm they actually work, turn the 21 mined golden-set task candidates into real eval files, and decide whether the eval pipeline runs on a schedule or stays a manual trigger. Past that, the same pattern scales past a personal CLI: an engineering org running the same router as a shared service in front of its AI coding tools would get a live, auditable answer to “how much are we spending per task type, and is the expensive model actually earning its premium over the cheap one” — the exact FinOps visibility that’s missing when everyone just picks their favorite model by habit. The eval-pipeline design (golden set plus human-approved policy changes, never auto-applied) is deliberately built so that kind of governance layer is where this goes next, not a personal-tool ceiling.