The problem

Everyone benchmarks AI coding tools by comparing model output on the same prompt, and everyone concludes the wrong thing from it. When a build goes badly, the instinct is to blame the model — swap Sonnet for GPT-5, upgrade to the next release, try a different agent. I wanted to know if that instinct was actually right, because if it’s wrong, a lot of people are optimizing the wrong variable.

This is the same mistake I see teams make when they pick a company-wide AI coding standard: they run a model bake-off, pick the “smartest” one, and roll it out — then get inconsistent results because the actual failure mode was never about model intelligence. If the plan is the real bottleneck, then no model upgrade fixes a team that skips specification, and every dollar spent chasing the newest frontier model is being spent on the wrong lever.

The approach

This isn’t a piece of software — it’s a structured evidence base I built to stress-test one thesis, organized so the argument is traceable back to source instead of asserted. The core artifact is a head-to-head test: the same nontrivial feature (a canvas sidebar with image loading, an agent selector, streaming chat, a file uploader, and resizable layouts) run through different tool-and-model combinations — Cursor with Sonnet 4.5 thinking, Cursor with GPT-5 High, Droid with Sonnet 4.5. Each combination gets graded, not on how clever the final code looks, but on one question: could I hand this plan to a different model and still get a good result? That’s the portability test — a plan that only works in the hands of the model that wrote it isn’t a plan, it’s a lucky guess.

The result: Cursor + Sonnet 4.5 thinking produced a plan specific enough — exact APIs, exact existing files to touch — that implementation needed zero correction passes. Droid’s plan graded D/C-, thin and unspecific. To confirm it was the plan and not the model, I took that same weak Droid plan and handed it to a completely different model, GLM4.6. It still failed. Same weak plan, different executor, same bad outcome — which is the actual proof, not just an anecdote.

Around that core experiment sits a running corpus of roughly 180 curated findings pulled from real sources — AI Engineer conference talks, Anthropic and OpenAI engineering posts, comparative reviews from practitioners running 4-10 coding agents in parallel — each one written as a standalone, source-cited note (what the finding is, why the mechanism holds, and a link back to the original talk, article, or video). It covers adjacent territory a team would actually need: how to tell a “delegation” problem (hand off an isolated task, walk away) from a “coordination” problem (agents need to share state across your existing tools) before picking an agent architecture at all; why long AI sessions degrade (“context rot”) and the fix of treating each session as a stateless function call with the codebase as the memory; why an actionability gate has to sit in front of any autonomous coding agent or you drown in noisy, plausible-looking but wrong PRs.

What I learned

The clean result was that degrading the plan hurt more than degrading the model. A weak plan handed to a different, still-capable model didn’t recover — it failed the same way. That inverts the default assumption that model choice is the lever worth pulling. What actually separated the A-grade run from the D-grade run wasn’t cleverness, it was concreteness: the good plan named the exact APIs and the exact files already in the codebase, so there was nothing left to guess at implementation time. A good plan also gives you recoverable intermediate state — if step 4 breaks, you know what step 4 was supposed to do and can retry it, versus a bad plan that hands you opaque code you can only debug by reading everything.

Where this could go

The natural next step is turning the grading criterion into an actual checklist — a short rubric a team applies to any AI-generated plan before letting an agent touch code: does it name the exact files and APIs it will touch, does it show it resolved the ambiguous product decisions instead of silently assuming, could a different model execute it and get the same result. That’s cheap insurance, structurally identical to a code review gate, just moved one step earlier in the pipeline — before implementation, not after.

At team or org scale this generalizes past coding tools entirely: it’s an argument for reviewing the plan an agent produces before the agent produces the deliverable, whether that deliverable is code, a contract review, or a generated report — and for standardizing that review the same way code review got standardized, instead of treating “trust the smartest model” as the risk-management strategy.