For a long time, the standard advice when choosing a language model was: check the leaderboard, pick the top performer, ship. That advice has aged poorly. As the model ecosystem has widened — closed frontier models, open-weight alternatives, specialist architectures — the gap between a model’s benchmark score and its performance on any specific task has become the central practical problem.
A recent hands-on workshop made this concrete in a way that benchmarks rarely do. Four frontier models — GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 — were run through six real scenarios side by side: basic text reasoning, retrieval-augmented generation (RAG — a technique where the model answers from provided documents rather than from memory), image understanding, tool calling, agent loops, and evaluation. The results show why swapping models without testing is a bet most teams should not take.
What held up, and what didn’t
The most durable finding is that failure modes differ by capability, not just by degree. DeepSeek V4 Flash, for instance, is text-only — no image input at all. That is not a tuning problem; no prompt engineering fixes it. Similarly, RAG grounding (staying within the provided documents rather than drifting into the model’s trained knowledge) varied in ways that tracked rough model scale: the smaller or weaker the model, the more likely it was to answer from weights when the correct answer was “that information is not in the sources.” Mistral repeated information it was not given, even after explicit prompting to stay grounded.
Tool calling — where the model must translate a natural-language request into a structured function call — looked uniform on the surface: all four models called the right function. The differences emerged in argument quality and efficiency. In an agent loop (a sequence of tool calls to complete a multi-step task), GPT-5.5 called two tools in parallel on the first turn; Mistral forgot two of the four required tools entirely. Same task, same instruction, different behaviour.
The cost side of efficiency
Tool call efficiency is worth dwelling on, because it tends to be invisible until it shows up in the bill. A math sub-problem that GPT-5.5 solved in one tool call took DeepSeek four. The answer was identical. The difference is latency and cost — multiplied across every user session in production. For most teams this is not a dealbreaker, but it is the kind of thing that benchmark tables do not surface and that only shows up when the model is running against an actual task.
Fixing failures before switching models
The workshop’s most transferable lesson is that the first response to a failure should usually be prompt adjustment, not a model swap. A chain-of-thought prompt (telling the model to break a problem into explicit steps before answering) recovered Mistral on a basic counting task. Clarifying parameter descriptions in the function’s JSON schema fixed spurious attendees in calendar-event tool calls. These fixes are cheap and portable — they improve any model’s performance on the scenario, not just one.
DSPy (a framework that, instead of requiring a hand-written prompt, takes a specification of what the output must satisfy and searches for a prompt that meets it) takes this further. Given a target model and a set of constraints, DSPy’s optimizer can find a system prompt tailored to that model — enabling teams to use cheaper or on-premises models without hand-tuning prompts from scratch. Dropbox and Shopify have used this approach to migrate off large hosted models to smaller ones. The economics of model selection look different once prompt optimization is in the toolkit.
Evaluation as a practice
What makes this kind of comparison reproducible is evaluation infrastructure — a way to check model behaviour against known-correct outputs at scale, not just by eyeballing a few responses. The workshop covered three approaches: string matching (fast, good for deterministic outputs like tool call arguments), LLM-as-a-judge (a second model scores the first model’s output — requires calibration against human labels to be trustworthy), and specification-driven test generation (describe the desired behaviour in a YAML file; a tool generates test cases automatically).
The practical recommendation from the last approach applies broadly: avoid generic evaluators (fluency, coherence) in favour of custom ones built around the specific task. A generic coherence score tells almost nothing about whether a RAG system stayed within its sources.
Where this points
The model landscape will continue to fragment — more providers, more open-weight releases, more specialist options. The teams that handle this well will not be the ones who track benchmarks most closely; they will be the ones who can set up a scenario, swap a model in, run an eval, and read the result in under an hour. That workflow is becoming a basic engineering competency, not a research exercise.
Text summarized and optimized using Anthropic’s models and reviewed by a human.