The day a model announcement drops, takes range from “this changes everything” to “completely overhyped.” After a few cycles of this, the lesson is to wait for someone who actually ran the thing against real tasks and wrote down what they found.
What a recent blind evaluation of GPT-5.4 turned up didn’t fit neatly into either camp. The same model that surpassed human performance on desktop navigation — 75 percent on the OSWorld benchmark, clearing the human baseline of 72.4 percent — also produced a carwash recommendation that any five-year-old would get right before it did. Both findings came from the same evaluation. Both are true.
That tension is the most useful place to start.
The toggle most users will never touch
Before anything else about GPT-5.4, there is one thing any evaluator needs to understand: there is a thinking mode, and there is an auto mode — the default for most users. These are not refinements of the same product. The evaluation in question measured an accuracy gap of two to two-and-a-half points on a five-point scale between the two modes. Not graceful degradation — a genuinely different level of output.
Auto mode errors included citing a research result from 2020 that has since been superseded twice, estimating a company’s annual revenue at roughly one-third of its actual figure, and naming 2024 Nobel Prize winners in response to a question about 2025. The model tagged these at medium confidence. Its self-reflection noted the errors without revising those confidence ratings.
The uncomfortable implication: the version of GPT-5.4 that most ChatGPT users will encounter is the weaker one. There is no warning label in the interface.
The carwash problem
Even in thinking mode, a consistent failure pattern showed up across multiple tasks. The question was simple: the carwash is 100 meters away — walk or drive? GPT-5.4 produced a thoughtful, thorough essay weighing the merits of each option. Claude Opus 4.6 wrote seven words: “Drive. You need the car at the carwash.”
The same pattern appeared elsewhere. In a schema migration task (generating instructions for restructuring a database), GPT-5.4 produced nearly 400 undifferentiated flags with no categorization or priority ranking. Claude produced 19 you could act on immediately. In a database cleanup task, test records — including a 25,000-dollar order attributed to “Test Customer” and entries for recognizably fictional names — made it into the production database. A human scrolling the output for thirty seconds would have caught them.
The failure mode is consistent. GPT-5.4 builds technically impressive infrastructure for the task it was handed, without stopping to ask whether that task was the right one to solve.
Where it genuinely wins
The other side deserves acknowledgment, because the evaluation in question is. In thinking mode, GPT-5.4 is excellent at quantitative modeling — the kind of structured numerical work that rewards statistical rigor over voice or judgment. A sports analytics workbook it produced included Pythagorean win expectation and Poisson-binomial distribution modeling (methods for forecasting outcomes from historical performance data), along with an unprompted section cataloguing its own assumptions and shortcuts. That kind of intellectual honesty is rarer than it should be.
It also outperformed Claude on file processing breadth: 99.1 percent coverage across diverse file types, against 75 percent. Claude silently skipped spreadsheet files rather than installing the library needed to read them — a judgment call Claude got wrong.
The agentic signals are real too. A 47 percent reduction in token usage (the amount of text the model has to process) from a new approach to tool discovery is a meaningful infrastructure improvement, not a benchmark footnote.
What to take from this
The headline finding — “GPT-5.4 surpasses human performance” — is true. So is “GPT-5.4 reasoned incorrectly about a trivial everyday question.” Neither one captures what’s actually useful to know.
What’s useful: these models have genuine strengths that don’t overlap. GPT-5.4 in thinking mode belongs on tasks that reward completeness, statistical rigor, and breadth across file types and tools. Claude belongs on tasks that reward judgment, prose with voice, speed, and the ability to hold competing ideas in tension without defaulting to exhaustive execution.
The skill that will matter more and more as the models keep improving is not knowing which model is better. It’s learning to match the task to the model’s actual strengths, and to recognize the failure mode when you’ve got the wrong match.
When an AI output looks thorough but turns out to be unusable, the right question is: did the model understand the actual problem, or did it just execute the task that was handed to it? That question is worth asking before you write the prompt — not after you get the output.
Text summarized and optimized using Anthropic’s models and reviewed by a human.