Last week's edition ended on a question and left it standing. If you swapped the model underneath your most important workflow tomorrow, what would you have to rebuild? Call it the replacement test — the exercise that tells a leadership team which parts of its AI actually move with it and which were quietly chained to a vendor.[^7] It is a good question to sit with. It is a harder one to answer honestly, because the honest answer is not a memo. It is a run.
This edition is about the run: how to find out, before a renewal forces the decision, whether the cheaper model can actually do the job. The method here is not ours. It belongs to the practitioner whose bakeoff discipline these pages have been drawing on, and it is worth setting out plainly, because most organisations answer the cheap-model question the wrong way.[^1] They watch a demonstration, form an impression, and choose the model they were already inclined to like. A bakeoff replaces the impression with a number.
The metric that replaces the price
Start there, because it is the whole reframe. The token price — dollars per million words in and out — is the figure every board now quotes, and it is close to useless as a basis for the decision. The metric that replaces it is the cost of a finished, accepted piece of work: the total model, tool, infrastructure, and review cost, divided by the number of artifacts you actually accepted.[^1]
The distinction is where the money hides. A model priced at a fifth of the token rate looks like an obvious saving, right up until it returns an answer that needs a retry and then a human to repair it. Count the second attempt, count the reviewer's time, count the minutes a provider stalled, and the fifth-the-price engine can quietly cost more per accepted answer than the one it was meant to undercut. In one fully logged run of thirty-four tasks, twelve came back for a second attempt — a retry rate that never appears on any invoice but lands squarely on the acceptance-cost line.[^2] A public standards-body evaluation made the same point from the other direction: across seven tasks, the end-to-end cost per correctly solved problem for one cheaper model ranged from roughly half the price of a frontier rival to about forty per cent more, depending only on the work.[^3] Same model, opposite conclusions, decided entirely by the job. The token price could not have told you which.
One more line belongs on that ledger, because it separates a model that recovers from one that merely finishes eventually: record the first-try pass rate apart from the eventual one. Eighteen of twenty accepted after a single retry describes a model you can lean on; eight of twenty on the first attempt describes one you are buying latency and extra failure paths from, however respectable the tally looks once the retries are folded in.
Break your own check first
None of this works without a check — an executable test that decides whether a piece of work is acceptable — and the check is where most bakeoffs quietly fail before they begin. The discipline has one non-negotiable rule: break your own check first. Before either model runs, point the check at a file you know is bad and confirm it fails; then point it at a file you know is good and confirm it passes.[^4] Skip that step and you have no idea whether your check can reject anything at all — in which case both models come back looking fine and you pick, once again, the one you already liked. A check you have not tried to fool is not a check. It is a rubber stamp.
The reward for a check that works is that it catches the failure the eye would miss. In that same logged run, a cheaper model was asked to pull verified quotations and reported two hundred and thirteen of them; the check found that thirteen were stitched together or paraphrased closely enough to fool a skimming reader.[^2] No reviewer scanning a long document would reliably have caught that. The check did — and the model, shown exactly what was wrong, corrected the file and passed on the second attempt. The lesson is not that the cheaper model is bad at research. It is that the check, not the reviewer's confidence, is what stands between a plausible answer and an accepted one.
Read the failure, not the headline
When a run does fail, resist the single verdict that "the model was bad." A wrong answer, an unsupported claim, a missed source, a stalled provider, a check that was itself mistaken, and a routing error that quietly sent every task to the same model are six different problems with six different fixes — and only some of them are the model's fault.[^6] One real run stopped after about four and a half hours not because the model failed but because a provider's throughput did; that is a hosting result, not a verdict, and it does not carry to the next version of the model.[^6] If you cannot yet say whether the model, the harness, the provider, or the check failed, you do not have a verdict. You have a row to investigate. Collapsing all of it into "the cheap model doesn't work" is how a fixable plumbing fault gets mistaken for a capability ceiling.
Freeze the conditions, hide the names
A bakeoff only means something if both models sat the same exam. That means freezing the conditions: the same assignment, the same source files, the same tools, the same time limit, the same acceptance check, the same number of attempts, the same bar.[^4] The moment you give the struggling model a friendlier prompt, a longer deadline, or one more try, you are no longer comparing two models — you are comparing two systems, and the result tells you nothing you can act on. If you do change something after seeing a model stumble, the honest move is to log it as a change and rerun the baseline for both.
Then, wherever it is practical, hide the model names while you review.[^4] Marketing, price, and national origin all arrive with a verdict already attached, and a reviewer who knows which output came from the expensive model will find reasons to prefer it. Blind the review and the work has to win on the work. It is a small piece of theatre with a large effect on honesty.
Five to explore, twenty to trust
How many cases make a fair test? Enough to be real, not so many that you never start. The discipline runs in two stages. First, three to five genuine cases — one ordinary, one difficult, one ugly edge case, and, where it fits, one where the instructions change halfway through, to see whether the model loses the thread when a constraint shifts.[^5] If it falls apart there, stop; you have your answer cheaply. If it survives, build a replay pack of known-behaviour examples drawn from real work, including old failures, human-corrected cases, and at least one case where the right answer is for the model to say it cannot finish safely.[^5]
The threshold is worth stating precisely, because it is the difference between curiosity and commitment. For a narrow, low-risk task, five examples may be enough to keep exploring. Before you make a model the default for recurring production work, you want closer to twenty.[^5] Five tells you the model is interesting. Twenty tells you it is dependable. Confusing the two is how a promising demonstration becomes a production incident.
The standard cuts both ways
The temptation, once a check starts catching the cheaper model's mistakes, is to conclude that frontier models do not need one. The same run settles that. The expensive orchestrating model, in the course of the same job, wrote a styling rule that made a pre-order button vanish in dark mode — a clean, confident, entirely broken piece of work that the check caught exactly as it caught the paraphrased quotations.[^2] Either the cheap worker or the frontier model can be wrong, and the wrongness does not announce itself by price tag. Which is the whole point: a standard that applies only to the models you already distrust is not a standard, it is a prejudice. The reason the bakeoff is worth running is that it holds the frontier model to the same bar as the challenger, and lets the finished work — not the reputation — decide.
Your Waypoint
Before your next renewal, run one bakeoff on one real job. Break your own check against a known-bad file first, so you know it can reject. Freeze the conditions so both models sit the same exam. Hide the names while you review. Start with five cases to see whether the cheaper model is worth exploring, and build toward twenty before you trust it with anything you cannot afford to get wrong. Read each failure for what it actually was, and score the run on the cost per accepted artifact, not the price per token. None of this needs a budget cycle; it needs an afternoon and the willingness to be surprised in either direction.
And here is what you are left holding when the run is done. The bakeoff turns "can the cheap model do it?" from a feeling into a costed fact — but the checks you wrote, the replay pack you assembled, and the record of what you accepted and why do not expire when the test ends. They are the correction log the last edition argued was the real asset: the accumulated, portable record of how your organisation decides what "good" means. Run the test to answer this quarter's pricing question, and you build, almost as a by-product, the one thing no vendor can sell you and no price cut can reach.
The full reading of this week's model economics — the price war, the open-weights terms, and the acceptance discipline behind them — is in The Capability Brief — Edition 19, and the archive sits at the intelligence hub.
References
[^1]: The bakeoff discipline — the argument that whether a cheaper model can do the job is decided by a specific job, an endpoint, and an executable check rather than by price or national origin, and that the metric which replaces token price is total model, tool, infrastructure, and review cost divided by accepted artifacts — is the practitioner's method (Nate B Jones), published 27 July 2026. Carried as the argument being made and attributed as method, not asserted as a BusinessGPS finding.
[^2]: The thirty-four-task logged run — a multi-agent system in which a frontier model orchestrates while cheaper models take bounded, individually checked tasks; thirty-four tasks went out and twelve returned for a second attempt; a cheaper worker reported 213 verified quotations of which the check found 13 stitched together or paraphrased, corrected on the second attempt; and the frontier orchestrator wrote a styling rule that hid a pre-order button in dark mode, caught by the same check — is the practitioner's own run (Nate B Jones), reported with his own caution that a single low-cost run does not prove the cheapest token produced the cheapest finished work. Carried as reported.
[^3]: A US standards-body evaluation (NIST/CAISI) of one cheaper model across seven benchmarks estimated the end-to-end cost per correctly solved task at between roughly 53 per cent lower and roughly 41 per cent higher than a frontier rival, depending on the task. Reported via the practitioner analysis (Nate B Jones); carried as an attributed figure, not an independent BusinessGPS finding.
[^4]: Break-your-own-check-first, frozen conditions, and blind review — that a check must be shown to reject a known-bad file before either model runs, that both models must face identical assignment, sources, tools, time, attempts, and bar, and that model names should be hidden during review wherever practical — are the practitioner's recommendations (Nate B Jones); carried as the argument being made.
[^5]: The two-stage replay method — three to five real cases to find obvious problems fast, then a replay pack of five to twenty known-behaviour examples, with five enough to keep exploring a narrow low-risk task and closer to twenty wanted before a model becomes a recurring production default — is the practitioner's method (Nate B Jones); carried as the argument being made.
[^6]: The failure-category discipline — separating a wrong answer, an unsupported claim, a missed source, a wrong tool, a partial completion, a provider failure, a timeout, a check that was itself wrong, and a routing error, rather than collapsing every failure into "the model was bad", with the example of a run that stopped after about four and a half hours because a provider's throughput stalled, a hosting result that does not transfer to the model's next version — is the practitioner's analysis (Nate B Jones); carried as reported.
[^7]: BusinessGPS, The Replacement Test, The Way Pointer — Edition 22, 27 July 2026 (the preceding edition, which posed the replacement test and named the correction log as the durable asset).
The Way Pointer is published by BusinessGPS — the transformation capability platform helping organisations make strategic choices about what they build, what they buy, and what they own.