The eval gate that caught a bad model swap

A model swap should earn its way in the same way a prompt change does, not get a diff read and a shrug.

The gap

In the AI pipeline behind an app I'm building, every prompt-text edit already runs through a real eval harness: ten golden test pairs, deterministic assertions, an LLM judge scoring fabrication and quality, all checked against a baseline before anything ships. But swapping which model powers the pipeline (bumping to a newer Claude release, or trying an OpenAI candidate instead) got none of that rigor. It shipped as a bare string edit: change the model name, read the diff, eyeball the cost delta. A swap could silently regress fabrication rate or accuracy, and nothing would catch it before production.

What I built

I added a swap mode to the eval CLI that runs any candidate model through the same golden-set gates the pinned model already cleared. Point it at a candidate, `swap anthropic:claude-opus-5` or `swap openai:gpt-5.3-codex`, and it runs that model through the same pairs, then gates the result the same way.

Four things get measured on every run:

• Quality: the LLM judge scores accuracy and overall output on a 1-to-5 scale, and 3.5 is the floor. Below that, the candidate fails.
• Hard failures: automated rules the output has to obey (for example, using only allowlisted achievement verbs). A candidate cannot exceed the pinned baseline.
• Fabrications: the judge flags any claim the source resume doesn't support. A candidate cannot exceed the pinned baseline.
• Cost and latency per call: tracked every run, reported but not gated yet.

Check A: does the candidate clear the same absolute floor the pinned model has already proved it can hit?

Check B: a same-session, head-to-head run against the pinned model, so a candidate is judged against today's version of the pinned model, not last week's recorded baseline.

Check B is the one that matters. A candidate that clears every baseline by a hair could still be uniformly worse than what's already running: worse on every pair, just not badly enough to trip an absolute floor. Without a direct comparison, that candidate ships anyway. Noise is a separate problem with its own rule: any regression gets re-run two or three times in isolation before it counts as real.

What I ruled out

A different vendor's coding CLI reviewed the design, no stake in defending my idea, and killed three things I'd been assuming: autonomous swapping, a “wait a month before trusting a new model” maturity window, and the idea that cross-vendor review eliminates bias rather than just pointing at it. Nothing swaps or rolls back on its own. Every result ends with a human reading a verdict before anything merges. (This is the same rule the code review team from Build Log 03 runs on: a second model, no attachment to the first, permission to disagree.)

The gap the review left standing: the judge that scores fabrication and quality is a Claude model that has only ever judged Claude's own output. Scoring an OpenAI candidate with it carries an unproven same-vendor bias risk. Building a second, cross-vendor judge is real work I don't have a concrete need for yet. So the swap command prints a caveat whenever the candidate isn't Claude: the automated checks are the signal, the judge's score is directional.

The first real run

First run was a sanity check: swap the pinned model for itself, across two pairs, see if the machinery works at all. It passed on every count, as it should have.

Second run was the real test: gpt-4o-mini, an OpenAI candidate, on a single pair. The kind of cheap, small model you'd only consider for one reason: token efficiency. If it held up on quality, it could mean a cost saving on every tailor action this pipeline runs. It didn't hold up. It failed the quality and hard-failure checks outright, and failing there means the cost question never gets asked at all. A cheaper model producing worse output isn't a token-efficiency win. It's just worse.

Numbers I don't trust yet

A second run of the same matchup happened hours later, as part of a verification pass rather than a deliberate re-run under my own rule. Two runs, same candidate (gpt-4o-mini) against the same pinned model (claude-sonnet-4-6):

This was one incidental repeat, not the two-or-three-run confirmation my rule calls for. Even so, it's informative. Check A failed the candidate both times, but not the same way: the first run tripped hard failures and quality together, the second tripped only quality, which means that run's failure rested entirely on the judge's score, the exact signal I'd just told the CLI to print a caveat about for non-Claude candidates. Check B is the surprising row: it passed the second run even though the pinned model beat the candidate on every other judge axis that session. I don't know why yet. I wouldn't know any of this from one run.

Testing the next candidates

The obvious next candidates were the new Claude 5 family (sonnet-5, opus-5) and OpenAI's current codex-tier model. Testing them broke something before it proved anything. Claude 5 models run extended internal reasoning by default, with no setting to turn it off, and that reasoning showed up as a hidden content block my own code had never been built to expect. Every call either timed out or came back silently truncated: the hidden reasoning ate the entire token budget on all 6 sonnet-5 calls and both opus-5 calls, leaving zero visible output. Not a model problem. A gap in my own harness, closed by scanning the full response instead of assuming the first block, and refusing to trust a truncated one.

With the harness fixed, three candidates. Each has its own sample size, none the full ten-pair set yet, and the pair sets differ by model (Claude 5 on pairs 1 and 7, Codex on pairs 1 and 4), so the pinned model's own mean below moves a little by session.

Read the table by column, not by row. Sonnet-5 trades quality for fewer fabrications and pays for it in latency. Opus-5 looks clean but one run is a coin flip. Codex is fast and mostly on par on quality, but wrong on the classification every single time.

Sonnet-5 is the one with real signal behind it: five runs, a consistent pattern, lower quality every time, fewer fabrications every time, four to six times slower every time. Opus-5 passed its one run clean, but one run is a coin flip. Different model tier, and it deserves its own real test. Codex is the cleanest result of the three. It never once got the job-match classification right across four runs. That isn't noise, it's the same wrong answer four times. Everything else about it (quality, fabrications, speed) looks as good as or better than what's in production.

Why the safer model scores worse

Here's what “quality” measures, straight from the judge's rubric: would a hiring manager find these edits useful and well targeted? Sharp and high-impact scores a 5, vague and generic scores a 1. Fabrication is a separate, stricter axis: a single invented fact fails the whole audit, no partial credit. Those two things can move in opposite directions. A model willing to assert something specific is taking a bigger swing. The swing that lands reads as sharp; the same swing that misses reads as a fabrication. A model that hedges and won't sharpen a vague line into a specific one is safer on fabrication and worse on quality, almost by construction.

That's my actual hypothesis for sonnet-5: the extended reasoning it runs by default may be making it more conservative about what it's willing to claim, which shows up as fewer fabrications and blander, less-targeted edits. I haven't confirmed the mechanism, it's a read of what the rubric rewards, not a traced cause. Confirming it would mean reading actual sonnet-5 output next to the pinned model's, card by card, not just comparing the scores.

None of this is about general capability. Published benchmarks measure broad reasoning, coding, instruction-following. They don't test this pipeline's specific rules: an exact classification label, a strict no-fabrication quoting constraint, a latency budget for a feature a real person is waiting on. A model can out-benchmark the one already in production on every general measure and still lose the only test that actually matters for this job.

Next steps

The mechanism proved itself on the first real thing it was pointed at: it ran a live candidate, produced a verdict, and the verdict held up under a second, independent look. It's since merged, as PR #59. What comes after:

1. Start referencing published cost and performance benchmarks before picking a candidate to test at all. Right now I go straight to running my own eval on whatever candidate occurs to me, with no outside filter first. A benchmark won't tell me if a model works for this pipeline. It would stop me spending a test run on a candidate that was never a serious contender.
2. Run the full ten-pair set, not the one- and two-pair samples tested so far. Opus-5 especially.
3. Figure out why Check B passed a candidate the pinned model clearly beat on every other axis. It's the check I'm relying on for exactly this kind of judgment call.
4. Decide whether “roughly monthly” becomes an actual scheduled check or stays a manual command I remember to run. Right now it's the second one by default, not by decision.

Only after those hold up: whether this same discipline is worth extending to the pipeline's other AI calls, and whether any part of it is worth automating. That case would go through the same review this design did.

How to steal this pattern

Four rules, in order:

1. Parameterize the model as an input to your existing eval harness. Don't build a separate system for swaps.
2. Compare every candidate head-to-head against what's live right now, not just against a fixed bar.
3. If a judge model scores something subjective, name its blind spot instead of hiding it.
4. Test against a real candidate before you trust the design. Re-test any failure before you trust the failure.

A cheaper model producing worse output isn't a token-efficiency win. It's just worse.