Build Log 04 — The eval gate that caught a bad model swap

The gap

In an AI pipeline powering an application, prompt edits undergo rigorous evaluation: ten golden test pairs, deterministic assertions, LLM judge scoring, and baseline comparisons before deployment. Model swaps, however, received no such scrutiny — they shipped as simple string edits without quality verification, risking silent regressions in fabrication rates or accuracy.

What I built

A swap mode for the evaluation CLI subjects any candidate model to the same gates the pinned model already cleared. Running swap anthropic:claude-opus-5 or swap openai:gpt-5.3-codex triggers evaluation across four dimensions:

• Quality: LLM judge scores on a 1-to-5 scale; 3.5 is the minimum threshold
• Hard failures: automated rules checking (e.g. allowlisted achievement verbs); candidate cannot exceed baseline violations
• Fabrications: judge flags unsupported claims; candidate cannot exceed baseline
• Cost and latency: tracked but not yet gated

Two critical checks run: absolute floor validation against baseline, plus same-session head-to-head comparison against the live pinned model. The direct comparison matters most — a candidate could clear every threshold yet underperform uniformly against current production.

What I ruled out

External vendor review eliminated three assumptions: autonomous swapping, mandatory "maturity windows" before trusting new models, and the belief that cross-vendor review eliminates bias. Every result requires human approval before merging. One unresolved gap remains: the fabrication judge is Claude-only, introducing potential same-vendor bias when scoring OpenAI candidates. Building a cross-vendor judge awaits concrete necessity.

The first real run

Initial testing confirmed the machinery worked: swapping the pinned model for itself passed all checks. The meaningful test used gpt-4o-mini, an OpenAI candidate chosen for potential token efficiency. It failed both quality and hard-failure checks immediately, eliminating the cost consideration entirely. A cheaper model producing worse output isn't a token-efficiency win — it's just worse.

Numbers collected

• claude-sonnet-5: passed 2 of 5 runs; consistent quality deficit (3.51 vs. 4.11) paired with fewer fabrications; ~5x slower
• claude-opus-5: one run, clean pass; insufficient data for confidence
• gpt-5.3-codex: zero passes across four runs; consistently failed job-match classification despite reasonable quality and speed

Why safer models score lower

The quality rubric rewards sharp, specific recommendations; fabrication scoring penalizes any invented facts. These dimensions sometimes move inversely — cautious models hedge claims, appearing safer on fabrication but blander on utility.

Next steps

1. Reference published benchmarks before testing candidates
2. Run the full ten-pair dataset, particularly for opus-5
3. Investigate why same-session head-to-head comparison passed a candidate the pinned model clearly outperformed
4. Decide whether monthly checks become scheduled or remain manual

How to steal this pattern

1. Parameterize the model as input to the existing eval harness
2. Compare candidates head-to-head against live production, not fixed benchmarks
3. Name blind spots in subjective judges instead of hiding them
4. Test with real candidates; re-test failures before accepting results