Build Log 03 — I built my AI agents a code review team

I built a team. A planner, an engineer, an adversarial code reviewer running on a deliberately different model than the engineer, a design critic, a brand critic, and a QA agent whose only job is to try to prove the work is broken. None of them are real. All of them are Claude, wearing different job descriptions, reviewing each other's work before anything reaches me.

I'd read the core idea in the AI-agent discussions online: never let a model grade its own work, force a second one to look at it cold. It sounded like a good rule. I wanted to see if it actually held up. It did, more than I expected. It also revealed the exact spot where the whole idea has a hole in it. Both parts are the point of this post.

Why build a fake org at all

I'm a product manager, not an engineer. I can direct a build, read a diff, tell you if a feature is the right thing to ship. I can't tell you if a scroll animation will render invisible the moment hydration fails. For most of this year that gap didn't matter much: ask an AI coding tool for something, read the result, ship it if it looked right. "Looks right" and "is right" turned out to be different claims, and I had no way to tell them apart on my own.

That's the actual stake here, not a hobby project. I'm the only quality control this product has, and I was checking things I'm not qualified to check.

The fix wasn't learning to code faster. It was building a process I could trust, the same reason real engineering orgs don't let one person write and approve their own code. So I built the org instead of hiring one.

How the pipeline works

Every task runs the same sequence, sized to the change. A typo fix doesn't get the full panel. A new feature does.

• A planner writes an implementation spec, and I approve it before any code exists.
• An engineer builds it.
• A code reviewer, pinned to a different model than the engineer on purpose, reviews the diff blind, with no access to the engineer's reasoning.
• A design critic and a brand critic weigh in on anything user-facing.
• A QA agent tries to prove the acceptance criteria are false.
• Only then does anything become a draft pull request. Never auto-merged, never auto-deployed, always my call.

What worked: decorrelation

Refusing to let the same model review its own output was the best single decision in the setup, and it matters more than it sounds. This product's entire pitch is that agents won't invent things on your behalf. A drafting pass that invents a detail and then approves its own draft isn't a rounding error for a product like that. It's the whole promise breaking quietly in the background.

This week that rule caught two real instances. A piece of marketing copy crossed the line into an absolute "we never invent facts" claim, when the honest version is narrower: nothing enters without your approval. Worse, a fabricated statistic made it onto a public page, written by the same agent that reviewed everything around it, which never noticed it had done so.

Neither would have been caught by the agent that wrote them rereading its own draft. A second agent caught both, with no attachment to the first draft and permission to disagree until proven wrong.

What didn't: a bug I could see in five seconds

Here's the honest part. The same pipeline, planner and engineer and reviewer and QA, passed a landing page rebuild through every stage clean. Then I opened the page myself, and the navigation bar was visibly broken: no layout, a transparent background where there should have been a solid one, and a "See how it works" link that pointed nowhere.

It got worse when I checked how long the bug had been live. It traced to a commit weeks old. Nobody had caught it. Not a person, not any pass of the pipeline.

The nav bug itself I'll grant some slack on: a variable that crosses a framework boundary, reads correctly in the source, and only breaks in the browser. That's a real blind spot, not laziness. The dead link is the one that stings. When I checked what the QA agent had done for that step, it had confirmed the link "worked" by reading its href attribute off the page. The destination was written correctly in the code, so the check passed. It never clicked it, even though it clicked plenty of other things in the same run. Where a link points and confirming that clicking it does anything are two different claims, and on this one check the pipeline had quietly treated them as the same.

What the retro fixed

This is where mimicking a real SDLC actually earned its keep. Not the pipeline itself, but the retro after it. Three rules got written down. Not three bugs quietly patched.

1. Code review now checks for that specific framework gotcha by name, because "the diff looks fine" wasn't going to catch it twice.
2. A design review that can't actually render the page is a blocking problem, not a "no findings" pass. A review that silently downgrades from looking at pixels to reading code was the real gap.
3. QA's rules now say plainly: an attribute existing is not proof a behavior works. Click it. Compare before and after.

None of those three exist if the bug just gets patched and forgotten. The system didn't get smarter because I built it well the first time. It got smarter because it has a mechanism for turning "we missed this" into a rule instead of a one-off apology. That mechanism is the whole reason to bother mimicking an SDLC instead of just prompting harder.

What this actually costs

The token cost is real: tens of thousands to well over a hundred thousand tokens per review pass, a dozen-plus agent calls before a real feature ships. That's heavy.

To me, it's worth it because of what it replaces. Decorrelation isn't just more checks stacked on top of each other, it's different ways of looking at the same code: a model reviewing blind catches what the model that wrote it can't see in its own work. And the pipeline doesn't stop at flagging the problem, it goes and fixes it. I'm not the one jumping in to diagnose a broken component or figure out why a link won't scroll. I approve a spec, I read a report, I merge it or I don't. The token spend is buying back the hours I'd otherwise spend being the bottleneck between "something's wrong" and "something's fixed."

I don't have a clean number proving the pipeline pays for itself in dollars. What I have is two real bugs caught before users saw them, without me being the one who had to catch them. That's the trade I'm making.

Where to start

1. Never let one model review its own work. A diff review from a genuinely separate pass, with no visibility into the first one's reasoning, is the single highest-leverage move.
2. The moment something slips through, write down the rule, not the fix. A bug fixed once happens again in a different shape. A named rule survives past the bug that taught it to you.
3. "The code is correct" and "the behavior works" are separate claims, especially for anything with a UI. If a check can pass without something actually clicking, it isn't verification yet.
4. Budget for review to cost more than you expect. Catching things early is the plan, not a flaw in it.

The pipeline didn't get smarter because I built it well the first time. It got smarter because every miss became a rule.