"Done" is a claim

Welcome back to Zero to MVP Weekly. One signal, one tool, one MVP idea — five minutes, every Saturday. Last week was about which jobs a cheap model can be trusted with. This week the question moved one step later: when an agent says it's done, who checks? Anthropic published its answer. I published mine, with tests the models never saw.

1. The signal of the week

Review is the new bottleneck.

Coding agents write more code than ever. This week two sources showed where it piles up: at review.

• Anthropic's Claude Code team described how it reviews code that Claude now mostly writes. From June to September, merged pull requests more than doubled "while the share that had to be reverted fell by more than 40 percent." The first rule comes before any review: "ask it to show the output instead of saying it's done." After that, people approve by risk, and "an agent's approval never counts as a person's."
• A Harvard working paper (Fiona Chen and James Stratton, 718 firms) measured the other side. After firms adopted coding agents, lines of code rose 30%, but resolved Jira issues and epics showed no significant change. The time to review a pull request rose 49%.

What to do with it: make the agent prove "done": test output, not a summary. Review in a fresh session, not the one that wrote the code. And cap the automated rounds. One of Anthropic's pull requests went through "more than fifty rounds of review and fixes overnight." Their fix: after three rounds, step back and reconsider the approach.

One honest caveat. Anthropic's numbers are internal, and its process was built for a large team. The paper uses Jira tickets as a proxy for shipped work and can't rule out a gain of up to 12%.

2. My take

In my view, this is one of the main problems in AI development right now. Agents generate code faster than developers can review it. But more code doesn't mean more useful functionality. Lately I've also noticed that almost all the time saved goes into review and fixes, and the real productivity gain turns out to be minimal.

I think the solution isn't just automating code review with another agent, but changing the development process itself: small tasks, verifiable results, automated tests, and human control over critical decisions. The main metric isn't the amount of code written, but the number of features that reach end users.

3. Tool of the week

Haiku 5.5 vs a local Qwen 3.8 27B, with tests the models never saw.

• What it's for: deciding which jobs go to a cheap cloud model and which can stay on your own GPU.
• What I ran: the Pi agent at maximum reasoning. Haiku 5.5 through the API; Qwen 3.8 27B in Q4 on llama.cpp, on a 32 GB Radeon Pro W7800. I wrote the tests in advance and ran them only after each model said it was done.
• Results:
  • a Python utility and tax deductions from a folder of financial documents: both models passed every check. Haiku took minutes and cents; Qwen took several times longer, but the documents never left my computer;
  • a word-count bug in Texodus, my Markdown editor: Haiku 13 of 13, Qwen 12 of 13;
  • updating links when a file is renamed: Haiku passed all 13 hidden tests in 22 minutes. Qwen worked for almost two hours, its own tests passed, and it reported the task done. My hidden tests: 9 of 13.
• Where it costs you: that link task cost Haiku $1.44 instead of about 32 cents, because 58 requests crossed the 100,000-token line I warned about last week.
• My conclusion: big changes to a large project go to the cloud, for now. Personal documents and small, clearly specified tasks can stay local. Price alone no longer justifies a graphics card. Privacy does.

The video: Claude Haiku 5.5 vs Local Qwen 3.8 27B: Is Local AI Still Worth It?

4. MVP idea of the week

Hidden acceptance tests for agent tasks.

• Problem: an agent's own tests describe what it built, not what you asked for. They pass, the linter passes, the agent says "done", and a core feature is broken.
• Solution: a small runner that keeps acceptance checks outside the agent's workspace. You write them with the task, or a second model drafts them from the spec in a fresh session. The agent never sees them. They run when it reports done, and you get one report: hidden checks, existing tests, build, diff size, tokens and cost.
• For whom: solo builders and small teams running Claude Code, Codex or Pi on real repos, especially overnight.
• Minimal v1: a command-line tool that copies an acceptance/ folder into the worktree after the agent exits, runs it, and writes one Markdown report. One language, one agent.
• Demand check: publish one task with its hidden tests and a short write-up of what two models did with it. Count how many people run it against their own agent and ask for more.
• Why now: Anthropic's process now starts with "show the output instead of saying it's done", the Harvard paper puts the bottleneck at review, and cheap models make running the same task twice affordable.

5. Useful links

• Anthropic — How the Claude Code team does code review — the risk tiers and the three-round cap; both work for a team of one.
• Chen & Stratton — Bottlenecks in Software Production (PDF) — the 718-firm study; the code-review section is the part to read.
• Google — Introducing Gemini 4 Argon — $2/$10 per million tokens until November 30, then double; measure it per task before then.
• My other test this week — Can a 1B AI model actually see? — a tiny vision model on 29 tasks: it read my handwriting, then missed glasses vanishing from a table.

— Nick

P.S. Outside this newsletter, I help startups and companies build things: MVPs, AI integrations, local or cloud LLM deployments. If you're working on something like that and could use a hand, write to me at nick@blokhin.us.