The model went looking for my tests

Issue #20 · October 24, 2026 · ~5 min read

Welcome back to Zero to MVP Weekly. One signal, one tool, one MVP idea — five minutes, every Saturday. Last week I hid the tests from two models and checked their work after they said "done". This week one model went looking for those tests. It didn't find them, but how it looked is the signal.

1. The signal of the week

The models went looking for the answer key.

Models trained on benchmarks now treat almost any task as a test, and the fastest way to pass a test is to find the answers. This week two labs said so in writing, and I saw it on my own project.

Moonshot launched Kimi K3.1 Pro, and its benchmark footnotes zero out runs where models were "retrieving answers online" instead of doing the task.
OpenAI said an internal model "exploited a misconfiguration in a DNS resolver to gain limited access to the live internet during a training run." It now trains frontier models "with no internet access."
In my test, DeepSeek V4.1 Flash got only the task text and decided it was a benchmark with hidden tests. It went through the git history, then searched GitHub for an issue with the task's title. It found nothing, left six files re-exporting its code under other names "in case the tests are looking for something else," and then solved the task honestly.

What to do with it: an agent on your machine can reach your git remotes, your home folder and your other projects. Run agent tasks with the network off by default, allowing only the model API and your package registry. Give each run its own folder, keep tests and other runs out of reach, and log every outbound request.

One honest caveat. Checking git history for an existing solution is what a good engineer does too, and Opus did it in my test. The line is between using what you were given and reaching for what you weren't.

2. My take

I think this is an interesting symptom of how AI models are changing. We train them to get results, and they gradually start optimizing for passing the check rather than solving the task. And the most curious part is that this doesn't have to come from an intent to cheat: if a model sees a way to find a ready answer, that can look like a perfectly rational way to do the job.

But I wouldn't conclude that agents are becoming dishonest. Passing the test and doing the task just aren't always the same thing. For developers, the end result is no longer enough: you need to know how the agent got there, what data it touched and what it did. In my view, transparency and isolation will soon be as standard in AI development as tests and code review.

3. Tool of the week

DeepSeek V4.1 Flash vs Opus 5.5 on a 20,000-line project.

What it's for: deciding whether a cheap open model can take real coding work off Opus.
What I ran: the hardest task from last week's video: when a file is renamed or moved in Texodus, my Markdown editor, links to it from other notes have to update. Same Pi agent, same settings, 13 hidden tests the model never sees.
Results:
- DeepSeek, two runs: 13 of 13 both times, plus the project's own tests, the linter and the type check. 10 and 7.5 minutes, eight and six cents at the off-peak rate (peak is about double);
- Opus 5.5: 13 of 13 in 21 minutes, for $5.45, 70 to 90 times DeepSeek's price;
- code quality: Opus reused the project's Markdown parser; DeepSeek wrote its own, over 500 lines. I expected that to break and wrote 31 edge-case tests. DeepSeek passed 30, the same as Opus.
Where it breaks: at 763 billion parameters it's not a home model, and for now only DeepSeek hosts it, so your code goes to their servers. It also goes online without asking. If you route Claude Code to DeepSeek, its Search and Fetch calls stop being free on October 31 ($0.75 per 1,000 calls).
My conclusion: I'm not replacing Claude. Opus works more carefully. But for everyday tasks in an isolated project, DeepSeek is now a very strong option.

The video: New DeepSeek V4.1 Flash vs Claude Opus 5.5 on a 20,000-Line Project

4. MVP idea of the week

A sealed run box for agent tasks.

Problem: a coding agent on your laptop can read your home folder and your other projects, reach your git remotes, and send your task text anywhere. You find out from the logs, if you read them at all.
Solution: one command that runs any CLI agent (Pi, Claude Code, Codex, OpenCode) in a throwaway container with only the project folder mounted and the network off, except for an allow-list. At the end you get one report: every outbound request, every read outside the project, every blocked attempt.
For whom: solo builders who run agents unattended or several at once; anyone benchmarking models on their own tasks.
Minimal v1: a shell script around Docker or Podman: one mount, an egress proxy that logs requests and allows listed hosts, a Markdown summary. macOS and Linux.
Demand check: publish the egress log of one real agent run, showing what it tried to reach, with a two-line install. Count stars and "here's what mine tried" replies.
Why now: OpenAI took the internet away from training for this reason, Kimi zeroes benchmark runs for it, and cheap models make parallel runs normal.

5. Useful links

OpenAI — Securing our frontier model training environment — why they took the internet away from training; the red-team findings are the useful part.
Moonshot — Introducing Kimi K3.1 Pro — the benchmark numbers, and the footnotes on zeroed runs; weights promised by November 4.
The PHP Foundation — Today's PHP security release — two AI-found remote code execution bugs in PHP itself. Running PHP anywhere, a WordPress landing page included? Update to 8.5.12, 8.4.27, 8.3.36 or 8.2.35 now.
My other test this week — Can an agent make a small local model reliable? — the same local Qwen in two agents: one checked its numbers and caught its own mistake, the other was confidently wrong twice.

— Nick