Fast, but is it smart?

Issue #18 · October 10, 2026 · ~5 min read

Welcome back to Zero to MVP Weekly. One signal, one tool, one MVP idea — five minutes, every Saturday. No link dumps. Last week I ended on routing: a cheap model by default, the expensive one only where it earns it. This week the parts for that shipped. And I spent my own week on the same question from the other side: a 125-billion-parameter model on a 16 GB card. Fast, but is it smart?

1. The signal of the week

The small question got its own price tag.

Most model calls inside a product aren't writing. They're small judgments: is this spam, which queue does it go to, does it need a human, is it worth waking the expensive model. This week three launches priced that work separately.

OpenAI's Decisions API (public beta). You send text or an image plus a question, and get a typed answer instead of prose: the probability that something is true, one option from your list, or a score on your own scale. OpenAI says it's "about 10x faster than the Responses API." It costs $0.10 per million input tokens, with no charge for output.
Claude Haiku 5.5 (Anthropic): $0.10 in, $0.50 out for prompts up to 100,000 tokens. Haiku 4.5 was $1 and $5. It's the first Haiku with an effort setting.
EmbeddingGemma 2 (Google, Apache 2.0): one small local model that puts text, code, images, video and audio into the same search index.

What to do with it: find the place in your product where a model runs on every event. Put a cheap gate in front of it, and keep the expensive path as the fallback. If you pay for Claude Max, check your account too: Anthropic says Max and Team subscribers get a monthly API credit starting this week, $100 on Max 5x and $200 on Max 20x.

One honest caveat. A probability with two decimals isn't a calibrated one. OpenAI's own guide says to set thresholds on labeled examples from your application. One developer on Hacker News ran a thousand trials: yes/no questions came out close to the true odds, but on a pick-one question, reordering the options moved the same answer from 86% to 73%. And the cheap model has last week's trap. At max effort Haiku 5.5 used about 162,000 output tokens per task in Artificial Analysis's test, more than Opus 5.5 at max. Set the effort yourself, and watch the 100,000-token line: above it, Haiku's price is five times higher.

2. My take

Most AI applications use expensive, powerful models even for simple work: classification, routing requests, making small decisions. With OpenAI's Decisions API, Claude Haiku 5.5 and small local models now available, that approach is more and more often unjustifiably expensive.

It makes much more sense to build in tiers: cheap or local models handle the simple tasks, and the powerful LLMs come in only where they're really needed. But that raises another problem: how do you judge how hard a task is, and pick the right model for it? Routing mistakes can wipe out all the savings. So what matters isn't only the cost and speed of each model, but also the quality of the system that decides which one to use.

3. Tool of the week

Strata + Qwen3.8-Flash-Next: a 125-billion-parameter model on a 16 GB card.

What it's for: running a large open model at home behind the APIs your tools already speak. Strata serves OpenAI- and Anthropic-style endpoints on localhost, so Pi, Claude Code or Codex CLI point at it by changing one address.
How it fits: it's a mixture-of-experts model, and only about 6B of its 125B parameters work on any one token. Strata keeps the experts it needs most often in VRAM and the rest in RAM, where the CPU handles part of them.
What I ran: an RTX 4060 Ti 16 GB, 128 GB of RAM, Strata built from source on Linux, the Q2 quantization, Pi as the agent. The model loads in about 30 seconds and generates 60–70 tokens per second.
Three tasks from my prompt collection:
- a page with a clock, a stopwatch and a timer: done, no plan needed;
- a pixel editor: working on the first try, in 5 minutes 20 seconds. I didn't expect that;
- a pyramid made of text characters that looks 3D when you print it and view it at an angle, a task I normally give Anthropic's and OpenAI's models: it understood the task and started in the right direction, but didn't get there.
Where it costs you: it takes the whole machine. About 36 GB of RAM (over 50 with a higher-quality version), VRAM almost full, half the CPU cores busy. This isn't a background service. You need a lot of RAM, not only a good GPU. Windows and Linux only, no Mac. And these are first impressions: I'll keep testing for a few weeks.

The video: A 125B Local Model on a 16GB GPU: Fast, But Is It Smart? · code: Niko1221/Strata

4. MVP idea of the week

A threshold workbench for decision endpoints.

Problem: a decision endpoint hands you 0.93, and the docs tell you to set thresholds on your own labeled examples. Almost nobody has a tool for that. People pick 0.8 because it looks reasonable, and find out in production what it lets through.
Solution: upload a CSV of 100–300 labeled items and your question. Get back a table of thresholds with the false positives and false negatives at each one, a stability check (the same question with the options shuffled), and the same run on a cheap general model, with the cost per thousand decisions.
For whom: developers adding a gate, a router or a moderation step to a product; small teams with a support inbox.
Minimal v1: a command-line tool that writes one HTML report. One provider, yes/no and pick-one questions only.
Demand check: publish one report on a public dataset, with the shuffle result in it, where the launch is being discussed. Count how many people ask to run it on their own data.
Why now: the endpoint is days old, general availability is promised within weeks, and one of the first things developers found was that the order of the options moves the number.

5. Useful links

OpenAI — Decisions API guide — the three question types with copy-paste examples; try the complaint-routing one on your own inbox.
Google — EmbeddingGemma 2 developer guide — which encoders to load, and how much quality each smaller vector size keeps.
Mistral — Mistral Large 4 — a trillion-parameter model in API preview; weights are promised by the end of the month, license not yet named.
Deno — Deno is joining Cloudflare — Deno Deploy shuts down in six months and runtime development ends in a year; check what you host there.
Anthropic — Investigating unintended model actions — what blocked agents did on live websites; read it before giving yours a fetch tool.

— Nick