Three candidate models landed on Milkman's GPU sweep with one claim in common: they were faster and lighter than the incumbent. All three lost on the metric that mattered to the engine, and a bigger model that finally won on quality was rejected on speed. The decision rule matters more than the leaderboard.
I built an on-device interactive fiction engine in August and September 2026 — a small language model running entirely on a phone (or a desktop shell for development), driven by a deterministic TypeScript game engine that owns canon state, pacing, branching and knowledge boundaries, and only lets the model write prose and dialogue inside code-defined boundaries. 58 commits landed across 6 active days. The project runs on a phase-gated PRD, a 38-entry numbered decision log, and 19 distinct scored evals with numeric thresholds, plus a replay mode that reruns guardrail checks against recorded model responses without a fresh GPU pass. Deterministic tests reached 1,010 tests green — the highest figure recorded in the repo.
The project is parked. Possibly awaiting a better model, because I think the harness is extremely good. If I look only at the bullet points that the prompt says need to be accomplished in any given passage, they are spot on. It's only the prose that struggles to actually achieve its goals. There's also the downside of time and battery life on a phone, so maybe the product itself was poorly conceived. Two reasons, both hedged: prose quality — possibly a model problem, not a harness problem — and product fit.
I'd said I'd swap models if one won on the golden tasks. That decision landed me in a loop.
The golden-task sweep
I built Milkman in August 2026 to run that swap: a Windows/NVIDIA local-AI operating layer that installs and manages a local inference runtime, keeps a verified registry of local models, derives per-model hardware-fit settings, and runs a benchmark/eval harness ("golden tasks") to compare candidate models against an incumbent under a statistical promotion policy, with an operator web UI. 19 golden coding/doc-chat tasks, each with hidden tests, back every model-promotion decision, with 254 golden runs recorded. Milkman is in development — a real GPU candidate sweep from the UI had not yet been run when the project was handed off.
But the golden tasks were built for the wrong job. They were built for Milkman itself: coding, documentation, plain question-answering. The fiction engine cares about prose style, genre coherence, pacing and story structure — none of which the golden tasks touch. A model that wins on the coding battery can write terrible fiction. A model that loses on the coding battery might write beautiful fiction.
I ran the sweep anyway, because I had the tool and I wanted the data.
Three rejections, one reason
Three candidates showed up with better speed and smaller memory footprints than the incumbent. All three lost on prose quality in manual spot-checks against the engine's actual prompts and scenarios. The incumbent wasn't great — the whole reason the project is parked is that the prose struggles — but all three candidates were visibly worse. Faster and lighter didn't matter if the output didn't work.
I rejected all three on the same grounds: the metric they won on wasn't the metric the engine needed.
A fourth candidate that won, then lost
A fourth candidate — larger than the incumbent, slower, heavier — finally won on prose quality in the spot-checks. The prose was cleaner, the pacing was better, the dialogue felt more natural. The golden tasks didn't predict it, but the manual eval did.
I rejected it anyway. It was too slow for the product as designed. The engine already had a time-and-battery problem on a real phone; a slower model made it worse. The candidate won on the metric that mattered to the harness and lost on the metric that mattered to the product.
The rule that came out of it
The decision rule matters more than the leaderboard. A benchmark tells you what it measures, not what you need. If the benchmark doesn't measure the thing your product actually depends on, the benchmark's winner might still be the wrong model for you.
For Milkman, the fix was straightforward: the golden tasks need to match the domain the model will run in. A fiction engine needs fiction tasks. A coding agent needs coding tasks. A general-purpose assistant might need both, or neither, or a third thing entirely.
For any builder running local models, the lesson is the same: write your own evals that match your own product's success condition, and treat the public leaderboard as a starting filter, not a final answer.
The engine is still parked. The prose problem might be a model problem, or it might be a harness problem I haven't found yet. The next candidate sweep will run fiction-specific evals, not coding evals, and I'll know which one it is.