I shipped the first six Android builds of an on-device interactive fiction engine without ever checking that the evals were testing what those builds actually ran. They weren't. The phone runtime silently filled in its own sampler defaults wherever my engine didn't set a field explicitly, so every eval battery that supposedly validated those builds was testing different settings than the app on my phone used.
I found it because the app never matched the behavior the evals recorded. On the phone, the Director, keeper and character roles ran with no repeat penalty, and every call carried a min-p value, settings no eval had ever explicitly measured. And every eval had passed.
The problem wasn't that the evals were bad. The evals were checking the things they were supposed to check. They all passed, and they passed correctly for the configuration they saw. That configuration just wasn't the one any user was running.
The fix came down to one new rule in the decision log: every sampler field gets set explicitly on every call. Decision D-038 made it binding; build 7 was the first build whose sampling actually matched the eval batteries. Up to six shipped builds ran settings no eval had validated.
The mechanic that breaks silently
Most model runtimes accept a sampling config as a dictionary or a JSON object: temperature, top-p, maybe a few penalty terms. If you don't set a field, the runtime picks a default. That default is usually sensible, but "sensible" and "what your prompt was tuned for" are not the same thing.
My engine is written in TypeScript, and it wasn't setting every sampler field itself. The fields it left unset got filled in by the phone runtime's own defaults.
The evals ran on the desktop inference backend. The shipped builds ran on Android, on a different one. Wherever the engine left a field unset, the phone's runtime made its own choice. Every eval passed, and every eval was testing something the phone was not running.
What broke and how I fixed it
I noticed it as a feel problem first: the app on my phone never quite matched what the eval results said it should do.
Decision D-038 made the rule: the engine sets every sampler field explicitly, on every call. Build 7 was the first build whose sampling matched the eval batteries.
The project is parked now, for unrelated reasons. But those six earlier builds shipped, and they ran settings that no eval had ever seen.
It's less a hard policy than a standing instinct now: if you write something, prove that it works. Most of my time still goes to UI and user journeys, not the backend, but projects like my Unreal Engine game, still in development, have shown me plenty can go wrong even when the code thinks it's proven itself and only a human clicking through catches it. I don't have a formal edge-case checklist. I experiment, iterate and refine more than I follow one fixed process.
When an eval only validates what it actually tested
An eval suite tells you whether the system you tested does what the test says it does. It doesn't tell you whether that system is the one anyone is running. That gap — the difference between what you validated and what shipped — is yours to close, and it doesn't close itself.
I've shipped other things where the gap was smaller and still mattered.
A generative audio app I built for my own use passed its entire automated test suite and still had two showstopper signal-processing bugs: a self-canceling stereo pan law (left and right gain were the same number at every pan position, so hard-left produced silence) and a limiter that was actually an unthresholded waveshaper applying constant audible distortion to every sample. A five-agent review pass with independent web research found both bugs and named them in the executive summary as reasons the project was "not currently shippable." I fixed both the day after the review landed. The automated tests had checked that the code did what it said; they didn't check that what it said made sense.
Tests check that code does what it says. They don't check that what it says is what the product actually needs. The interactive fiction engine's eval batteries passed. They were checking the right things. They just weren't checking them against the config any user would ever run, because I hadn't pinned the fields the runtime was choosing for me.
The lesson I took from it is simple: an eval only validates the exact configuration it actually ran. If you leave fields unset, you don't know what the runtime is doing with them, and the runtime isn't going to tell you. Set every field explicitly, then run the eval. Otherwise the eval is testing a system you didn't ship, and you shipped a system the eval never saw.
Corrected 2026-09-29: five named sampler fields with specific values, found by comparing phone logs to eval transcripts → no repeat penalty on most roles plus a min-p value on every call, noticed because the app never matched the eval results; the six builds "weren't rolled back" because they were "already parked" → the project was parked later, for unrelated reasons; and other details that did not match the record.