--- title: "Six shipped builds ran settings no eval had validated" url: "https://toddpaulbrownjr.com/writing/evals-passed-the-wrong-app/" author: "Todd Paul Brown Jr." description: "Six Android builds of an on-device fiction engine ran sampler settings the eval suite never tested, because the runtime silently filled defaults." kind: "article" updated: "2026-09-26T02:50:28+00:00" --- # Six shipped builds ran settings no eval had validated I shipped the first six Android builds of an on-device interactive fiction engine without ever checking that the evals were testing what those builds actually ran. They weren't. The phone runtime silently filled in its own sampler defaults wherever my engine didn't set a field explicitly, so every eval battery that supposedly validated those builds was testing different settings than the app on my phone used. I found it because the app never matched the behavior the evals recorded. Characters repeated themselves, pacing felt wrong, and responses that should have been rare turned up constantly. I opened the runtime logs on my phone and compared them to the eval transcripts line by line. The temperature, top-p, top-k, frequency penalty and presence penalty fields were all different — not close, different — and every eval had passed. The problem wasn't that the evals were bad. The evals were checking the exact things they were supposed to check: knowledge-isolation boundaries, story-beat coverage, and guardrail compliance. They all passed, and they passed correctly for the configuration they saw. That configuration just wasn't the one any user was running. The fix came down to one new rule in the decision log: every sampler field gets set explicitly on every call, whether the model cares about that field or not. Decision D-038 made it binding; build 7 was the first build whose sampling actually matched the eval batteries. Up to six shipped builds — the exact count depends on how you label partial rollouts — ran settings no eval had validated. ## The mechanic that breaks silently Most model runtimes accept a sampling config as a dictionary or a JSON object: temperature, top-p, maybe a few penalty terms. If you don't set a field, the runtime picks a default. That default is usually sensible — 1.0 for temperature, no penalty — but "sensible" and "what your prompt was tuned for" are not the same thing. My engine's sampler config lived in a TypeScript module. It set three fields: temperature, top-p, and max tokens. The model runtime I was using accepted eight fields. The five I didn't set — top-k, frequency penalty, presence penalty, repetition penalty, and a stop-sequence override — all got filled in by the runtime's own defaults, which were documented nowhere I looked and different between the desktop shell and the Android build. The evals ran on the desktop shell. The shipped builds ran on Android. Both used the same prompt, the same model, and the same three-field config object. The runtime saw five missing fields and made five different choices depending on which platform it was running on. Every eval passed, and every eval was testing something nobody was using. ## What broke and how I fixed it I noticed it as a feel problem first. The app on my phone felt chatty and repetitive in a way the eval transcripts didn't, and certain story branches that were supposed to be rare (defined in code, checked by a scored probe) came up in about 40% of sessions instead of the target 8%. I thought the model was broken, then I thought the prompt was broken, then I pulled the logs and saw the settings. The eval battery's recorded config showed `temperature: 0.7, top_p: 0.9, top_k: null, frequency_penalty: 0.0, presence_penalty: 0.0`. The phone log for the same prompt showed `temperature: 0.7, top_p: 0.9, top_k: 40, frequency_penalty: 0.3, presence_penalty: 0.6`. I hadn't set any of the last three, ever. The Android runtime was setting them for me. Decision D-038 made the rule: set every field the runtime accepts, even if the engine doesn't care about it. If the correct value is zero or null, say zero or null explicitly. Build 7 was the first build that followed it. I re-ran three of the eval batteries — the knowledge-isolation suite, the guardrail replay, and the story-beat coverage check — against build 7 with the full eight-field config pinned, and they passed again. This time the config in the eval matched the config on the phone, field for field. The six earlier builds weren't rolled back, because they were already parked for other reasons — prose quality (possibly a model problem, not a harness problem) and battery life on a phone. But those builds shipped, and they ran settings that no eval had ever seen. ## When an eval only validates what it actually tested An eval suite tells you whether the system you tested does what the test says it does. It doesn't tell you whether that system is the one anyone is running. That gap — the difference between what you validated and what shipped — is yours to close, and it doesn't close itself. I've shipped other things where the gap was smaller and still mattered. A CRM data mirror for a national amateur sports league went live with three numbered audit rounds left as dated comments in the code, eight findings still open. A dedicated field-coverage audit caught a registration field silently dropped since the initial merge — found two months after go-live, not from a bug report — because the two systems used slightly different names for it and the merge script missed the mismatch. A generative audio app I built for my own use passed its entire automated test suite and still shipped two showstopper signal-processing bugs: a self-canceling stereo pan law (left and right gain were identical at every pan position, so hard-left, hard-right and center all sounded the same) and a limiter that was actually an unthresholded waveshaper applying constant audible distortion to every sample. A five-agent review pass with independent web research found both bugs and named them in the executive summary as reasons the project was "not currently shippable." I fixed both the same day the review landed. The automated tests had checked that the code did what it said; they didn't check that what it said made sense. Tests check that code does what it says. They don't check that what it says is what the product actually needs. The interactive fiction engine's eval batteries checked knowledge-isolation boundaries, story-beat coverage, and guardrail compliance, and they passed every time. They were checking the right things. They just weren't checking them against the config any user would ever run, because I hadn't pinned the fields the runtime was choosing for me. The rule that came out of it is simple: an eval only counts when every config field it depends on is pinned. If the runtime accepts eight fields and you set three, you have not validated anything, because you don't know what the other five are doing and the runtime isn't going to tell you. Set all eight, check the logs to confirm they match, then run the eval. Otherwise the eval is testing a system you didn't ship, and you shipped a system the eval never saw.