At Banyan in 2016, the revenue goals for the year were built source by source: itemizing each revenue source and forecasting sales based on historical data and growth patterns, contributing to the year's strategic financial planning before the year began. The target and how it would be tracked existed before the work that was supposed to hit it. That's a pre-defined metric in the plainest sense.
Ten years later, I built an on-device interactive fiction engine and ran eval batteries against it for weeks. One of those evals compared the model's own self-reported claim about which story beats it had covered against a separate, code-computed definition of the same thing. Agreement came back at chance level — kappa approximately zero. The model's answer correlated with its position in the response, not with what it actually covered. The self-reported field was deleted and replaced outright with the code-computed definition.
The eval ran. The eval passed. The eval measured nothing.
The principle
Decide what counts before the result arrives. Define success and how you'll measure it before the work starts, not after — and make sure what you're measuring is actually the thing you care about. An eval whose metric was never defined in code will agree with whatever the model says about itself.
Then: itemized, source by source
The Banyan revenue-goal process was straightforward: take each revenue source, forecast sales based on historical data and growth patterns, add them up. The goals themselves and the figures are excluded, but the method is the point. The metric existed as a list of specific, trackable items before the year began. Whether those goals were hit is not recorded.
The habit stuck. In every real project I've built since — at Banyan and in the last two years with AI agents — the things that worked had their success conditions written down and their measurement pipeline verified before the result showed up.
Now: the eval that agreed with everything
The interactive fiction engine is parked now, possibly waiting for a better model. Phases 1 through 8 of a nine-phase roadmap had all passed their exit gates by August 2026, and Phase 9 was nearly complete except for checks that needed a real phone. It ran on a 3,020-line phase-gated PRD, a 38-entry numbered decision log, and 19 distinct scored evals with numeric thresholds, plus a replay mode that reruns guardrail checks against recorded model responses without a fresh GPU pass. 1,010 deterministic tests green — the highest figure recorded in the repo.
One of those 19 evals was supposed to measure story-beat coverage: whether the model had actually mentioned the beats the game engine had asked it to cover in a given passage. The eval compared two values. One was the model's own self-reported list of beats. The other was a code-computed definition of the same thing, derived from what the model had actually written.
I expected them to agree most of the time and disagree at the edges — a useful signal about when the model thought it had done something it hadn't, or the other way around. Instead, agreement came back at chance level. Kappa approximately zero. The model's self-reported answer didn't correlate with what it had actually covered; it correlated with where in the response that answer appeared.
The eval had been passing for days. It measured nothing.
I deleted the self-reported field and used only the code-computed definition. The eval that replaced it actually tested the thing I cared about: not whether the model thought it had covered a beat, but whether the prose it wrote contained that beat according to a definition the model didn't get to grade itself.
The sampler-defaults failure
A related failure, same project. Up to six shipped Android builds ran different model sampler settings than every eval battery that had supposedly validated them. The phone runtime silently filled in its own defaults wherever the engine didn't set a sampler field explicitly. The evals validated one configuration; the shipped app ran another.
I found it because the app on my phone never matched the behavior the eval batteries recorded. The fix became decision D-038: the engine now sets every sampler field explicitly on every call. Build 7, committed 10 September 2026, was the first build whose sampling actually matched the evals.
The evals passed. They validated the wrong app.
What broke and how I fixed it
The pattern in both cases: a measurement pipeline that looked rigorous but quietly stopped measuring the real thing. In one case, the model graded itself. In the other, the thing being tested defaulted to settings the test never saw.
Tests check that code does what it says. They don't check that what it says is what the product actually needs. An eval that passes every time tells you the model agrees with itself, not that the output is correct.
The fix in both cases: stop trusting an unset default or a self-report. Derive status from what actually exists and whether its dependencies are satisfied, or set every field explicitly so nothing can fill in a value you didn't choose. Where a model's behavior matters, test the behavior against a ground truth the model doesn't get to define.
This habit shows up everywhere in my agent work now. Five distinct status words — built, live, demo, shipped, sold, parked, in development — held as separate, non-interchangeable claims, verified against my own live portfolio on one day in September 2026. A formal audit process for a national amateur sports league's CRM data mirror left findings visible as dated, severity-coded comments in the shipped code itself; at least three marked resolved, eight still open as of the last commit. A review agent with independent web research found two showstopper signal-processing bugs in a generative audio app — a self-canceling stereo pan law and a limiter that was actually an unthresholded waveshaper — that the build agents and the automated test suite had both missed.
The itemized, source-by-source planning habit from Banyan and the explicit-every-field discipline from the eval fix are the same instinct: don't let anything default silently. A pre-set target is only useful if the measurement pipeline underneath it is verified to be measuring the real thing, not a proxy for it. "It exists" and "it's actually done" are different claims, and conflating them is the most common way status drifts from reality.
Define the metric in code. Run it before you trust it. If it agrees with everything, it measures nothing.