I had a 10-billion-token headline for the game I've been building. The actual number was 36 million. The gap isn't rounding error — it's a factor of 280, and it happened because I quoted the wrong column.
A separate audit cut my Brain Vault document count from an earlier figure of about 1,850 down to the measured number: 1,765. I built a collector that reconstructed how the game project was actually built from session transcripts and repository history, then ran four verification passes on it in a single day. Those passes found three real bugs in the collector itself. One bug had reported 13 completed items in a category where the true count was zero.
This is what I mean when I say the receipts come with the claim: point the same verification discipline you use on a product at the report you're about to publish about it. If your audit tool has never been wrong, you haven't run it enough times.
Cache reads are not generated tokens
The game project — an action RPG in Unreal Engine, built in C++, in development, playable build under playtest — recorded 10.29 billion tokens in its session transcripts. That figure is real. It's also the wrong thing to quote.
Cache-read tokens are the model re-reading context it has already seen, billed at a steep discount. Generated tokens are the new code, documentation and explanations the model writes. The 10.29 billion figure is cache reads. The generated-token count, the one that describes how much the model actually wrote, is 36.8 million.
Both figures cover only 14 of the project's 79 calendar days — the two-week window when three-quarters of the production C++ was written, measured with git blame. About 90% of the project's commits landed in those 14 days. The other quarter is a two-day scaffold from July, whose transcripts no longer exist. I measured 157 prompts in that two-week window, and I say it that way — prompts in the window when most of the code was written — not "157 prompts produced 81,707 lines."
The lesson is simple: before repeating any AI usage number, ask what it's counting and what window it covers. Most published figures answer neither question.
The hour estimate came with a sensitivity table
The same project report includes an estimate of human hours. I say estimate because that's what it is, and I published it with a sensitivity table showing the number moves from 42 to 97 hours depending on where you draw the judgment line for how long I'm still paying attention after my last prompt.
The instructions in the report say never to quote the hours figure bare and never to call it measured. It's labelled as an estimate in the sentence, every time. The variance table is there because the collector sees my prompts, not my attention, and how long I keep watching a running session isn't crisp, and pretending it is would make the number look more certain than it actually is.
The distinction between measured and estimated isn't pedantry. It's the difference between a claim that survives scrutiny and one that doesn't.
The eval that measured nothing
A separate project — an interactive fiction engine, not the game — reached 1,010 deterministic tests, the highest figure recorded in the repository. It also had an eval that measured nothing.
I built an eval to check which story beats the model had covered in a generated narrative. The eval compared the model's own self-reported claim about coverage against a separate, code-computed definition of the same thing, run over a replay corpus. Agreement came back at chance level — Cohen's kappa approximately zero. The model's self-reported answer correlated with its position in the response, not with what it actually covered.
I replaced the self-reported answer with the code-computed definition, rather than trying to tune the prompt so the model would answer more accurately. The rule is: never let the model grade itself on the exact thing you're measuring. Compute the answer in code wherever the truth is code-computable, and reserve the model for what code genuinely can't determine.
The project is parked now, but the lesson holds: an eval that doesn't measure the thing you think it measures is worse than no eval, because you'll trust a number that means nothing.
Three bugs in the collector itself
The game project's report-builder ran four verification passes in a single day:
- The collector's first run, which produced the draft report.
- A fresh-context check that found the draft out of step with the repository, two weeks' figures swapped in the prose, and two gaps in the collector's own classifier.
- A third pass that found a counting bug: 13 reported items in a category where the true count was zero.
- A fourth pass, at my request, that added a count of the project's documentation.
The collector was purpose-built for this report. It parsed Claude Code session transcripts, matched them to repository commits, and reconstructed what happened when. The second and third passes found the collector's three real bugs. The category bug was a logic error in the collector's own code — its pattern counted blank entries as completed. The swapped figures were a mistake in the report's prose, not the collector.
If I'd published on the first draft, the headline would have claimed 13 things that never happened and had the busier and quieter weeks swapped.
The Brain Vault count dropped for a similar reason: an earlier summary had said "roughly 1,850 documents," but the measured figure — counted across the whole vault — came in at 1,765. The concept-pillar count dropped too, from 56 to 50.
Labels in the sentence
Every number in this article carries a label: measured, estimated, recorded but not re-run, or stated as a range. The game's 36.8 million generated tokens and 10.29 billion cache reads are measured, counted from the Claude Code session transcripts. The 157 prompts are counted from the session transcripts. The 42-to-97 hour estimate is explicitly called an estimate and comes with the variance table that shows why.
The Brain Vault's 1,765 documents, 7,075 links, and 50 concept pillars are measured counts from a Python script that walks the vault and reads the frontmatter. The median of 5 links per document is computed from the same data. The 17 agent playbooks are a directory listing.
None of these numbers round up. If the source says "about" or gives a range, the copy keeps it. If a test suite was recorded at a specific commit and hasn't been re-run since, the copy says "recorded, not re-run" rather than "measured." The receipts page on this site follows the same rule: Sadyr's 2,816 tests are labelled "recorded, not re-run" because the suite wasn't executed again to produce that table.
The discipline is: say what the number is counting, say what window it covers, and say whether it's measured or estimated. In the sentence, not in a footnote three pages later.
Checking the thing you're about to ship
I built the game project's report the same way I build the game itself: the agent writes the code and the documentation, I direct and validate. The report got four verification passes, run in sequence.
The second pass found two weeks swapped. The third found a category with 13 phantom completions. If the report had been a product feature instead of a document, those would have been production defects, and they would have shipped if I hadn't checked.
The lesson isn't "run more tests." It's that the same discipline you use to ship software applies to the claims you make about it. Build the audit, run it, and expect it to find things — because if it never does, you're not actually checking.
Corrected 2026-09-29: "three of the four passes found real bugs in the collector" and "the same interactive fiction engine, before it moved to Unreal" → the second and third passes found the collector's three bugs, and the interactive fiction engine is a separate project from the game; and other details that did not match the record.