---
title: "I built a tool to audit my own AI claims. It made my numbers smaller."
url: "https://toddpaulbrownjr.com/writing/i-audited-my-own-ai-claims/"
author: "Todd Paul Brown Jr."
description: "I audited my own AI usage claims with a purpose-built collector and four verification passes. The audit cut a 10-billion-token headline to 36 million and…"
kind: "article"
updated: "2026-09-26T02:50:31+00:00"
---

# I built a tool to audit my own AI claims. It made my numbers smaller.

I had a 10-billion-token headline for the game I've been building. The actual number was 36 million. The gap isn't rounding error — it's a factor of 280, and it happened because I quoted the wrong column.

The same audit cut my [Brain Vault](/builds/brain-vault/) document count from an earlier figure of about 1,850 down to the measured number: 1,765. I built a collector that reconstructed how the game project was actually built from session transcripts and repository history, then ran four verification passes on it in a single day. Three of those passes found real bugs in the collector itself. One bug had reported 13 completed items in a category where the true count was zero.

This is what I mean when I say the receipts come with the claim: point the same verification discipline you use on a product at the report you're about to publish about it. If your audit tool has never been wrong, you haven't run it enough times.

## Cache reads are not generated tokens

The game project — an interactive fiction engine in C++ and Unreal Engine, in development, playable build under playtest — hit 10.29 billion tokens on the Anthropic bill. That figure is real. It's also the wrong thing to quote.

Cache-read tokens are the model re-reading context it has already seen, billed at a steep discount. Generated tokens are the new code, documentation and explanations the model writes. The 10.29 billion figure is cache reads. The generated-token count, the one that describes how much the model actually wrote, is 36.8 million.

Both figures cover only 14 of the project's 79 calendar days — the two-week window when three-quarters of the production C++ was written, measured with `git blame`. About 90% of the project's commits landed in those 14 days. The other quarter is a two-day scaffold from July, whose transcripts no longer exist. I measured 157 prompts in that two-week window, and I say it that way — prompts in the window when most of the code was written — not "157 prompts produced 81,707 lines."

The lesson is simple: before repeating any AI usage number, ask what it's counting and what window it covers. Most published figures answer neither question.

## The hour estimate came with a sensitivity table

The same project report includes an estimate of human hours. I say estimate because that's what it is, and I published it with a sensitivity table showing the number moves from 42 to 97 hours depending on where you draw the judgment line between supervision and active work.

The instructions in the report say never to quote the hours figure bare and never to call it measured. It's labelled as an estimate in the sentence, every time. The variance table is there because the line between "I'm watching Claude Code work" and "I'm doing work" isn't crisp, and pretending it is would make the number look more certain than it actually is.

The distinction between measured and estimated isn't pedantry. It's the difference between a claim that survives scrutiny and one that doesn't.

## The eval that measured nothing

An earlier project — the same interactive fiction engine, before it moved to Unreal — reached 1,010 deterministic tests, the highest figure recorded in the repository. It also had an eval that measured nothing.

I built an eval to check which story beats the model had covered in a generated narrative. The eval compared the model's own self-reported claim about coverage against a separate, code-computed definition of the same thing, run over a replay corpus. Agreement came back at chance level — Cohen's kappa approximately zero. The model's self-reported answer correlated with its position in the response, not with what it actually covered.

I deleted the self-reported field outright and replaced it with the code-computed definition, rather than trying to tune the prompt so the model would answer more accurately. The rule is: never let the model grade itself on the exact thing you're measuring. Compute the answer in code wherever the truth is code-computable, and reserve the model for what code genuinely can't determine.

The project is parked now, but the lesson holds: an eval that doesn't measure the thing you think it measures is worse than no eval, because you'll trust a number that means nothing.

## Three bugs in the collector itself

The game project's report-builder ran four verification passes in a single day:

1. A category-completeness check that found 13 reported items in a category where the true count was zero.
2. A date-range check that caught two weeks' token figures swapped in the prose.
3. A cross-reference pass confirming every cited commit hash existed in the repository.
4. A final read-through that corrected the prose in place.

The collector was purpose-built for this report. It parsed Claude Code session transcripts, matched them to repository commits, and reconstructed what happened when. Three of the four passes found real defects. The category bug was a logic error in the collector's own code — it counted planned items as completed. The swapped figures were a copy-paste mistake in the report draft.

If I'd published on the first draft, the headline would have claimed 13 things that never happened and placed half a billion tokens in the wrong week.

The Brain Vault count dropped for a similar reason: an earlier inventory had rounded an estimate up to "about 1,850 documents," but the measured figure — counted directly from the vault's synthesized layer, excluding raw imports — came in at 1,765. The 50 concept pillars I'd quoted stayed correct; the total didn't.

## Labels in the sentence

Every number in this article carries a label: measured, estimated, recorded but not re-run, or stated as a range. The game's 36.8 million generated tokens and 10.29 billion cache reads are measured, pulled directly from the Anthropic usage API. The 157 prompts are counted from the session transcripts. The 42-to-97 hour estimate is explicitly called an estimate and comes with the variance table that shows why.

The Brain Vault's 1,765 documents, 7,075 links, and 50 concept pillars are measured counts from a Python script that walks the vault and reads the frontmatter. The median of 5 links per document is computed from the same data. The 17 agent playbooks are a directory listing.

None of these numbers round up. If the source says "about" or gives a range, the copy keeps it. If a test suite was recorded at a specific commit and hasn't been re-run since, the copy says "recorded, not re-run" rather than "measured." The [receipts page](/receipts/) on this site follows the same rule: Sadyr's 2,816 tests are labelled "recorded, not re-run" because the suite wasn't executed again to produce that table.

The discipline is: say what the number is counting, say what window it covers, and say whether it's measured or estimated. In the sentence, not in a footnote three pages later.

## Checking the thing you're about to ship

I built the game project's report the same way I build the game itself: the agent writes the code and the documentation, I direct and validate, and nothing ships without passing its tests. The report had no tests at first, so I added them — four verification passes, run in sequence, each one capable of failing the build.

The first pass found a category with 13 phantom completions. The second found two weeks swapped. If the report had been a product feature instead of a document, those would have been production defects, and they would have shipped if I hadn't checked.

The lesson isn't "run more tests." It's that the same discipline you use to ship software applies to the claims you make about it. Build the audit, run it, and expect it to find things — because if it never does, you're not actually checking.