---
title: "My READMEs lied about my tests"
url: "https://toddpaulbrownjr.com/writing/my-readmes-lied-about-my-tests/"
author: "Todd Paul Brown Jr."
description: "Every stale test count Todd found this year undercounted, across three projects and three stacks. Why docs drift one way, and why agents make it worse."
kind: "article"
updated: "2026-09-26T02:50:36+00:00"
---

# My READMEs lied about my tests

Every stale test count I've found this year undercounts. Not once has a project's documentation claimed more tests than the repo actually holds.

## Three projects, one direction

**An on-device interactive fiction engine, now parked.** Its README states 610 deterministic tests, under a status header dated 10 September. Its own roadmap, one milestone and five commits earlier, had already logged 1,010 tests green. The number went backwards in the doc while the real count kept climbing in the repo, and nothing anywhere in the project reconciles the two.

**A proofreading app I built for myself, finished and in daily use.** Its last written summary said 309 backend tests. That was true once, in mid-July. When I actually counted, it was 368. And the counting itself had a trap in it: a plain search of the test files finds 305 test functions. Five of those are parametrized, and the test runner expands them into 68 separate cases. 305, minus the 5, plus the 68, is 368, which matches the project's own last logged total exactly. So even the "just count the files" check undercounts, unless you count the way the test runner does.

**A Godot game, parked and not shipped.** Its implementation notes claimed "8 test scripts." There were 10. That one wasn't caught by any documentation habit. A refinement audit on the project tripped over it.

Three projects, three stacks, one identical failure.

## It happens in the repo with the tooling, too

I'd like to say this only happens in side projects. It doesn't.

LEGION, the system I run my company on, keeps a hand-maintained manifest of what each product is and where it stands. On 21 September, while checking numbers for exactly this kind of writing, I found the manifest declaring 1,175 tests for our revenue agent. The deployed commit had 1,334. Undercount again.

And when I measured the game I'm building now, the first draft of that measurement report understated commits, decision entries, work packets and subagent conversations, all by small amounts. A fresh-context verification pass caught it before anything was published. Also undercounts.

Every discrepancy ran the same way. That's not noise. It's structural.

## Why it only drifts down

Tests are cheap for an agent to add and nobody's priority to describe.

When an agent fixes a bug or builds a feature, it adds tests. Nobody asks it to go back and update the sentence in the README that says how many there are, and it isn't going to volunteer. So the suite grows every session and the number in the doc stays where someone last typed it.

The number in a document isn't a live fact. It's a snapshot of the last day someone updated it. Because suites only grow, the snapshot is always low.

## Why agents make it worse

A person skimming a stale doc usually notices the number feels old and checks. An agent reading the same doc treats "610 tests" as a fact and reasons from it, confidently, for the rest of the task, with no instinct that a number in a document can simply be wrong.

That's what turns a cosmetic problem into a real one. An agent that believes the suite has 610 tests will write a plan, a status report or a release note on that basis. The staler the doc, the more convincingly an agent will act on it, and the longer a session runs unattended, the longer it stays wrong.

## What I do about it

**For publishing, a rule.** Where documents disagree, use the highest count recorded in the repo, or the measured count if that's higher. That's why the interactive fiction engine is 1,010 in anything I write, not 610. It's a rule for being accurate in public. It doesn't fix the drift.

**For the drift, a script.** I built a small one that regenerates the facts (versions, test counts, git state, whether a file exists) straight from the source into one generated file, and keeps judgment in files only a person edits. A collector that fails reports "unavailable"; it never invents a value. A second run has to print "already in sync". How it works is a post of its own.

It only protects what it derives, though. That 1,175 in LEGION's manifest sat in a hand-typed source the generated panel quotes but doesn't check. The script couldn't flag it. A person cross-checking by hand did.

**Fewer numbers in prose.** The cheapest fix is to stop typing numbers that a machine could derive. A README that says "run the suite to see the current count" is less satisfying and never wrong.
If a document in your repo states a number instead of deriving it, assume it's already behind. The fix isn't writing better summaries. It's writing fewer of them, and building something that regenerates the ones that are just facts.