Reading this with an AI? Start at /for-agents/ (orientation) or /llms.txt (index).

Skip to content
TODD BROWN
« Writing

The tests proved every fix. They couldn’t see the bug.

I'm building an action RPG in Unreal Engine with AI coding agents. They write the code, the tests and the documentation. I direct, decide and playtest with a controller. The game is in development and stays unnamed here.

On 3 October 2026 the project had about 68,000 lines of game code and about 87,000 lines of tests. That same morning I played a build and wrote seventeen notes in nineteen minutes.

This is about what that much testing did for the project, and what it could not do.

The same bug, one at a time

One kind of bug kept coming back: something that should have reset when the player died, and didn't. Dead enemies still thinking. A respawn inside a pillar. A controller that stopped responding after respawn. A boss that kept its damage. A lock-on that survived my death. A boss that attacked me at the rest site as I came back.

When the playtest records were checked, there were eleven findings of that kind across five playtests.

Each one got the full treatment. The agents found the cause, wrote the fix, wrote a test proving the fix, had a second agent verify it, and moved on. Every fix was correct. Every test passed.

Nothing in that loop asks why the same thing keeps happening. I asked it, on 3 October, in these words:

"We've had like 3 or 4 different playtest bugs over the course of the project that have been tied to things not being reset upon death. If we had more architectural oversight, we could have implemented a better death-resolution process and not needed so many individual fixes."

I had undercounted. It was eleven.

What heavy testing does in an AI-first codebase

When agents write the tests, tests are nearly free. So they accumulate, and three things follow.

Passing tests start to feel like progress. Each fix arrived with proof. The proof was real. It was proof that one symptom was gone, and I was reading it as proof that the problem was gone.

A test checks the thing it was written for. A test written after a bug confirms that bug stays fixed. It has no opinion about the next bug of the same family, because nobody has written that one down yet. The tests were good at proving each fix and blind to the class.

The ceremony has a cost you can't see from inside it. On 3 October, six fixes from that morning's playtest took about three hours of agent work in writing, verifying and re-checking before a line of code moved. The build I had just played was missing a dodge roll I had supplied the animation for, because it was still in the queue behind verified work.

What the rigor did catch

It would be easy to turn this into "tests are overrated." The same two days say otherwise.

Fresh verification caught a change that would have silently retuned every enemy in the game. It caught an enemy "walk home" behavior that would never have worked in the packaged build. It caught a measurement gate that had quietly loosened from 1.4 centimeters to 45 for three rounds of work. Each of those would have reached me as a vague note that something felt off.

So both things were true. The checking paid for itself in specifics, and it was not producing what I measure, which is how the game feels on a controller.

The question was never verification or speed. It was which verification earns its cost.

What changed

Three decisions, in one afternoon.

No more playtests until everything planned is built. Small builds were giving me small pieces to judge and hiding how they fit together.

Architecture before fixes. I paused the fix queue and had the agents audit the code for places where one design would replace many patches. Death was the model case. They found seven such areas from the bug history, and two more by reading the list of planned features, which the bug history could not show.

Speed over ceremony, where the risk is low. The test suite stays as the safety net. The step-by-step verification of every change is now reserved for changes that touch shared foundations or can't be undone.

The death design is the part I'd point another builder to. Death, resting, retrying and continuing a saved game now go through one place. What survives is a table with one row for every piece of information the game saves: keep it, or reset it. The test for that table restores a saved game and checks that everything marked "keep" came back identical and everything marked "reset" actually changed. If the game saves something that has no row, the test fails.

That is a different kind of test. The old ones each said "this bug is fixed." This one says "nothing is unaccounted for." It can fail for a bug nobody has found yet.

What I don't know yet

Whether it holds. The first pieces landed the same afternoon, and each passed. The test that enumerates every saved field is the next thing to be built, and I want to see it fail before I trust it passing.

The lesson I'm taking is narrower than "test less." A wall of passing tests told me each repair was sound. It could not tell me I was repairing the same thing again. Seeing that took someone outside the loop, holding the controller.

More on the project is on the game's build page.