Reading this with an AI? Start at /for-agents/ (orientation) or /llms.txt (index).

Skip to content
TODD BROWN
« Writing

Three production defects, found and fixed the same night

The first time Content Ops ran at real concurrency — three parallel jobs rewriting Adroit's own live website — it surfaced three separate production defects in one night. All three traced to the same architectural habit: legacy code that deleted and re-inserted a whole database table on every write. That pattern is cheap at one job at a time; at concurrency 3 it races or overloads the database.

All three were fixed before morning, because the logs were data, not decoration.

This was 8 September 2026.

Defect one: the runs table

Jobs failed within seconds of starting, before their first stage. The error was a duplicate-key collision on the runs table.

The cause: the runner's open/heartbeat/close logic for a run record still used a whole-table write on every update. It deleted every row in the table, then re-inserted every row. At one job at a time, that's harmless. At three jobs, two jobs race to delete and re-insert the same row at the same time, and you get a duplicate key or a lost run.

This was found directly from the raw crash, before any postmortem tooling existed, and fixed that same evening with row-scoped writes.

Defect two: the artifacts table, found by a tool built hours earlier

That same evening, I also built a new capability on a hunch: every terminal job path would now write a postmortem record. The system would export those records on demand, and a script (summarize-postmortems.mjs) would digest that export before a human read it raw.

It paid for itself before midnight, on the same night it shipped.

I deployed the next version. The app fell over. I uploaded the first postmortem export — 227 entries — and ran the summarizer. It surfaced the shape of the failure in one screen: 29 duplicate-key errors and 151 entries where a project had simply stopped.

The cause was the same architectural habit, in a third table. A function that rewrote the whole artifacts table on every stage — which both collided at concurrency 3 and generated enough database load to exhaust the connection pool. Every API call started failing, the health check restarted the app, and every project stopped.

This was the third table found with this exact defect. The jobs table had already been fixed earlier; the runs table was fixed earlier that same evening. Fixed the artifacts table the same night with row-scoped writes.

Defect three: found in a different log entirely

The third defect was found through a separate upload — a structured Sevalla platform log with 749 entries — not the postmortem export.

It showed 311 of 323 API errors were GET /api/jobs/<id> pool timeouts, arriving in bursts of 40-plus distinct IDs every 10 seconds.

The cause: the Content Plan screen polled one request per card, for roughly 92 cards, every 10 seconds. Each of those requests read the whole jobs table to find a single row.

Fixed with one batched request that returns summary data for all requested IDs at once.

What broke and how I fixed it

All three defects traced to one habit in agent-written code: write to a table by deleting the whole table and re-inserting every row.

The agent had written that pattern into three unrelated tables across different parts of the codebase. It ran fine in development, where I test one job at a time. It broke in production, where three jobs ran together.

The fix for each one was the same: replace the whole-table delete-and-insert with a row-scoped write. The runs table got row-scoped updates. The artifacts table got row-scoped upserts. The polling got replaced with a single batched request.

It's less a fixed habit than something that depends on the project's actual bottleneck. On the Unreal Engine build the constraint is machine resources, since I only run the engine on one box, so I scope features into packets instead of raising concurrency. Content Ops was different: those early failures were more about me being new to real version control, at a time when coding agents were worse at acting as their own orchestrators. Hard to boil down to one lesson. It was a rapid-fire trial-and-error era.

Two lessons

First: agent-written legacy code can carry one architectural habit across multiple unrelated tables. Finding it once does not mean it is gone. It showed up in three separate tables.

Second: build the diagnostic and export tooling before the incident that needs it, not after the second one. The postmortem summarizer was built on a hunch, for visibility into the first crash. It ended up finding the second defect the same night.

The raw crash, the system's own postmortems and a separate platform log surfaced a runs race, an artifacts race and a polling fan-out in one night. All three were fixed before morning.

Corrected 2026-09-29: the system's own postmortems surfaced all three defects → only the second came from the postmortems (the first from the raw crash, the third from a separate platform log); a claim about how long Claude Code had been public → removed.