I built a local-model benchmarking tool that does its arithmetic before touching your hard drive. Milkman installs and manages a local inference runtime, keeps a verified registry of models, derives per-model hardware-fit settings, and runs a benchmark harness to compare candidate models against an incumbent under a statistical promotion policy. The agent writes the code, tests and documentation; I direct, decide and validate. Status: in development — the project's own handoff record lists unresolved risks, including that a real GPU candidate sweep from the UI had not yet been run.
I built it in roughly 31.4 hours spanning two calendar days, 22 to 23 August 2026. 15 commits landed: 485 files, 40,155 insertions from the initial commit. Every commit but the first two is agent-co-authored, and the repo is explicitly agent-operated by design.
The hardware refusal, computed rather than assumed
The tool runs four rules before touching a candidate model. A dense model that fits runs unchanged. A dense model that doesn't fit gets a capped context. A mixture-of-experts model routes only its overflow experts to CPU. And if even that CPU-resident share would need more RAM than the machine has to spare after a safety margin, the tool raises a named error and refuses outright, before any download starts.
Every hint carries a field marking it "derived" (arithmetic) or "measured" (confirmed by an actual benchmark run), so a hardware guess is never silently treated as fact. The decision record for this logic is numbered DEC-0013 in the project's own decision log.
19 golden tasks, 254 runs recorded
19 golden coding and doc-chat tasks, each with hidden tests, back every model-promotion decision. 254 golden runs are recorded. The last full test run: 290 passed, 1 failed, across 294 test functions in 26 files. (Those numbers are measured: I re-ran the suite to produce them.)
The repo structure is fixed: a single entry-point file, a session handoff file rewritten at the end of every session rather than appended to, and a structured, ID-based memory of learnings, experiments and decisions. Each of those carries an epistemic-status ladder — assertion → observation → experimental_evidence → validated_learning → policy — that may only move up with linked evidence.
What broke and how I fixed it
Ten parallel subagents burned a five-hour session budget in 33 minutes. A lead agent launched 10 subagents at once for research; several sub-forked further; the shared web-search allowance, 200 per session, emptied almost immediately; 8 of 10 agents were killed mid-write by the API, and one report — the project's own bootstrap document — was lost and had to be rewritten.
The fix was codified as a hard rule, not a one-off lesson: cap concurrent subagents at three, ban any subagent from forking its own subagents, put word caps on subagent output, tier models to task difficulty, and checkpoint work to disk early. The failure is numbered FAIL-0001, dated 22 August 2026, in the project's own failure log.
A test's cleanup call deleted a real model file. A subagent-written unit test for the "evict a model" command called a cleanup function mid-test that reverted every patch on that test's sandbox, including the one redirecting it to a fake install root. The rest of the test then ran against the real machine: it deleted a real 4.69 GB model file and overwrote a real registry record's checksum with a fake one.
Recovery worked because Milkman's own eviction policy writes an inventory record, including provenance, before any deletion — the file could be re-verified and re-downloaded. The fix: an autouse hermetic-sandbox fixture that a test's own cleanup call cannot strip, plus a new rule that no test may call that cleanup function at all.
Tests check that code does what it says. They don't check that what it says is what the product actually needs.
The broader lesson, one I've hit a few times now: a green test suite tells you the code agrees with itself, not that it does what the product actually needs. Closing that gap takes an adversarial pass with a different brief, or a check against a ground truth the model doesn't get to grade itself. In this case, the sandbox fixture was real, the test suite was green, and the test still deleted production data because nobody had told it to confirm the sandbox stayed in place.
Agent-operated by design
The repo is built to be handed off. The entry-point file is fixed. The handoff file is rewritten, never appended to, so the next session starts with a clean state rather than a transcript. And the memory structure — learnings, experiments, decisions — all carry IDs and epistemic status, so an agent (or a human) can trace how a claim moved from assertion to validated learning.
That structure matters when you're asking an agent to pick up work you left weeks ago. The session handoff tells it what the current state is. The decision log tells it what was already decided and why. The failure log tells it what not to repeat. And the epistemic-status ladder keeps speculation separate from fact until evidence moves it up.
What is next
The tool runs. The tests pass. The hardware refusal works. The failure that deleted a real model file is fixed, and the fix is documented as a rule the next session will inherit. Whether I finish it, use it regularly, or park it depends on whether I actually need to benchmark local models often enough to justify maintaining it — a question I don't have an answer to yet.