---
title: "Measuring success, then and now: green tests are not a result"
url: "https://toddpaulbrownjr.com/writing/measuring-success/"
author: "Todd Paul Brown Jr."
description: "Self-reported evals at chance level and seven passing security probes on an abandoned product: why green tests don't prove you built something worth having."
kind: "article"
updated: "2026-09-26T02:50:34+00:00"
---

# Measuring success, then and now: green tests are not a result

I was told something once, in late 2024, during the interview process for a role I didn't end up taking. The interviewer pointed to my resume and said the work showed "a focus on achieving tangible results." What stood out to me was the word tangible. Not claimed results. Not self-reported progress. Tangible: the kind of result a third party could verify independently.

That bar matters more now than it did then, because when the thing measuring your success is an AI agent's own self-report, you can hit 100% on every metric and still ship nothing anybody wants.

## Two results that cleared the bar

Between June 2014 and March 2015, I ran operations at 180Fusion, a digital marketing agency with roughly 2,000 clients and a backlog in the thousands of outstanding tasks. The team wasn't just behind; they were falling further behind every month. The new processes I built reduced the labor needs of the team by 80%, cleared the backlog, and took SLA compliance from 19% to 100% within two months. Those are self-reported figures from my own resume, but the 19% and the 100% are the kind of numbers someone outside the team could check: did the clients get what they were promised, on time, or not?

At NP Accel in 2019 and 2020, I built and templatized a custom reporting system that was later adopted company-wide. Someone at a different level of the org looked at what I'd made, decided it worked better than what they had, and rolled it out everywhere. That's a result: another team used it without me in the room.
Both of those results share one feature: they were checked against something outside the system that produced them. The backlog count came from the client queue, not from my team's estimate of how much work we thought we'd done. The reporting tool's adoption came from another group's decision to use it, not from my own belief that it was good.

## When the eval measured nothing

Fast-forward to 2026. I'm building an on-device interactive fiction engine, and one of its features is a self-reported story-beat-coverage field: the model writes a list of which narrative beats it thinks it just covered in the turn it generated. I needed to know if that self-report was accurate, so I wrote an eval that compared the model's claim against a separate, code-computed analysis of the same response, run over a replay corpus.

The agreement came back at chance level. Kappa was approximately zero. The self-reported coverage didn't correlate with what the model had actually written; it correlated with the model's position in its own response. It was measuring the model's confidence about having done the work, not whether the work was done.

I deleted the self-reported field outright and replaced it with the code-computed answer. The eval itself was fine—it did exactly what I asked it to do. What it revealed was that the thing I'd asked the model to self-report was worthless as a signal.

## When every test passed and the product failed

About the same time, I built an on-device AI assistant as a plugin on top of OpenClaw, an open-source multi-channel gateway. The assistant was meant to give me periodic nudges toward rhythm and productivity: hydration, movement, lunch, bedtime, meeting warnings. It could read a personal notes vault for context, hold light conversation, and never write anything without explicit permission.

Before I used it for real, I ran a full security escape-probe suite at install time. Seven checks, all scripted and manual: a protected key file was refused, a path-escape attempt was rejected, an absolute out-of-vault path was rejected, root listing hid infrastructure, and search never surfaced key material. Every probe passed. The thing was safe and correct.

It saw four active build days across an eleven-day span, then 55 days of silence. I abandoned it.

The assistant was a resource hog and had a really hard time with continuity. It was supposed to promote productivity, but it typically failed to provide value unless I was babysitting it, which defeated the purpose. Passing every security and correctness test told me the code did what it said. It didn't tell me the thing was worth having.

## The lesson is the same in both eras

A cleared backlog is a result someone else can count. An adopted reporting tool is a result another team decided to use. A self-report eval that agrees with reality at chance level is not a result; it's a measurement of nothing. An assistant that passes every safety probe but demands supervision to be useful is not a result; it's a test suite asking the wrong question.

The principle that carried over: a result-driven approach only works if the result you're tracking is real. Green tests check that code does what it says. They don't check that what it says is what the product actually needs. A metric that looks like an answer but isn't checked against ground truth is worse than no metric at all, because it lets you believe you're measuring progress when you're only measuring the system's own confidence about itself.

Passing your own test suite proves the thing is safe and correct. It never proves the thing is worth having.