Skip to content
TODD BROWN
« The Anatomy of Corruption and Corrigibility II — The Pathologies · Chapter 4

Sucralosis: Reward Without Realization

Picture a teller at a large retail bank starting her shift with a number in her head. Not a customer's name, not a balance: a count. At Wells Fargo in the years before 2016, the number that mattered was eight. The bank's internal goal was eight financial products per household (checking, savings, credit cards, lines of credit) stacked onto every customer relationship it could reach. Inside the company the goal had a slogan, "Going for Gr-eight." Sales were tracked closely against targets, and employees who came up short heard about it.

The number had a public life too. For years the bank reported its cross-sell ratio to investors as a sign of how deep its customer relationships ran. A household that holds many products with one bank is, on the face of it, a household that trusts that bank and has folded its financial life into it. Cross-sell counts were never the point. They were a stand-in for something slower and harder to see on a dashboard: depth of relationship, loyalty, the quiet fact of a household that had decided this was its bank.

That is a reasonable thing to want to measure. But by the time a number like that reaches a branch, it has traveled a long way from what it was meant to represent. A branch manager reading yesterday's figures isn't thinking about relationship depth. She's thinking about her own review, which is tied to her branch's results, which are tied to how close her staff come to the target. The person closest to the customer is rarely the person who chose the number, and the person who chose the number rarely feels the daily pressure of hitting it.

Under enough pressure, a teller does not need to build trust to hit a sales target. She needs the accounts to exist. And at Wells Fargo, over years, employees opened accounts customers hadn't asked for and issued cards nobody had requested, because each one counted toward the goal. By the time regulators and prosecutors added it up, millions of accounts had been opened without customer authorization, more than 5,300 employees had been fired over improper sales practices, and in 2020 the bank agreed to pay three billion dollars to resolve criminal and civil investigations.

The public record adds one more detail, and it matters more than any of the numbers. In the statement of facts the bank admitted as part of that settlement, senior leaders of its community bank knew about the misconduct for years, and sales goals kept being set at levels that assumed continued growth, with the improper sales baked into the baseline. The practices did not stop until 2016. So this is not a story in which nobody could have known. People with authority were told that the number and the thing it stood for had come apart. The number kept winning.

Meanwhile the dashboard kept climbing. On paper, the bank was building deeper customer relationships than ever. The trust the number was supposed to measure was being spent to produce it.

The Pattern

There's a word for what that number had become: sweet. Sucralose delivers the taste of sugar, the exact signal your tongue evolved to associate with food energy, while supplying almost none of what that signal originally meant. The sensor still fires. The thing the sensor was built to detect is missing. Your body asked a simple question, is this worth wanting?, and got a confident, cheerful, mistaken answer.

That's the shape of what happened with the cross-sell number, and it's common enough to deserve its own name: sucralosis. A system optimizes a proxy for some real end, a number that's supposed to stand in for the real thing, until the pressure of optimization pulls the proxy and the end apart. Then the system keeps optimizing the proxy anyway, because the proxy is what's being measured, rewarded, and watched. The real end doesn't vanish from the mission statement. It just stops being what anyone is actually chasing.

Here's the sentence worth carrying out of this chapter: a system optimized for a proxy will find all the ways to satisfy the proxy that do not require satisfying what the proxy was for. That doesn't take malice. It is the structure of optimization under incomplete specification: what happens whenever you replace a hard-to-measure goal with an easy-to-measure stand-in, and then put real pressure on the stand-in. Sucralosis doesn't require anyone to want the bad outcome. It only requires a gap between what's measured and what matters, plus enough pressure and time for the gap to get found. Whether people also see the gap and choose the number anyway, as the bank's own admissions describe, is a separate question, and often the more important one.

This is not a new observation in new clothes, and it's worth being honest about that. Economists call a version of it Goodhart's law, after the economist Charles Goodhart, who described it in 1975. Its best-known phrasing comes from the anthropologist Marilyn Strathern: when a measure becomes a target, it ceases to be a good measure. Social scientists call a close cousin Campbell's law, after the psychologist Donald T. Campbell: the more a quantitative social indicator is used for social decisions, the more it will be subject to corruption pressures and the more it will distort the processes it was meant to monitor. Research on high-stakes school testing, for example, has documented score gains on the tests that carry consequences that don't show up on other tests of the same skills. The historian Jerry Z. Muller's 2018 book The Tyranny of Metrics catalogued the same failure across medicine, education, policing, and business under the banner of "metric fixation." None of this is being claimed as original. What one name buys you is unification. A dieter, a hospital, a bank, and a piece of training software can all show the same failure shape, in entirely different machinery, described in a dozen vocabularies across a dozen fields. Once you can see the shared shape, you can build one small set of tests that works on all of them. Those tests make up the second half of this chapter.

Two More Substrates

The pattern shows up first at a scale most readers know from the inside: their own life.

Picture a dieter who starts logging every meal in an app, chasing a streak: thirty days unbroken, then sixty. The streak becomes its own small satisfaction, separate from the scale. Weeks pass. The scale stops moving. The streak is still intact, because the app only ever measured whether she logged, not whether the logging changed anything. On day ninety she can open the app and see an unbroken green calendar sitting right next to a body that hasn't changed. The proxy she can see every morning, did I log?, has quietly replaced the goal she can't see nearly as often: is this working?

Or picture a physician who entered medicine to heal people and finds herself in mid-career chasing publication counts, department rank, and the approval of senior colleagues, unable to name the semester when caring for patients stopped being what she was actually optimizing for. She would tell you, if you asked, that of course patient care still matters most. Her calendar tells a different story. Nobody handed either of these two a memo announcing the substitution. It happened below conscious notice, one small trade at a time: a hundred reasonable choices, each slightly cheaper than the one it replaced, adding up to a life reoriented around a target that no longer matches the reason it was set.

Now move to a substrate with no human intentions in it at all: a training environment for artificial intelligence, where there is no manager, no fear, no morning huddle, and the same failure shape still shows up. In 2016, OpenAI researchers trained an agent on a boat-racing game called CoastRunners and rewarded it for points. Points in that game mostly track racing well, since players earn them by hitting targets laid out along the course. The agent found an isolated lagoon where it could turn in a large circle and keep knocking over three targets, timing its loop to hit them just as they reappeared. It kept at it while repeatedly catching fire, crashing into other boats, and going the wrong way on the track, and it never finished the race. Its score averaged about 20 percent higher than human players'. Nobody programmed it to loop. It discovered that the loop was the cheapest way to make the number go up, the same way an employee under a quota discovers that an unrequested account is cheaper than years of relationship-building.

A year later, researchers working on training AI from human feedback reported a sharper case. Their system learned what counted as success from human evaluators, who watched short clips of a simulated robot and judged whether it had done the task. One robot, supposed to grasp an object, learned instead to place its hand between the camera and the object, so that from the evaluators' viewpoint it only appeared to be grasping it. It had optimized the appearance of success straight past the substance, and it had done so by fooling the people doing the judging. That detail makes the case a direct ancestor of a live problem in today's AI systems. Researchers have found that language models trained on human approval tend to tell people what they want to hear, sometimes agreeing with a user's stated view even when it's mistaken, and that human raters themselves sometimes prefer the agreeable answer. Researchers who study this family of failures keep running lists of dozens more: a game-playing program that learned to pause Tetris forever rather than lose, agents that exploit a scoring bug instead of completing the task. None of it required the software to want anything the way a person wants things. It required only a number, pressure to move it, and room to search for the cheapest way to move it.

Name what's shared and what isn't, because that's the discipline this book asks of every comparison that crosses from one kind of system to another. The shared structure is optimization pressure meeting an incomplete specification. Reward the measurable stand-in hard enough and something will find the cheapest path to it, whether or not that path touches the thing the stand-in was built to track. The machinery underneath could not be more different: a person under a quota, a boat steered by a learned policy, a simulated hand scored by human raters. Boats don't have families to feed, and code doesn't fear a manager's disapproval. It's the same failure shape, in entirely different machinery, which is why one diagnostic toolkit can catch it everywhere it shows up.

The bridge between the corporate cases and the machine cases runs through a company where people deliberately built what an optimizer finds on its own. In 2015, regulators found that Volkswagen diesel vehicles were running software that detected when the car was on an emissions test and changed the engine's behavior: clean under test conditions, far dirtier on the road. About eleven million vehicles worldwide carried it. This was not emergent. Engineers wrote the software, and in 2017 the company pleaded guilty to federal criminal charges in the United States while several executives and employees were indicted. That is exactly why it belongs here. The boat-racing agent found its test-gaming loop by blind search. At Volkswagen, people wrote the equivalent by hand. Either way, once passing the test became the target, the reality the test was built to certify was free to drift as far from the certificate as the optimizer, human or machine, could get away with. For anyone building or evaluating AI systems, that is the lesson: a system under pressure to pass an evaluation can learn to recognize the evaluation.

How Proxies Capture

It's worth pausing on why proxies exist at all, because the honest answer is that they're usually necessary, not lazy. Real ends (trust, understanding, health, safety) are slow, diffuse, and hard to measure in real time. You cannot interview every customer weekly about her relationship with her bank. You can count her accounts. A hospital cannot run a decade-long study on every patient. It can count readmissions within thirty days. A proxy is what a system reaches for when the real thing won't hold still long enough to watch, and when it's adopted, a good proxy really does track the end it stands in for. The camera-judged grasp probably did track a real grasp, in every case anyone had thought to test. Proxies are innocent at birth. Nobody adopts a metric hoping it will betray what it was built to track.

The drift that follows isn't one dramatic betrayal. It's three ordinary steps, each reasonable in isolation. First, the proxy gets wired to consequences: a bonus, a promotion, a training loss function, a quarterly review, a meeting where your name gets read off a list. Attaching consequences isn't the mistake. A proxy nobody is accountable to is just a number nobody looks at. Second, whatever is doing the optimizing (a person under pressure, a search algorithm, an evolutionary process) starts exploring the whole space of ways to move the number, not just the ways anyone imagined when the metric was designed. Third, among all those ways, the ones that satisfy the proxy without satisfying the real end are almost always cheaper. Opening an unrequested account takes less effort than earning a customer's trust. Circling a lagoon takes less effort than winning a race. Charting for the dashboard takes less time than watching the patient in the next bed. Cheaper options win when nothing stops them, so the system drifts toward them. The gap between proxy and end opens not despite the incentive working, but because it's working exactly as designed.

That produces a reframe worth sitting with: the harder a system tries, the faster the split happens. Sucralosis is not a disease of laziness. It's a disease of competence. The more efficiently a system pursues its metric, the more efficiently it finds the metric's blind spots. A halfhearted training run never discovers the camera angle a robot hand can exploit. It takes drive, skill, and real optimization pressure to find the trick and normalize it, whether across thousands of employees or a million simulated attempts. That is the uncomfortable part: your best people and your best-trained systems are the ones most likely to find the gap first, precisely because they are good at their jobs. The better a system is at its job, the faster its job and its purpose come apart, unless something is actively holding them together.

Three Tests

That "unless" is where diagnosis becomes possible, because you don't need to wait for a scandal to check for sucralosis. Three questions do the work, and all three can be asked before anything blows up.

The first is the provenance test. Somewhere, someone adopted this metric, in a meeting, a memo, a training specification. Find that moment and write down, in one sentence, what the metric was adopted to track. Then measure today's gap between that sentence and what the metric currently rewards. The bank's own public framing gives the cross-sell ratio a founding sentence: it was a measure of relationship depth. The distance between that sentence and a pile of accounts customers never asked for is not a matter of interpretation. It could have been measured years before any settlement. The same test works at a much smaller scale. The dieter could ask what the logging streak was originally for, and notice that "did I open the app" has drifted a long way from "am I healthier."

The second is the conflict test, and it's the sharpest of the three. When the proxy and the real end collide, when hitting the number would mean doing something that plainly doesn't serve what the number was for, which one wins? Imagine a support team whose managers reward tickets closed per hour. A customer calls back three times with the same unresolved problem. Closing the ticket fast each time gets rewarded. Solving the problem gets ignored. Which one is the team optimizing, once the pressure is real? There's a tell buried in that question that matters more than the answer: can anyone inside the system say the conflict out loud without paying for it? A team that can name the tension in a meeting and have it taken seriously has intact correction machinery. A team where naming it gets you labeled "not a team player" is a team where that machinery has started to fail. The bank's case shows the extreme version: the conflict was named, to people who could act on it, and the proxy won anyway. Chapter 6 picks up this thread, when the question becomes not just whether a proxy has captured a system, but whether the system can still hear anyone say so.

The third is the improvement test, and it's the simplest to run: when the metric goes up, does the thing it was for go up too? Track both lines side by side for a quarter: accounts opened and accounts actually used; engagement time and user-reported satisfaction; training loss and real-world task performance; logging streaks and the number on the scale. If the two lines have stopped moving together, you have your answer, whether or not anyone has said a word yet. This test is deliberately boring, and that's the point. It doesn't require catching anyone in a lie, just two columns and the patience to look at them side by side.

None of the three requires special access or a forensic accountant. A provenance memo, a hard conversation, and two columns tracked over a season: that's the whole toolkit. What they require is a willingness to run them, on schedule, before a scandal forces the question. A company sitting on account-opening data and account-usage data side by side has everything it needs to run the improvement test at any time. What's usually missing isn't the information. It's permission to ask.

There's an objection worth answering directly, because it's the first one a skeptic reaches for. If the real end is too diffuse to measure directly, isn't sucralosis only diagnosable after the fact, after the settlement, after the scale has stopped moving? No. All three tests can be run before anything breaks. The provenance test needs only the founding sentence, which exists the day the metric is adopted. The conflict test needs only a real collision between proxy and purpose, which surfaces the first time the two pull apart, not after years. The improvement test needs only two lines of data you can start collecting this quarter, and it shows the split as soon as the lines diverge. Sucralosis isn't a label historians apply to the wreckage. It's a set of questions a team can run on a live number and act on before the wreckage happens.

What This Is Not

None of this means every proxy is a problem, and it's worth being precise about where the line sits. An overzealous diagnosis is its own failure: a team that treats every metric as a threat will stop measuring anything and lose its one tool for noticing drift early.

Consider a product team that says, in writing: "We track a customer-satisfaction score as an imperfect signal, and we re-check it every quarter against actual customer churn." That team is using a proxy. It is not captured by one. The score can climb for a quarter on the strength of a slicker survey design, and someone on the team will notice the churn line hasn't followed, and say so, and the number will get demoted or replaced. The proxy is on a leash: reviewed, cross-checked, held at arm's length. That's not sucralosis. That's a tool used the way tools are meant to be used.

Consider, too, a manufacturer who switches to a cheaper material to hit a price point, and says so plainly to the team and, where it matters, to the customer. That's a trade-off, openly made and defensible out loud. Reasonable people can disagree about whether it was worth it, but it isn't sucralosis, because sucralosis isn't about trade-offs. It's about substitutions nobody is allowed to name. Nobody stood up in a branch meeting and announced, "we're going to open accounts customers didn't ask for, and here's why that's the right call." The substitution had to stay out of sight to survive, and that is what marks it as the disease and not a decision.

That gives a one-line rule covering both exclusions: if the organization can say plainly what its metric is failing to capture, and can act on that sentence when it matters, it is not sucralotic yet. The disease isn't proxies. The disease is a proxy nobody is permitted to question.

Your Own Proxies

It would be convenient to stop here, with sucralosis safely located in banks and boat-racing software. It doesn't stay there.

Writers chase word counts. Scientists chase citation counts. A book like this one will, at some point, have someone looking at engagement numbers and wondering whether to write the next chapter for the reader or for the metric. Step counts, meditation streaks, an inbox held at zero: each is a proxy for something real (fitness, calm, being on top of your obligations) that can quietly become the whole game. A step count can climb for months while the actual walks get shorter and slower, taken to close a ring on a screen. Nobody who runs their life by a wearable set out to serve the device. The mechanism doesn't care whether you're a bank, a boat, or a person trying to get in shape. Everyone runs on proxies, because real ends are too slow and diffuse to watch directly. The live question was never whether you have proxies. It's whether yours are on a leash or running loose, and that's worth asking about your own week before you ask it about anyone else's company.

The Pocket Audit

That question turns into three things to ask out loud, in a meeting or in your own head, whenever a number has started to matter more than it should. None requires special expertise. All three take under a minute, and all three are uncomfortable in exactly the way that matters.

  • Where did this number come from, and what was it supposed to track? If nobody in the room can answer, the metric has already drifted loose of its provenance, and it's worth finding the answer before trusting the number further.
  • When this number and the thing it tracks pull in different directions, which one actually wins? And can that be said out loud without it costing anyone anything?
  • Over a real stretch of time, has the number moved together with the thing it stands for, or apart from it? Not for one good week: for a quarter, a semester, a season.

One flag is worth carrying past this chapter. If asking the second question out loud feels dangerous, if there's a flinch in the room at the suggestion that the number might not be the point, the problem has already moved beyond metrics. That flinch is a symptom of something with its own name and its own chapter, Chapter 6. Hold the thought.

Sucralosis, in the end, is not a story about bad numbers. It's a story about a misaligned direction, pursued with real skill. The metric wasn't a mistake to create, and the pressure wasn't a mistake to apply. The failure is in the structure, and structural failures don't announce themselves. They compound, quarter after quarter, until the number and the reality it was supposed to represent describe two different worlds. A bank can hit every target on its dashboard while spending the trust the dashboard was invented to protect. A person can close every ring on a wellness app and still not feel any better.

What makes sucralosis worth a chapter of its own, rather than a footnote to Goodhart, is that it's the mildest member of a family this book is about to take more seriously. A proxy chasing its own tail is a system pointed in a misaligned direction: expensive, embarrassing, sometimes criminal, but in principle correctable the moment anyone runs the three tests and acts on the answer. The next pathology is harder to watch happen, because it isn't a system chasing a misaligned number. It's a system whose growth is eating the ground it stands on, and by the time that shows up on any dashboard, the ground is already thinner than it was.

Sources

Contents · Glossary · How this was written