---
title: "Stated Goals and Revealed Goals"
url: "https://toddpaulbrownjr.com/corrigibility/stated-and-revealed-goals/"
author: "Todd Paul Brown Jr."
description: "Every system has a goal it states and a direction it actually moves. How to find the second, and how to judge it without guessing at anyone's soul."
kind: "guide-chapter"
updated: "2026-09-26T02:50:19+00:00"
---

# Stated Goals and Revealed Goals

*What would change your mind?* Keep that question close. Every instrument in this book comes back to it, and the last pages will ask it of you.

A regional hospital installs a new dashboard. It is a good dashboard, built by people who care about patients: a screen in every department showing throughput (how fast beds turn over), the 30-day readmission rate (how many discharged patients bounce back within a month), and patient satisfaction scores (how people rate their stay). The mission statement on the wall by the front doors still says what it always said, something about healing and dignity and the whole person. Nobody changes the wall. Nobody needs to.

For the first year, the dashboard does exactly what dashboards are supposed to do. A unit with a rising readmission rate gets a look from the quality committee. A department with long turnover times gets an extra hire. The numbers are a flashlight, pointed at real problems, and the hospital gets a little better at the things the numbers can see.

By year two, something has shifted, and almost nobody can say when. A patient who needs one more night gets discharged instead, because the bed has to turn. When a patient comes back a few days later, the unit learns to hold her "under observation" rather than formally readmit her. On paper that is a different category, and the unit's number has consequences for staffing. At discharge, staff start mentioning how much the hospital's survey scores matter, and the survey starts measuring the goodbye as much as the stay. None of this required a decision. No committee met and voted to put the screen ahead of the patient. It happened the way water finds a slope. Every small choice, made under real pressure, ran downhill toward the number.

Nobody can point to the day the drift started. But somebody could point to the day the dashboard started, if you asked the right person. Some of those numbers were picked in a quality-committee meeting, because they seemed like the closest thing to a fair, comparable proxy for "is this hospital doing its job." The readmission number arrived partly from outside. The payers who fund hospitals like this one began tying money to it. (In the United States, Medicare began penalizing hospitals for excess 30-day readmissions in 2012.) Either way, each number was adopted on purpose, for a reason. Throughput stood in for efficient use of scarce beds. Readmissions stood in for whether a discharge had actually solved the problem. Satisfaction stood in for whether a frightened person had been treated like a person. Each was a reasonable stand-in for something real. That is worth remembering, because it means the dashboard was never the enemy of the mission. It was supposed to be the mission's flashlight. The trouble is what happens to a flashlight when the room starts organizing itself around staying in the light.

Ask a nurse on a bad day, off the record, what the hospital is organized around now, and you may hear something like this: the dashboard is the patient. Not because anyone believes that. Because that is what the floor has learned to act like, shift after shift.

Now leave the hospital and go somewhere that seems to have nothing to do with it: a technology company building a recommendation engine, the software that decides what shows up next in your feed. The engineers did not set out to build a machine for provoking people. They set out to solve a real problem, an ocean of content and a finite human attention span, and they needed a number to tell them whether they were solving it. They chose engagement: time spent, clicks, replies, shares. Engagement was never supposed to be the point. It was a rough, measurable stand-in for something harder to measure: whether people were finding the feed worth their time.

The system does what these systems do. It runs experiments, millions of them, adjusting what it shows and watching the number. Somewhere in that process, without any engineer writing it down as a strategy, a pattern turns up. Research on social media has found that posts carrying moral and emotional language tend to spread further than posts without it, and an optimizer hunting for engagement can find the same thing on its own. So can it find the thing just short of outrage: the video that almost resolves, the argument that never quite lands, the notification that might be good news or might not. None of this is a plan. It is a correlation, surfaced by an optimizer doing what it was told to do: raise the number. The dashboards inside the company go up and to the right. The quarterly review is good.

Somewhere inside that company, an engineer probably notices, and maybe says something in a meeting: the number is going up, but is the thing it was supposed to stand for going up with it? The honest answer is usually a shrug. Nobody kept a separate measurement of "did this make people's lives better," and the number that was kept is sitting right there, trending the right way, easy to report and easy to defend. The question isn't rejected, exactly. It just has nowhere to land, because the only instrument in the room is the one built to measure the proxy.

Put these two stories side by side and the resemblance is hard to miss, even though a hospital ward and a recommendation engine share no moving parts. A committee of doctors is not a training loop. A discharge note is not a gradient update. But look only at the shape. In both cases, a system built to serve a real, hard-to-measure purpose adopted a proxy for it, optimized the proxy with real competence, and drifted from the purpose without anyone deciding to drift. It is the same failure shape, showing up in entirely different machinery. That shape is what the rest of this book is about.

## The Gap

Here is a way to state what just happened that does not depend on either story's details. Every organized system (a hospital, a company, a piece of software, a government agency, a friendship) has two directions worth asking about. One is the direction it announces: the mission statement, the charter, the thing you'd read on the wall or in the onboarding deck. The other is the direction its actual operation pushes toward. You see that one not in what the system says but in what it measures, rewards, punishes, and protects when something has to give.

This second direction is what this book calls a system's *telon*: the direction revealed by what a system does, under real pressure, regardless of what it says about itself. The word is new on purpose. "Goal" already means too many things, and most of them point at intention, at what someone meant to happen. A telon is not what anyone meant. It falls out of a system's structure: its incentives, its feedback loops, what gets rewarded and what gets quietly punished. A thermostat has no intentions, but it has a very clear telon. It moves the room toward a target temperature, and you find that target by watching what it does, not by asking it. Every larger system has something like that target buried in its design, whether or not anyone chose it, and whether or not anyone running the system could describe it.

Consider a sales team whose commissions pay out on signed contracts, with nothing tied to whether the customer renews next year. The company's stated goal is happy, retained customers. Its telon is signed contracts this quarter, by whatever means close the deal. Nobody in leadership wrote "get customers to sign things they'll regret" into a strategy document. Nobody had to. The commission structure did the writing, and the sales team, acting rationally inside the structure it was given, followed the incentive rather than the slogan. That's a telon: not a secret intention, but the direction that falls out of the machine once you stop reading its paperwork and start watching what it pays for.

There is a well-known rule for finding it. The management cybernetician Stafford Beer gave it its best-known form in nine words: "The purpose of a system is what it does." People shorten this to POSIWID. In this book it works as a rule of measurement, not a definition. It tells you where to look: watch the system, don't interview it. A thermometer tells you the temperature. It does not tell you what temperature the room is supposed to be.

That distinction matters more than it looks, so it is worth being blunt about it here, before the habit of collapsing the two can form. If POSIWID were the whole truth, if a system's purpose simply *were* whatever it does, then the hospital would not have a problem. It would be a throughput machine, correctly executing its purpose, and the dashboard would be the mission, accurately described. There would be no gap between what the hospital does and what it is for, because those would be the same sentence. The recommendation engine would not be a proxy that drifted. It would simply be an outrage machine doing its job.

Nobody who has stood in that hospital believes that. The nurse who says "the dashboard is the patient" is not describing the hospital's purpose. She is describing a failure, a gap she can feel because she still remembers what the wall says and still, most days, wants to live up to it. Watching what a system does shows you where its dynamics push now. It says nothing about where they are supposed to push. That second half is where the rest of this book lives, in the space between the system revealed by its behavior and the system it was supposed to be.

## What It's For, Without Guessing

So what is a system for? This question has a bad reputation, and it has earned it. "What is this organization really about, deep down" lets anyone answer however their priors want. A diagnostic tool that only ever confirms what you already believed is not a diagnostic tool. It is a mirror with extra steps.

The way out is to refuse the question "what does it really want" and ask three narrower, checkable ones instead, in a fixed order of authority. You reach for the next source only when the one before it runs out.

The first and strongest source is the constitutive mandate: the actual document a system was built or chartered to serve. The hospital has a founding charter and a mission statement adopted by a board on a real date. A public agency has the statute that created it. A machine-learning model has a training objective, written into code, that someone can point to. This source ranks first because it needs the least interpretation. You don't guess what a charter says. You read it. If the hospital's charter says its purpose is patient healing, that settles what the hospital is for, whatever the dashboard says about what it does.

The second source is proxy provenance. When a system adopts a metric, it does so for a reason, in a specific room, at a specific time, to track something. The hospital's readmission metric was not always on the wall. It arrived because someone, a committee or a payer, decided that readmissions were a reasonable stand-in for "did we actually fix what we said we'd fix." That decision is recoverable. Metrics don't fall from the sky. They get adopted, for stated reasons, and the reasons usually leave a trail. The measure of drift, later, is the distance between what the metric was adopted to track and what the system now does to move it. That distance is not a feeling. It is measurable, in principle, by anyone willing to do the archival work. This source does the most work in this book, because most systems worth diagnosing have a charter too broad to settle the hard questions and a metric history specific enough to.

The third source is the last resort. Where no charter exists and no adoption record can be found, a system's proper direction has to be bounded by behavior. You ask whether its current orientation sustains the kind of ongoing correction that keeps a system answerable to what it depends on. Call this the coevolution criterion for now. It gets its full treatment in Chapter 5.

Notice what the ordering buys you. A charter beats a metric history because a charter was written to state a purpose, while a metric was only chosen to track one. A metric history beats behavioral inference because a documented decision beats a pattern read from outside. Each step down the list trades certainty for availability. You use the weaker source only when the stronger one has nothing to say, never because it's easier to reach.

What none of the three permit is an appeal to what the system "would really want if it were honest with itself," untethered from a charter, a metric history, or an observable pattern. That move is the most tempting error in this kind of work: deciding in advance what a system's soul is, then treating every piece of evidence as confirmation. The three-source standard is a discipline because it refuses that move. If you can't point to the charter, the metric's adoption, or the coevolution pattern, you don't get to claim you know what the system is for. You get to say you don't know yet.

## The Day You Win

One more piece of the definition, easy to state and easy to miss in practice: a telon has direction, but no finish line. It is a vector, not a destination.

"Serve patients' health" is not something a hospital finishes. There is no Tuesday on which the mission is complete and the staff disperse to other work. A healthy hospital's telon is a heading, the way a ship holds a bearing rather than sailing to a point where it stops. When the direction its incentives actually push is toward patients' health, day after day, you can call that a *persistent telon*: an ongoing orientation with no completion condition. Success for a telon like that is not arrival. It is whether the system is still tracking, or has started to drift.

Contrast that with the targets that do have a finish line. Build the new wing. Pass a specific bill. Cure a specific disease. Hit a sales number by a specific quarter. Call these *terminal goals*: achievable end-states a system can reach, after which the target simply stops pulling. Most healthy systems run on both at once. A hospital pursues terminal goals (open the wing, meet the certification deadline, cut a specific infection rate below a specific threshold by a specific date) in service of a persistent telon that doesn't go anywhere when the wing is built. The terminal goal is a project. The telon is what the project was for.

Notice where the dashboard sits. Its numbers never finish the way a project does. Throughput and satisfaction just keep reporting, quarter after quarter, always available to be pushed a little further. A milestone, once hit, stops pulling. A number on a dashboard never stops pulling, which is exactly why it can quietly take over the job of the ongoing heading without anyone noticing the handoff.

The opposite failure is just as instructive. Sometimes a system's whole sense of direction collapses into whichever terminal goal is in front of it. The persistent "what for" goes missing, and the only organizing force left is the current fight. A system in that condition faces a structural problem on the day it wins, because the thing holding it together disappears at the moment of its greatest success.

A well-documented version of this involves a charity founded in 1938 as the National Foundation for Infantile Paralysis, now known as the March of Dimes. It began with about as terminal a goal as an organization can have: defeat polio. For seventeen years that goal organized everything: the fundraising drives, the research grants, the volunteer chapters across the country. Then, in April 1955, the results of the field trial of Jonas Salk's vaccine were announced, and the goal that had organized the whole institution was, for practical purposes, met. Polio did not vanish overnight, but the organization's reason for existing, find and fund a cure, had been answered.

This is the moment that breaks many goal-organized institutions. Picture the more common ending first. Local chapters built around a single fight don't know what to do once the fight is over. Volunteers who spent years knocking on doors have no obvious next door. Fundraising letters that used to name a specific enemy have nothing specific to say. An organization in that position usually does one of two things. It dissolves, its machinery scattering to other causes. Or it keeps running on momentum, chasing whatever fight is nearest, no longer quite sure why.

This organization did neither. In 1958 its leadership turned the whole structure toward a new, ongoing commitment: preventing birth defects. In 2003 it added premature birth. The same donor base, volunteer networks, and fundraising machinery were redirected toward child health, a heading with no single finish line. The sociologist David Sills studied the organization in his 1957 book *The Volunteers*, and organizational theorists use the term *goal succession* for this kind of reorientation. It is hard enough to deserve its own name. Many institutions built around a terminal goal don't manage it. They dissolve when the goal is met, or, the more interesting failure, they quietly substitute a new terminal goal without admitting the substitution, and keep acting as though the fight is the point, because for years the fight was the only thing they knew how to be.

Movements built mainly around opposing something face a starker version of the same problem. When the thing they opposed is gone, an identity built on being against does not automatically know what it is for, and it may go looking for a new opponent, because opposing is the only muscle it has exercised.

Both stories point at a diagnostic question worth holding onto for the rest of the book: *What happens to this system on the day it wins?* A system with a healthy shape has an answer, because its terminal goals have always served something ongoing that survives any single victory. A system in the warning shape has no answer, because its persistent direction has collapsed into the current fight. A system like that is one victory away from a crisis it will not see coming, because from the inside, winning never looks like the moment to worry.

## What This Isn't

This vocabulary is easy to over-apply, so it's worth saying plainly: most systems, most of the time, are fine. A hardware store whose stated purpose is selling hardware, and whose daily operation is selling hardware, has no gap between stated goal and telon. There is nothing to diagnose. A coffee shop that says it wants to serve good coffee, and spends its days serving good coffee, is not a candidate for any chapter that follows. That describes most organizations, most days, on most questions. This book is not a lens for finding hidden corruption everywhere. It is a lens for the specific, real, and common cases where a gap has opened. A tool that finds a crisis in every system is not measuring the systems. It's measuring the tool.

It is also worth being honest about how these gaps usually open, because the honest account is less dramatic and more useful. A stated goal drifting from a revealed one is rarely a lie. Almost nobody decides to say one thing while building toward another. More often the gap opens slowly, in a thousand individually reasonable decisions, each made under real pressure, none watched closely enough to notice the accumulating drift. The unit that reclassifies a returning patient is protecting its staffing, one case at a time. Chapter 4 is built around naming this pattern properly and asking what it takes to catch it early.

## The Instrument

What this chapter hands you is three questions you can run on any system you're part of this week: a workplace, a committee, a piece of software you use, an organization you volunteer for. None requires special access.

1. What does this system actually measure, and what does it actually reward, in practice and under pressure, not on paper?
2. For each of those measures, what was it originally adopted to track? Can anyone say, or has that history gone missing?
3. When the measure and the mission conflict, and eventually they will, which one wins? Can anyone in the system say so out loud?

Run these on the hospital and you get the chapter back in miniature. The dashboard measures throughput, readmissions, and satisfaction. Each was adopted to track a real piece of the mission. Under pressure, the dashboard wins, quietly, most of the time, while almost nobody says so on the record. That last clause carries more weight than it looks. Whether a gap can be named out loud inside a system, without cost to the person naming it, turns out to be the difference between a system that can repair itself and one that cannot. That question, not whether a gap exists but whether the system can hear about it, is the thread that runs through the rest of this book. Chapters 6, 8, and 9 take it up once the machinery for measuring it is in place.

## Close

The gap between what a system says it's for and what it actually does is ordinary. It opens in hospitals and in code, in charities and in friendships, in almost any system built by people under real constraints. In most cases it is measurable and repairable, provided the system can still hear about it. Some systems can. The March of Dimes could, at the exact moment its old purpose ran out, and it built a new one before the void had time to do anything worse. Some systems can't, and by the time you can see that they can't, the gap has usually stopped being the interesting problem. The interesting problem, by then, is why the system can no longer hear. That's where this book is going.

## Sources

- Stafford Beer, "What is cybernetics?", *Kybernetes* 31(2), 2002, pp. 209–219. https://www.emerald.com/insight/content/doi/10.1108/03684920210417283/full/html
- Wikipedia, "The purpose of a system is what it does" (accessed 2026-09-25). https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does
- Centers for Medicare & Medicaid Services, "Hospital Readmissions Reduction Program (HRRP)" (program began October 1, 2012). https://www.cms.gov/medicare/payment/prospective-payment-systems/acute-inpatient-pps/hospital-readmissions-reduction-program-hrrp
- William J. Brady et al., "Emotion shapes the diffusion of moralized content in social networks," *PNAS* 114(28), 2017. https://www.pnas.org/doi/10.1073/pnas.1618923114
- March of Dimes, "History of March of Dimes" (founding 1938; 1958 birth-defects mission; 2003 Prematurity Campaign). https://www.marchofdimes.org/about-us/mission/history/history-march-dimes
- Encyclopaedia Britannica, "March of Dimes Foundation." https://www.britannica.com/topic/March-of-Dimes-Foundation
- David L. Sills, *The Volunteers: Means and Ends in a National Organization*, Free Press, 1957.