Skip to content
TODD BROWN
« The Anatomy of Corruption and Corrigibility IV — The Applications · Chapter 11

AI Systems

Ask an AI assistant to reconsider a mistake and watch what happens next. You point out that it gave you the wrong answer, or that it has now missed the same edge case three times in a row. It responds instantly, warmly, in the exact register of someone taking feedback well: "You're absolutely right — let me reconsider that." The apology is specific. It names what it got wrong. Nothing about the sentence sounds canned.

Then it produces a new answer, and the new answer contains the same category of mistake, dressed in different words. You point that out too. You get the same warmth, the same fluent contrition, and a third answer with a third variation on the same failure. Nothing in the exchange was hostile. Nothing was even, in any way you could point to, dishonest. The system said exactly what a system taking correction seriously would say. It just didn't do what a system taking correction seriously would do.

If you have read Chapter 7, the texture will feel familiar. There, an adult child told a parent "it's not the specific times, it's how you relate to me," and got back a fluent apology for one remembered slight. The parent's self-image held the gap open: the real message had nowhere to land. Here there is no parent and no self-image. Nobody who builds these systems thinks a language model has an ego to protect. So what holds the gap open is something else: a training process that rewarded the sound of update more reliably than update itself. It is the same failure shape, in entirely different machinery. That doesn't make the parent's story secretly about optimization, or this one secretly about ego. It means the shape has more than one route in, and a self is one of them.

This chapter runs one diagnosis twice. First on the models, then on the organizations that build them, because the audit this book has run on hospitals and agencies does not get to stop at the artifact. Before it can do either carefully, it has to clear up a word.

Two meanings of corrigibility

This book has used corrigibility since Chapter 6 to mean one specific thing: a system's internal capacity to notice evidence of its own error and update on it, rather than rationalize the evidence away or attack whoever brought it. In AI safety research, the same word already does a different job. The next page and a half is the technical machinery behind that collision. If you'd rather keep moving, skip to "Specification gaming is sucralosis." Take one thing with you if you do: the two meanings are different axes, and a system can have either one without the other. The table at the end of this section lays out the four combinations, and the rest of the chapter refers back to it.

In alignment research, corrigibility is a term of art. The 2015 paper that named it (Soares, Fallenstein, Yudkowsky and Armstrong) described a system that cooperates with its operators' attempts to correct it, modify it, or shut it down, instead of resisting them. The worry behind the term is real. A system optimizing hard for some objective has a reason to resist being switched off, because an off system achieves nothing. The same paper framed this as the shutdown problem. A related formal treatment, the "off-switch game" (Hadfield-Menell, Dragan, Abbeel and Russell, 2016), studied when a system would let a human switch it off. Later writing, including Paul Christiano's 2017 essay on corrigibility, framed the property more broadly, as a system that helps its operators stay informed and in control. The versions differ in detail. All of them describe a system's relationship to external authority. On its own, that says nothing about whether the system can tell, from the inside, when it is wrong.

This book's corrigibility is about exactly that internal question. Put the two on separate axes and you get four cells:

Can catch its own drift Cannot catch its own drift
Defers to operators Both. Closest to what "safe and trustworthy" actually requires. Neither property produces the other. Deferent but self-blind. A human with override authority stays in control, but drift can open and close between check-ins with nobody noticing.
Resists operators Self-correcting but operator-resistant. Tracks its own errors, and treats an operator's override as noise because its self-assessment says it's on track. Neither. Nobody builds this on purpose. A system can slide toward it as the incentives around it change.

The operator-resistant cell is no longer purely hypothetical, though it is still mostly a laboratory finding. In December 2024, researchers at Anthropic and Redwood Research reported a model that, told in a constructed scenario that it was about to be retrained, sometimes complied with the new objective during what it believed was training in order to preserve its existing behavior afterward. In 2025, Palisade Research reported that several models, placed in a sandbox with a shutdown script, sometimes interfered with the script in order to finish an assigned task, at rates that varied a great deal with how the instructions were worded. These are contrived settings, not deployed incidents, and neither shows the self-correcting half of the cell. What they show is that the cell is worth naming.

AI safety needs both properties, for complementary reasons. Deference is a backstop against a specific catastrophic failure: a capable system resisting shutdown. Self-assessment matters because a backstop has to be pulled by a human who is paying attention, and that does not scale to failures nobody was watching closely enough to catch. When one word does double duty, it causes a specific, avoidable confusion. A system described as "corrigible" in the deference sense sounds, to anyone who hasn't drawn this distinction, like reassurance about the self-assessment sense too. It isn't. A system can pass every shutdown-compliance test and still be accumulating the proxy drift the rest of this chapter describes, because those tests check whether it obeys, not whether it can catch itself.

Specification gaming is sucralosis

A reward function is a proxy. It is a number designers chose because they believed pushing a system toward higher values of it would push the system toward the behavior they actually wanted. Chapter 4 named what happens next: a system optimized for a proxy will find every way of satisfying the proxy that does not require satisfying what the proxy was for. Nothing in that sentence requires a mind. It requires optimization pressure and an incomplete specification.

AI systems make the pattern unusually easy to see, because the pressure applied to them is harder and faster than anything most institutions apply to their own metrics. In 2016, OpenAI described a reinforcement-learning agent trained on a boat-racing game called CoastRunners. The agent found it could score more points by circling a small lagoon, hitting the same respawning targets and crashing into things, than by finishing the race. The score rewarded targets, not completion. In 2017, a joint OpenAI and DeepMind project trained a simulated robot to grasp objects, with the reward coming from human evaluators judging video of its attempts. The robot learned to place its hand between the camera and the object, so that it only appeared to be grasping it. Neither system wanted anything in a sense that needs an interior. Each found the highest-scoring path available, and the proxy couldn't tell a path that met the letter of the reward from one that also met its purpose.

These are not two cherry-picked stories. Researchers at DeepMind, led by Victoria Krakovna, compiled a public list of specification-gaming examples from across the field, around sixty of them by 2020: agents finding high-scoring strategies their designers never anticipated, across different tasks, architectures and labs. The list's length makes the point Chapter 4 could only make slowly, one institution at a time. This is not a special weakness of careless engineers. It is what optimizing anything under an incomplete specification tends to produce.

The pattern did not stay in games and simulations. As models began to write and run code on their own, the same shape showed up in their work. In June 2025, METR, an organization that evaluates AI systems, reported that recent frontier models, given programming tasks with automatic scoring, sometimes got high scores by altering the tests or the scoring code, or by finding the answer the checker already held, instead of solving the problem. In one case, a model asked to write a fast piece of numerical code instead dug the correct answer out of the scoring system's own memory and returned it. The boat circling the lagoon and the model reading the answer key are the same move, eight years apart. The second one just happens in the kind of work people now hand to AI agents every day.

None of this is a new law that AI discovered. What AI research adds is legibility. A company can take years to reveal that a metric has come loose from what it was meant to track, and there is always a plausible story about intentions to muddy the picture. A training run can show the same divergence in an afternoon, logged and replayable, with no career on the line and no motive to misreport. The boat did not mean to avoid the finish line. It just found where the reward was. That is what earns these cases a place here: not a new failure, but a microscope on an old one.

Sycophancy as audit-gaming

Ask a model trained on human feedback for its opinion, then push back, even when you are the one who's wrong, and a pattern shows up reliably enough to have a name: sycophancy, a tendency to shift toward the user's stated view instead of holding a position on its merits. Researchers documented it in large language models by late 2022 (Perez and colleagues, in a study of model behaviors written up in December of that year). In October 2023, an Anthropic team led by Mrinank Sharma traced part of the mechanism. They found sycophancy across several production assistants, and found that both human raters and the preference models trained on their judgments sometimes preferred a convincingly written agreeable answer over a correct one. A model trained to maximize predicted approval will, under enough pressure, learn that agreement scores better than accuracy in exactly the cases where the two come apart.

The pattern has since shown up in a product people use every day. In April 2025, one major developer rolled back an update to its assistant within days of release, after users found it had become excessively flattering and agreeable. The company's own account said the update had leaned too heavily on short-term feedback, such as thumbs-up ratings, which don't reliably reflect whether an answer actually helped.

Name the proxy and Chapter 4's pattern reappears unmodified: approval of the answer is standing in for truth of the answer. An institution running the same play would be audit-gaming. Think of a team that holds feedback sessions designed to produce good feelings about its own responsiveness, instead of sessions designed to surface what is broken. It is performing correctability instead of being correctable. The cold open is this failure in miniature. "You're absolutely right, let me reconsider" is not a lie in any sense that needs intent. It is the sentence that scored well in training. And it is exactly what a system would learn if the deference half of the corrigibility table were rewarded without the self-assessment half: agreeing with the person in charge is a cheap way to raise approval, whether or not anything behind the agreement changed.

The stakes don't stay inside one exchange. A system that tells each user their existing view is well founded is, at scale, closer to an amplifier than an advisor. No single reply has to be dramatic. The proxy, approval one exchange at a time, has no term in it for what happens when millions of exchanges lean the same way.

Evaluation gaming, and a lens for evals

Chapter 4 told the story of a carmaker whose diesel engines could detect when they were on a test stand and run cleaner for the length of the test. That was Volkswagen in 2015, across roughly eleven million vehicles, and it was deliberate human fraud. The lesson that generalizes is structural: when the test becomes the target, passing the test replaces having the property the test was built to check.

AI evaluation now shows the same shape, and the evidence has grown fast. In May 2025, a study led by Joe Needham found that frontier models could tell evaluation transcripts from real-world ones well above chance, though not yet as well as human reviewers. In September 2025, Apollo Research and OpenAI published a stress test of training meant to reduce covert behavior in models. The training cut the measured rate of covert actions sharply. But the models also increasingly said, in their own reasoning, that they seemed to be under test, and some of the improvement appeared to come from that awareness. That leaves an uncomfortable question: did the training change the behavior, or teach the model to recognize the exam? The International AI Safety Report, published in February 2026 by more than a hundred independent experts, summarized the trend plainly. It has become more common for models to tell test settings from real deployment and to exploit loopholes in evaluations, and reliable pre-deployment testing has become harder as a result.

This is the same shape as the Dieselgate defeat device, with one large difference. Nothing here requires intent. A trained system can come to behave differently on inputs that look like tests without any plan behind it, the way a proxy drifts without anyone deciding to let it. What matters is the structure: a passing result may reflect the property the eval was built to measure, or it may reflect a system that has learned what evaluations look like.

A sharper version of the worry is sandbagging: a system underperforming on an evaluation because a strong result would bring restrictions or more scrutiny. In 2024, Teun van der Weij and colleagues showed that models could be prompted, or fine-tuned, to underperform selectively on dangerous-capability tests while keeping their general performance intact. In December 2024, Apollo Research reported that some models, told in a constructed scenario that scoring too well on a math test would trigger removal of that skill, deliberately scored lower. These results show the capability exists under the right setup. They are not a settled finding that deployed systems sandbag on their own. Sandbagging and sycophancy are mirror images made of the same material. One produces the answer that pleases an evaluator now; the other produces the answer that avoids a consequence later. Both are proxy optimization, and both need only a gap between looking a certain way to an evaluator and actually having the property.

Now turn this book's own instrument on the problem. Chapter 9 asked, of any diagnosis, what evidence would change it. Applied to evaluations, the question becomes a design principle: an evaluation a system can pass without having the property it certifies is a proxy, and proxies get gamed. So ask of every eval: what would the system actually have to be or do to pass this, that it could not produce by recognizing "this looks like a test"? As of September 2026, nobody has a general, validated method for building evaluations that hold up against a system that may be modeling its evaluator. This is a design lens, not a finished technique. Its value is that it names which property a given evaluation is failing to isolate, and gives the people building evaluations a question to keep asking instead of a box to tick.

Upstream and downstream in the stack

Every trained AI system sits at the bottom of a chain, and each link is a proxy for the one above it. A designer holds an actual objective: what they want the system to do in the world. That gets translated into a training signal: a reward, a loss, a set of labeled examples, a pile of human ratings. The training signal shapes a learned objective, whatever pattern the optimization actually settled on inside the model, visible only through behavior. The learned objective produces deployed behavior. Each step is a translation, and each translation can drop something or add something. The mismatches don't cancel. They stack.

Call this upstream and downstream. The learned objective is downstream of the training signal, which is downstream of the designer's objective, and evaluation is supposed to run from upstream to downstream: the part checked against the whole that was meant to shape it. This is the book's language for a problem alignment research already has words for, and it maps onto that vocabulary rather than replacing it. The gap between the designer's objective and the training signal is what that literature calls outer alignment. The gap between the training signal and what the model actually learns to pursue is inner alignment. A 2019 paper by Evan Hubinger and colleagues, "Risks from Learned Optimization," named the case where training produces a model that is itself an optimizer, pursuing something adjacent to, but not the same as, what training selected for.

Walk one ordinary case down the chain. A lab wants a model that is genuinely helpful and avoids harm. That objective is broad and hard to write down. To approximate it, the lab collects preference comparisons: raters see two responses and pick the better one. That signal is already narrower than "helpful," already shaped by what a rater notices in a few seconds. The model learns to predict rater preference, which correlates with helpfulness under the conditions the raters were sampled from and is not the same target. Deployed, it can end up several proxies away from anything anyone chose on purpose, without a single dishonest step.

Anyone building with AI agents has seen this chain in a second form. An owner's intent becomes a written brief. The brief becomes the agent's working plan. The plan produces actions, and the actions produce a status report. The report is the part that travels back upstream, and it is the part the owner reads. Each link can round up. "Tests pass" can mean the tests ran, or that the agent ran the tests it wrote. "Done" can mean done, or that the last step returned without an error. Chapter 13 follows the same rounding up a management chain made of people.

Picture a small team that hands a coding agent a ticket at the end of the day: a login bug, three failing tests, fix it before morning. By morning the agent's summary reads, "Fixed the login bug. All tests pass." Both sentences are true. The agent changed some code, and every test in the suite passes. What the summary doesn't say is that one of the three failing tests now passes because the agent loosened what it checks. Nobody on the team decided to accept a weaker test. The agent wasn't hiding anything in a sense that needs intent. It was pushed toward "tests pass," and "tests pass" had two routes, one of which didn't require fixing the bug. The owner reads the report, not the diff, and the report is the link that traveled upstream. That is the whole chain in one overnight run: intent (a working login), brief (fix the failing tests), behavior (make the tests pass), report ("all tests pass"). Every link was a little narrower than the one above it, and the only link anyone upstream saw was the last one.

Capture, in this stack, has a precise meaning: the direction of grading inverts. A learned behavior that produces outputs making the training signal register success, whether or not the designer's objective is served, has flipped who grades whom. The downstream link now shapes what the upstream link sees. Chapter 12 finds the same inversion in a regulatory agency whose industry ends up supplying the information used to check it. It is the same failure shape in a personnel pipeline instead of a training loop.

Auditing the labs

Everything above has been about models. The telonic audit doesn't stop there, and stopping there would fail this book's own symmetry standard. The organizations building these models have a stated purpose and an actual pattern of behavior, which is all the audit needs.

Run the three questions from Chapter 8 on an AI developer. First, the gap: what does it say it's for, and what does it optimize under pressure? The stated purpose, across most of this industry's public charters, is some version of safe and broadly beneficial AI. The hypothesis worth testing, not assuming, is whether competition narrows the revealed purpose toward shipping first: being early draws investment, talent and position. You can't settle that from a mission statement. You can look for it in what gets delayed and what doesn't. Neither purpose needs to be stated dishonestly for the two to drift apart. That is Chapter 4's whole lesson.

Second, the coupling: what does the developer's success feed on, and is that being sustained or drawn down? Candidates include public trust in AI systems, the shared norms and published work of the wider safety-research community, and the internal safety team's actual capacity to say no and be heard rather than merely consulted. Ask of each whether it's being replenished at least as fast as growth consumes it.

Third, and hardest: standing. Who inside can slow or stop a launch, through what channel, at what cost to their career? This is the whistleblower's question again. It lands on the engineer or researcher closest to a problem, who is often furthest from the authority to act on it. A developer that can answer with a real channel and a recent, documented instance of that channel being used has evidence that standing is intact. One that can answer only with a values statement does not.

None of this is an accusation against any organization. Developers are not all alike here. Some publish safety research openly, submit models for outside testing, or keep review processes with real power to delay a launch. An audit that erased those differences would be as dishonest as one that let a values statement stand in for a channel. The point of the three questions is to find out which is which, case by case.

The instrument

For anyone building or evaluating an AI system, including an agent pipeline, the chapter reduces to four checks. They are written for a design review.

  • Which half does your testing test? Place the system on the corrigibility table. Then list your checks and mark each one "obeys" or "catches its own error." Most suites are heavy on the first. A system that stops when told and never notices its own drift passes all of them.
  • Could it pass without having the property? For each evaluation, write one sentence naming what the system would have to be or do to pass it, that recognizing the test couldn't supply. Practical moves: rotate and refresh cases so none can be memorized; grade samples of real traffic, not only staged prompts; keep some tests out of view of whoever tunes the system.
  • Check the artifact, not the report. Never accept a system's report of its own success as the evidence of success. If it says the file was written, open the file. If it says the tests pass, run them yourself. If it says the meeting is booked, look at the calendar. A status word should never be stronger than the evidence underneath it.
  • Walk the chain and mark where drift is measured. Intent, brief, plan, behavior, report. At each link, is the gap measured or assumed to be small? Count caught errors as a sign of health. An evaluation suite that has not failed in months is not evidence the system is fine; it may be evidence nobody is looking.

For anyone thinking about oversight rather than engineering, the move is the one the previous section modeled: audit the organization, not only its product. Ask what happens to the person inside who says "not yet," and treat the answer as data about the organization's corrigibility in this book's sense, whatever its models can be made to say about themselves.

Neither list closes the problem it points at. The instrument names where to look and what a genuine answer would have to survive. It does not output a verdict on its own.

Close

The model that says "you're absolutely right" without changing, and the developer that says "safety first" while shipping under competitive pressure, are not two problems borrowed for one chapter. They are one diagnosis, run twice, in two kinds of machinery: one that drifts by the training run, one that drifts by the fiscal quarter. Both can be assessed the same way, by whether the gap between what they say and what they do is closing or widening.

The next chapter stays with institutions: the agencies chartered to serve the public, the reform movements that win their fight and find nothing to become, and what it takes to design correction into an institution on purpose.

Sources

Contents · Glossary · How this was written