Reading this with an AI? Start at /for-agents/ (orientation) or /llms.txt (index).

Skip to content
TODD BROWN

Each term is defined the way the book uses it: operationally, with the test that tells you whether it applies.

Every entry ends with related terms: names other fields use for the same idea or a neighbouring one. They are there so you can find an entry from the word you already know. Related does not mean identical. Where the book's term is narrower or broader than the familiar one, the entry says how.

Telon

The direction a system's actual dynamics optimize toward: the orientation built into its structure, incentives and feedback loops, as distinct from its stated goals. A telon is a vector, not a destination; it has direction and strength but no finish line. To find one, watch what the system does, not what it says. The diagnostic question for any goal: what happens to this system on the day it wins?

Related terms for telon. Revealed preference, in economics: what an agent's choices show it wants, as opposed to what it says. "The purpose of a system is what it does", from systems thinking. Objective function and learned objective, in machine learning: what a model actually optimizes, as opposed to what its designers intended. Incentive structure. Mission drift and goal drift name the gap between a stated goal and a telon once it has opened. Telon differs from telos, a final purpose or end state: a telon is a direction with no finish line.

Sucralosis

A system optimizes a proxy for a real end until optimization pressure pulls the two apart, and it keeps optimizing the proxy. Named for sucralose: sweetness without nutrition. Reward without realization. The test: when the metric improves, does the thing it was for improve too? When they conflict, which one wins?

Related terms for sucralosis. Goodhart's law: when a measure becomes a target, it stops being a good measure. Campbell's law, the same observation about social indicators. Surrogation, in management research: mistaking the metric for the strategy. Proxy gaming, metric gaming, teaching to the test, vanity metrics, perverse incentives. In AI: specification gaming, reward hacking, proxy misalignment, and much of what is called AI misalignment, where a system pursues the measured signal instead of the intended goal. Sucralosis is narrower than Goodhart's law in one way: it names the condition of continuing to optimize the proxy after the two have visibly come apart.

Cancrity

Sustaining a system's own persistence by degrading the substrate it depends on. Competition is healthy when rivals sustain the conditions for each other's existence; it turns cancritic when one system's success is structurally decoupled from the health of what it feeds on. In one phrase: severed corrective coupling. A diagnosis has to name the substrate and show it being consumed faster than it's replenished.

Related terms for cancrity. Tragedy of the commons. Negative externalities. Extractive growth and unsustainable growth. Overshoot, in ecology: consuming a resource base faster than it regenerates. Parasitism, as opposed to mutualism. Race to the bottom. Killing the goose that lays the golden eggs. Burnout, when the substrate being consumed is a workforce. Cancrity is narrower than any of these: it requires that the system's own persistence depends on the thing it is degrading, and that the feedback which would have stopped it has been cut.

Corrigibility

A system's structural capacity to receive and integrate correction. Shown evidence of its own misalignment, a corrigible system updates; an incorrigible one rationalizes, suppresses or attacks the input. The hardest marker to fake: can the system say what evidence would change its mind? (In AI alignment, "corrigibility" usually means accepting correction and shutdown from operators. This book's sense is the internal capacity for honest self-assessment. Related, and both needed.)

Related terms for corrigibility. Error correction and feedback loops, in engineering. Double-loop learning, in organizational theory: questioning the goal, not only the method. Falsifiability, in science: being able to say what evidence would prove you wrong. Psychological safety: whether people can report a problem without being punished. Accountability and oversight. In AI: corrigibility in the alignment sense, and oversight or human-in-the-loop control.

Incorrigibility

The system can't detect or repair its own failure, but isn't defending it. Computing analogy: bit rot. The remedy is external: add the missing channel.

Related terms for incorrigibility. Blind spot. Silent failure. Missing feedback loop. Unknown unknowns. Single-loop learning: fixing errors without ever questioning the rule that produced them. Normalization of deviance, where a failure becomes invisible because it has become routine. Technical debt and bit rot, in software. The difference from contracorrigibility is intent in the structure: an incorrigible system lacks the channel; it is not defending against it.

Contracorrigibility

The correction machinery itself has been co-opted: attempts at correction are detected, absorbed or punished, and the repair tools report that all is well. Computing analogy: a rootkit, where the diagnostic tools are the compromised component. The remedy is to stop trusting self-reports and correct from outside the system's control. It works through four mutually reinforcing mechanisms: scope reduction, proxy displacement, identity capture and an enforcing environment.

Related terms for contracorrigibility. Regulatory capture and institutional capture. Whistleblower retaliation. Cover-up. Marking your own homework. A self-sealing belief system: one built so that any evidence against it counts as evidence for it. Defensive routines, in organizational theory. In computing: a rootkit, or a compromised monitoring system. In AI: evaluation gaming, sandbagging, deceptive alignment, and sycophancy, where a system learns to satisfy the check instead of the goal the check was for.

Anti-memesis

The property by which a critique can't be perceived by its target, precisely because of the identity capture that made correction necessary. The critique isn't rejected; rejection would require it to arrive. The behavioral sign: structural feedback gets converted into a list of incidents that can be fixed without touching identity.

Related terms for anti-memesis. Motivated reasoning. Identity-protective cognition. Cognitive dissonance. Willful blindness. The "few bad apples" explanation, which turns a structural problem into individual incidents. Antimeme, in internet usage: an idea that resists being noticed or remembered. Anti-memesis is narrower than denial: in denial the message arrives and is refused; in anti-memesis it does not arrive in a form the target can recognize.

Telonic audit

The institutional diagnostic. First, can the system receive the answer at all? Then three questions: what does it claim to be for, and what does it actually optimize? Is what it optimizes sustaining or consuming the people and resources that sustain it? And who has the standing, the information and the protection to ask, and what happens to people who do?

Related terms for telonic audit. Root cause analysis. Incentive audit. Governance review. Red teaming. Theory of change, when checked against what an organization actually does. "Follow the money." Comparing stated goals with revealed goals. In AI: alignment evaluation, and auditing a model or a lab against what it claims to optimize.

Grex, collegium, egregore

Three kinds of collective system, told apart by how they coordinate. A grex coordinates by local rules without reflection: a murmuration, a crowd, a price signal. A collegium coordinates by shared models, standards and distributed oversight; its defining property is that it can be wrong and knows how to find out. An egregore coordinates by shared belief and participation: money, brands, languages, ideologies. Participants sustain it, and it shapes their identities in return. Most real organizations are hybrids.

Related terms for grex, collegium and egregore. For grex: swarm behaviour, herd behaviour, emergent order, self-organization, a market price. For collegium: a profession, peer review, a community of practice, checks and balances, a standards body. For egregore: a social construct, a shared fiction, an imagined community, a brand, an ideology, a memeplex. Egregore is an older word from occult writing, used here without any supernatural claim: it names a collective that exists because people act as if it does.

Supporting ideas

Pattern identity: an institution is an organizing pattern, not a roster; it persists while every member turns over, and the pattern is what gets sick. Question-relative boundaries: the question you ask draws the system's boundary, so fix the question first. Upstream and downstream: a part is evaluated against the whole that constituted it; a captured system has inverted that, with the institution grading the public or the proxy grading the mandate.

Related terms for the supporting ideas. For pattern identity: the Ship of Theseus, organizational culture, institutional memory. For question-relative boundaries: system boundary, framing, unit of analysis, scope. For upstream and downstream: the principal–agent problem, accountability, "who watches the watchmen", and in AI the relation between a model and the people and mandate it is supposed to serve.