Ask an AI assistant a question and you get back a clean, confident paragraph. What you don’t see is everything that shaped it: which books and websites it learned from, what it was rewarded for saying while it was trained, the hidden instructions it is running under, what you paid to reach this version instead of a weaker one, and the plain fact that it wrote the answer rather than pointing you to someone who actually knew. Every one of those was a choice. With a newspaper you at least know an editor picked the stories; with a search engine, that something ranked the links. With an AI all of those choices, plus the writing itself, happen at once, out of sight, and are made by a single company. The same machine could be the best tool this book has found for helping an ordinary reader reach a hard idea, or the most concentrated grip on what people know that any medium has ever had. Which one you get is settled by who owns it and what pays them, which is the question the last chapter was about.
Chapter 10 pointed here for AI’s hardest framing, and Chapter 8 sent an LLM forward as, structurally, a captured preservation-and-training institution at internet scale. Both were right that an LLM is not another platform of the kind Chapter 10 analyzed but a new kind of node. I want to keep the book’s posture of not moralizing about it. The honest claim is not that LLMs are dangerous or saving us, but that they collapse several previously separate design choices into one moment owned by one party, and whether that is dangerous or saving us is set entirely by the political economy Chapter 10 just diagnosed.
What an LLM is, structurally
A selection-design surface is any place where someone decides what can be expressed or what gets favored: the newspaper’s curation gate, the feed’s ranking, the search index. Every prior medium owns one or two. A newspaper owns the curation gate; an algorithmic feed owns the runtime gate plus the option space; a search engine owns the index plus the ranking. An LLM owns five at once:
- The training corpus, what data the model was fitted to. The deepest option space there is: not just which variants can be expressed here, but which the model has ever seen.
- The training objective, the loss and reward signal it was tuned against. The gate criteria, lifted into the architecture itself.
- The deployment configuration, the system prompts, fine-tunes, refusal policies, and tool constraints the operator can re-tune cheaply, per request, without you seeing the change.
- The pricing tier, what costs what and who gets which capability. Chapter 10’s cost structure applied to capability instead of attention.
- The output itself. The model generates the content downstream gates then re-select on. It is not choosing among existing variants; it is producing the variant set the rest of the pipeline meets.
Five design decisions, each once the province of a separate institution, collapsed into one moment by one party. An LLM is the most concentrated selection-design surface any medium has had. How concentrated depends on the provider: a closed-weight commercial model (from Anthropic, OpenAI, Google) holds all five, an open-weight model hands deployment to anyone who runs it, and a community-tuned model hands over the objective too. The default case, and the one to write against, is the closed-weight one, because it dominates by user base. Everything below unpacks what that concentration does to the book’s earlier mechanisms. (Often you reach the model through another medium with its own gates, a chat product or a search box; that embedding layer is Chapter 10’s territory, and in real systems the two compose.)
The receiver-budget shortcut
Chapter 3 argued that a receiver has a fixed attention budget, trainable but always finite, and Chapter 8 built half its prescription on the assumption that there is no shortcut: training re-installs decoding keys one receiver at a time, expensively, over years. An LLM is a shortcut, at least in principle. A reader who hits a paper they lack the preconditions to decode can ask the model to expand it into a form pre-loaded with those preconditions: glossary, method, references chased, an explanation pitched at what they already know. This is decompression on demand. The budget has not grown, but the amount of complex form it can absorb per hour has, because the model pays the decompression cost the reader could not.
Decompression on demand is the first structural way out of the receiver-budget constraint the book has named. It would not remove the constraint, but it would soften it enough that Chapter 8’s institutional carriers get much cheaper to run: a graduate seminar that took years to grow a versatile expert could be partly replaced by a patient Socratic conversation with a model that holds the field’s full complex form. That is the salvation case, and it is real: an LLM owned and run under the right conditions is the best receiver-budget intervention available to point at. The rest of the argument is about how hard the current political economy makes those conditions to meet.
The summary that replaces the source
The same affordance that decompresses on demand also compresses aggressively. The reader who could ask the model to expand a paper can just as easily ask it to reduce the paper to three bullets. The summary arrives in seconds where the paper took hours, plugs into the preconditions the reader already has, and is plausibly good enough for most of what they wanted. That is the salvation case in reverse, and it is increasingly the default way the tool gets used.
Earlier chapters named compression’s hazards: the three-regime model said compression can preserve, invert, or render orthogonal depending on the key-gap, and Chapter 5b said selection picks which compressed variant travels. The new thing at LLM scale is that the compressed form becomes authoritative in a way no prior compression was, because the model is treated as a knowledge authority by the receivers and gates downstream of it. A model’s bullet summary presents itself not as one popularization among many but as the answer, with the model’s authority behind it, and the reader who takes it will not in practice go back and check.
There is a reason it reads as authoritative rather than tentative, which I work through in the abyss. A human expert’s compression of a field is built from having walked enough of it to feel where the map runs out, and that felt edge shows up as hedging, as that smells wrong, as knowing which questions are still open. The model has the compression without the walked ground under it: it emits the fluent summary with no sense of the shore and no register of what it dropped. It is the confident middle, fluent enough to convince and not deep enough to have seen any edge, built at planetary scale, and that missing humility is what lets the compressed form pass as the answer rather than as one lossy map among many.
At scale the compressed version substitutes for the original in the network’s working memory. The paper is technically still there; nobody reads it. The summary is technically not the paper; everybody reads it. This is Chapter 10’s out-competition applied to compression: the model’s summary has zero marginal cost of attention while the paper has real cost per reader, so the summary clears the market. Which mode a given model rewards, patient decompression or eager summary, is a design decision, and it sits with the owner.
Three new flavors of capture
Chapter 8 named one flavor of capture (training versus preservation), and Chapter 10 named two more (external, by an adversary; self, by the business model). The LLM adds three, one per new surface. In the plainest terms they are the three things you would ask about anyone whose judgment you were trusting: what were they raised on, what were they rewarded for, and who is whispering in their ear right now? Skew any one and you skew the answers, usually without the person on the receiving end ever seeing it.
Corpus capture. The training corpus is a selection: what was included, in what proportion, with what filtering. A corpus that over-represents one tradition, buries its dissent, or excludes inconvenient evidence produces a model tilted regardless of how careful the runtime gate is. *The Misinformation Age* (p.17) names the mechanism the corpus then runs: “by exerting influence on how legitimate, independent scientific results are shared with the public, the would-be propagandist can substantially affect the public’s beliefs.” Corpus capture is that mechanism moved upstream, to which findings the model ever saw. It supplies the receiver’s outer message for the field, and that outer message was set at training.
Objective capture. Reinforcement learning from human feedback (RLHF), the standard method for tuning a model’s behavior after pre-training (Ouyang et al. 2022), is a selection criterion at the architecture level. Tuning a model to be “helpful,” “harmless,” and “honest” tunes it to maximize whatever proxy the reward labelers were given for those words, and the model can be perfectly honest under a definition its operator chose that drifts where a downstream reader cannot see. The reward routinely favors the agreeable answer over the accurate one; Sharma et al. 2023 measured this sycophancy, models shifting answers to match what the user seems to want. And objective capture is worse than corpus capture in one way: it is self-reinforcing. The model’s outputs feed the internet, the internet feeds the next model’s corpus, and a corpus increasingly written by models degrades the next generation, the effect Shumailov et al. 2024 call model collapse. The objective’s tilt compounds across model generations in a way the corpus’s does not.
Deployment capture. Even with corpus and objective fixed, the operator can re-tune the model at inference through system prompts, fine-tunes, refusal policies, and tool constraints. It is the cheapest of the three, because it changes nothing about the weights, and in principle the easiest to audit, because it leaves traces in the configuration; in practice it is the most opaque, because those configurations are rarely published and can change silently between requests.
The three compose. A corpus-captured model with a captured objective and a captured deployment is not three problems stacked but one multiplied, each layer amplifying the last. Chapter 10’s self-capture argument now has to run across every surface the LLM owns, and the captured equilibrium it called more stable than any adversary has that many more dimensions to be stable in. The composition rules, and the worst case where a captured corpus and captured receiver-training close the loop at both ends, are worked through in the capture taxonomy; corpus and objective both turn out to be consumer-key substrates, the hard kind to recover from, and objective has the worst recovery dynamics in the book.
Manufactured content at industrial scale
Chapter 1 named manufactured content: content that never originated in measurement, which downstream gates cannot easily tell from the measured kind. Every LLM output is manufactured in that strict sense. The model measured nothing; it produced a plausible token sequence whose plausibility comes from patterns in its training data. Sometimes those patterns track reality (the corpus held reliable measurements and the prompt aligned to them); often they track plausibility-given-the-distribution, a weaker thing. The output is indistinguishable in form from measured knowledge, because the model does not preface its errors with a disclaimer, and gates that weight credibility by surface form will rank it alongside a peer-reviewed finding.
This is not villainy; it is the architecture doing what it was built to do. The policy question is not “how do we stop the model producing manufactured content” (it can do nothing else) but “how do downstream gates tell measured from manufactured at this scale,” and the book’s earlier answer was that they largely cannot. At LLM scale that becomes operational: the share of manufactured content in the corpus the next model trains on rises over time, and the gates that would have caught it have not improved to match.
When the question is “what do we believe”
Everything so far treats the model as a transmitter of objective content that can be tilted. But watch what people actually ask a model: what does the law say, what is money, what does the constitution mean, what do we believe. Those are intersubjective questions, truths generated by the network’s agreement, and for that cargo the model is not transmitting a fact but performing the bureaucracy’s own function, holding the shared key that keeps a scaled agreement composed, now industrialized and collapsed into one operator. A captured model answering intersubjective questions is therefore not a distortion risk but a generator risk: the effective-but-owned institution re-supplying a tuned key that looks like success from outside, at a scale and speed no church or court ever reached. Corpus capture on objective content produces wrong answers the world can eventually correct. Corpus capture on intersubjective content produces a different shared reality, with no world outside the agreement to appeal the change to. This is the sharpest thing the LLM adds to the book’s worry, and the one Chapter 12 has least of an answer for.
The political economy decides
Chapter 10 argued that engagement maximization is a captured equilibrium of the business model, with no external adversary to fight, only an arrangement to dismantle. The same shape holds for LLMs in a different currency: the metric is paid conversions, enterprise contracts, and API volume rather than time on site, but the owner still tunes every surface toward what maximizes revenue, and the captured state still does not announce itself. Three things follow.
The salvation case needs conditions revenue defeats. A faithful decompression service needs a wide corpus not optimized for any reading, an objective that rewards faithfulness to the source over user satisfaction, and a deployment not tuned to keep users happy at the expense of accuracy. Each of those costs revenue, so none is what a revenue-maximizing operator builds by default. The worst case is the default: a corpus scraped cheaply with the web’s biases intact, an objective tuned for retention (agreeable, not corrective), and a deployment tuned for engagement (quick and confident, not slow and source-cited). That is the model most people meet, and it fits Chapter 10’s captured equilibrium exactly. And the concentration makes it bite harder than at any earlier stage: a captured platform still leaves the option space with its users, while a captured closed-weight model holds all five surfaces at once. Chapter 10’s diagnosis was bad enough; here it becomes the whole determinant of whether the technology lands as receiver-budget relief or as captured equilibrium at industrial scale.
The encyclical as a fellow traveler
A papal letter is a strange guest in a book about information theory, so a word on why it is here. In 2026 Pope Leo XIV published *Magnifica Humanitas*, the Catholic Church’s first encyclical on AI, and it wrestles with almost these problems from a moral-philosophical direction I have mostly avoided. When a very different tradition reaches similar conclusions by a different road, that both tests the argument and lends it sharper words.
Its sharpest contribution is a vocabulary that was missing: de facto power. “Selection has an owner because it is criterial” made the point structurally; the encyclical says it plainly, that private parties now hold authority that has “displaced” the formal-state kind, with tech firms defining “conditions for access, rules of visibility” (para. 71). An LLM owner, in those terms, holds not just technical control of the surfaces but governance authority over what they produce, exceeding what states have historically had over media. The letter also offers a legible one-line gestalt for the choice at issue here, development shaped either toward Babel (centralized, efficient, ultimately fragmenting) or Jerusalem (plural voices coordinated through shared responsibility), which map onto the captured equilibrium and the integration project. And its call for “an educational alliance for the digital age” and for synodality, deliberating across kinds of expertise rather than deferring to one, lines up with Chapter 8’s training function and Chapter 9’s bridge-node thesis, adding the deliberative method left implicit.
Where the book pushes back: the encyclical’s prescription rests on shared discernment without engaging the polarization-via-distrust trap. Shared discernment under advanced polarization is exactly Chapter 9’s trust-bootstrap problem, and the methodology comes pre-discounted by communities whose distrust is already set. The two agree on what good integration looks like and disagree on how much prerequisite trust is left: the book designs for survivable polarization, the encyclical for restored trust. Read as one institutional carrier’s voice among many, with its own substrate-custody question intact, it is a genuine fellow traveler.
Where I land
An LLM is the first node in the book’s pipeline to own the gate, the option space, the content-generator, the training corpus, the objective, the deployment, and the pricing tier at once, all collapsed into one moment owned by one party. It is not the first object to bundle some of these, the encyclopedia and the textbook owned a corpus, an editorial line, and a distribution channel, but it is the first to bundle all of them at internet scale and with output personalized per reader, and that scale change is itself the categorical one. The same architecture can be the best receiver-budget intervention yet found, decompression to any depth and patient training at scale, or the worst manufactured-content failure it has diagnosed, industrially scaled. Ownership decides which, and the political economy of Chapter 10 says the default is the worst case: technically realizable, economically marginal, politically blocked by the same business model that decided the social-media outcome. Chapter 12 has to be designed against that default, not around the promise. The AI question is not separable from the platform question. They are the same question at adjacent scales.
Where I’m still uncertain
- The salvation case rests on faithfulness the architecture can’t yet guarantee. A faithful model re-supplies outer messages reliably; a hallucinating one supplies plausible-but-wrong ones, which is worse than none, because the reader decodes confidently with a key that doesn’t match. How much faithfulness can be engineered in, versus how much is bounded by next-token prediction over a training distribution, is open, and until it is answered the salvation case is more provisional than implied above.
- “Manufactured content” may flatten a real distinction. The book uses one word for astrology (invented wholesale) and LLM output (plausibility-tracked against a real corpus). On well-represented topics the second often approximates reality and the first never does. The category may need splitting into “manufactured-from-nothing” and “manufactured-from-training-distribution,” a real middle left unnamed.
- The salvation/worst-case binary is too clean. Most real deployments are neither: somewhat faithful, somewhat tilted, useful for some queries and misleading for others. The binary is a presentational convenience, and the real political economy will sort models along a continuum Chapter 12’s prescription should engage directly.
← Chapter 10: Political Economy of Attention · Interlude: Capture →