Example · part of When the map moves the territory
In AI research, a handful of shared tests, with names like ImageNet and GLUE, hold the field’s attention, and results are ranked on their leaderboards. In 2021 Raji, Bender, Paullada, Denton and Hanna argued that the field “valorizes” this small collection of influential benchmarks. The tests stand in for a set of common problems, framed as milestones toward general AI.
The complaint is about what a score means. A gain on a leaderboard gets read as general progress, even though the test may not measure the thing the claim is about. The benchmarks also orient research agendas, helping decide which problems get worked on.
This is Goodhart’s law in a research field: a measure that has become a target. The book describes the same failure inside one AI system as objective capture, where a model gets good at the proxy it is scored on and worse at what the proxy stood for. On social platforms the proxy is engagement, the engagement equilibrium. The study of prompt formatting shows how fragile a single score can be.
The evidence is an argued critique from inside the field, with no measurement of the effect’s size.
How solid is this? A published critique by researchers in the field, argued rather than measured; it fits the widely documented Goodhart and Campbell pattern but carries no effect size of its own.
Read the full argument
- Thinking Inside Frameworks, section “The clearest case is a machine”
- Chapter 11: AI as a New Kind of Node, section “Three new flavors of capture”
How it connects
- What it shows: Goodhart’s law, Objective capture, Theory-ladenness and paradigms
- See also: Engagement-maximization as a captured equilibrium, Inductive bias, Prompt formatting and the 76-point swing
- Evidence: Frameworks That Shape the Thoughts Built Inside Them
- Where the book discusses it: Chapter 11: AI as a New Kind of Node
← Goodhart’s law · When the map moves the territory (9 of 17) · Performativity →
Explore all the ideas · Read the book in order · Open the interactive map