Example · part of Thinking inside frameworks
Take a test question for an AI language model and change only its layout: a different separator, different spacing, a capital letter. The meaning stays the same. Sclar, Choi, Tsvetkov and Suhr (2024) did this across more than 50 tasks with several widely used open-source models. For one model, LLaMA-2-13B, accuracy differed by up to 76 points between formats. The median spread was 7.5 points.
How well a format worked on one model only weakly predicted how it worked on another, which undermines comparisons made with a single, arbitrarily chosen format. The sensitivity also persisted with larger models, more examples in the prompt, and instruction tuning.
A prompt format is a default nobody thinks of as a choice, and this is framework pricing measured in a machine. The book calls steering a model through its prompts and settings, with the model itself unchanged, deployment capture, one of the many design choices an AI owner holds at once. The authors’ advice is a form of framework rotation: report a range across plausible formats.
The tests were few-shot tasks on open-source models, and 76 points is the largest gap found.
How solid is this? A direct measurement across 50+ tasks and several open-source models in few-shot settings; the 76-point figure is the maximum for one model, against a median spread of 7.5 points.
Read the full argument
- Thinking Inside Frameworks, section “The clearest case is a machine”
How it connects
- What it shows: Defaults, Deployment capture, Framework pricing, Framework rotation
- See also: AI benchmarks as milestones, Inductive bias
- Evidence: Frameworks That Shape the Thoughts Built Inside Them, Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design
- Where the book discusses it: Chapter 11: AI as a New Kind of Node, Thinking Inside Frameworks
← Inductive bias · Thinking inside frameworks (21 of 21)
Explore all the ideas · Read the book in order · Open the interactive map