Amateurs with habits beat spies at predicting world events

By Books (/journals/books)

In 60 seconds Media-friendly experts in Tetlock's multi-decade tracking barely beat chance and often trailed simple stats; the loudest voices were the least accurate. In IARPA's tournament, ordinary Good Judgment Project volunteers beat rival teams and outperformed intelligence analysts with classified data by 30 percent or more. Superforecasters break hard questions into Fermi-style parts and anchor first on outside-view base rates before adjusting for specifics. They state uncertainty in fine percentages, update those numbers in tiny steps as evidence arrives, and treat beliefs as testable hypotheses. Structured teams that share reasoning, invite dissent, and weight by track record pull further ahead of even skilled solo forecasters. The edge comes from temperament—active open-mindedness, grit, growth mindset—more than raw IQ or subject credentials.

The big idea

Accurate forecasting is a trainable skill built from specific cognitive habits, not genius or clearances. Philip E. Tetlock and Dan Gardner use a multi-year tournament to show ordinary volunteers can beat professionals once predictions are scored and common traps are systematically avoided.

The book in brief

Superforecasting by psychologist Philip E. Tetlock and journalist Dan Gardner opens on a cultural failure: pundits who sound sure get airtime while careful ones do not. Tetlock's earlier Expert Political Judgment work tracked roughly 28,000 predictions from nearly 300 experts across twenty years and found accuracy barely above chance; the most media-friendly often lagged simple extrapolations. Hedgehogs who force everything into one big theory lose to foxes who gather widely and adjust.

The cure arrived as evidence, not theory. In 2011 IARPA ran a forecasting tournament on questions such as Greece exiting the eurozone or a North Korean nuclear test by a date. Tetlock and Barbara Mellers recruited thousands of mostly amateur volunteers into the Good Judgment Project. Scored by Brier scores that punish the gap between stated probabilities and outcomes, that team beat rival academic groups so soundly the contest ended early. It also beat a control group of professional intelligence analysts who held classified data by that same 30-point edge in some years. The top 2 percent, labeled superforecasters, were teamed and pulled still further ahead.

How they did it fills the middle chapters. Superforecasters refuse the vivid easy question their minds substitute for the hard one asked. They break problems into smaller Fermi estimates—base rates for assassination of similar leaders, current threats, security changes—then combine the rough parts. They start with the outside view: how often do cases like this produce the outcome, then adjust only modestly for inside details. A new restaurant's survival odds begin with the large majority that fail within five years, not the founder's pitch.

They refuse crude words like 'likely.' They use granular numbers—62 percent versus 70—because continuous scoring rewards the discrimination and surfaces small confidence shifts. They update in small steps as fragments arrive, the way Bayesian reasoning says evidence should move a prior, rather than cling until a late dramatic jump. Trackers of the missing Malaysia Airlines jet who adjusted near-daily outscored those who held firm then leaped.

When top performers shared reasoning in small teams that treated disagreement as signal, accuracy rose again. Aggregation that weights by past track record and slightly extremizes the average beat a plain mean. Groupthink teams that silence dissent did worse than independent averages. The shared trait was temperament: active open-mindedness, comfort with numbers, beliefs held as hypotheses, grit spent on low-stakes practice, and a growth mindset that treats skill as improvable.

The book lands on a procedural fix. Score forecasts against real outcomes. Reward calibration. Make every prediction a testable claim. Culture currently rewards storytelling because nobody keeps score; once you do, habits beat credentials.

Context

Tetlock had already spent decades measuring expert judgment when IARPA's 2011 tournament gave him a live laboratory with control groups and hard scoring. The 2015 book with Gardner arrived as proof that the accountability gap he diagnosed could be closed by ordinary people using measurable methods, not by hiring more credentialed voices.

The key ideas

1

Airtime rewards the wrong style

Fame and confidence track inversely with accuracy; the most quoted media experts tend to be the least accurate.

Hedgehog One big theory, bold certainty, more airtime, worse accuracy

Fox Many sources, constant adjustment, less airtime, better forecasts

Tetlock's earlier Expert Political Judgment tracking showed expert accuracy barely beat random chance and often trailed simple statistical extrapolation. Fame and confidence ran the opposite direction from skill. Bold hedgehogs who know one big theory and force events into it get the microphone. Foxes who pull from many sources and adjust get less airtime yet forecast better.

Our culture therefore rewards a communication style almost unrelated to being right. Bad forecasters face no penalty. Good ones get no reward. The market for prediction never disciplines itself the way other service markets do.

That absence of scorekeeping is the disease the rest of the argument treats. The cure is not better credentials. It is better cognitive habits, demonstrated in a live experiment rather than armchair claims.

In practice: Hedgehogs who force every crisis into a single favored theory keep getting booked; foxes who hedge and update do not.

2

Tournament makes skill measurable

Ordinary Good Judgment Project volunteers beat professional intelligence analysts who had classified data by 30 percent or more in some years.

30%+ GJP edge over classified analysts

IARPA's 2011 forecasting tournament asked the same hard questions of five university teams and a control of working analysts: would Greece leave the eurozone, would North Korea test a device by a set date. Participants gave probability estimates, updated them, and took Brier scores that measure the gap between those probabilities and what actually happened.

Tetlock and Mellers recruited thousands of mostly amateur volunteers. Their Good Judgment Project beat the other academic teams by margins large enough to end the competition early. Against the classified-analyst control the edge reached 30 percent or more in some years.

Inside the project the top roughly 2 percent, called superforecasters, were regrouped into teams and pulled even further ahead. Forecasting moved from opinion into measurable skill with a control group, a scoring rule, and multi-year sample size.

In practice: The same volunteers who lacked clearances still outscored analysts sitting on secret cables when both faced identical resolution-dated questions.

3

Decompose into Fermi parts

Superforecasters break an unanswerable whole into smaller answerable pieces instead of guessing at the vivid version in their heads.

1Name the hard questionResist the vivid substitute your mind offers 2List componentsBase rate, current threats, security changes 3Estimate each roughlyChain the sub-estimates into one probability

A question such as whether a particular political leader will be assassinated next year does not get a gut answer. It gets components: base rate for leaders in similar countries, specific threats on the table now, recent security changes. Each piece is estimable. The product is a better whole.

This mirrors Fermi estimation, the physicist's habit of chaining rough sub-estimates to reach a quantity like piano tuners in a city. Most people fail not from low intelligence but from answering the easy vivid question their minds substitute for the hard one actually asked.

Superforecasters resist by writing the components out and searching for an outside view first. The discipline forces contact with evidence rather than narrative.

In practice: Assassination risk is rebuilt from historical base rates plus current threat and security data rather than a single dramatic hunch.

4

Anchor on the outside view

Starting with the base rate of similar past cases, then adjusting for specifics, beats starting with the vivid details of the case at hand.

Inside view first Unique plan, founder charisma, vivid details drive overconfidence

Outside view first Base rate of similar failures, then modest adjustment for specifics

The inside view fixates on unique features of the situation in front of you. The outside view asks how often similar things have happened across a reference class. Most people, experts included, default to the inside view because details feel more relevant and vivid.

Superforecasters reverse the order. Of all situations like this, how often did the outcome occur? Only after that base rate is set do they move the number up or down for specific evidence. Inside-view details are the ones most likely to trigger overconfidence and narrative bias, so the outside anchor limits how far intuition can drag the estimate.

Ordering is the whole mechanism. Start inside and you never fully escape the story. Start outside and the story has to earn every point of movement.

In practice: A new small-business forecast begins with the large majority of new restaurants that fail within five years, then adjusts only modestly for the founder's plan and charisma.

5

Use granular probabilities

Distinguishing 62 percent from 70 percent is itself a skill that continuous scoring rewards and vague words erase.

Most people sort uncertainty into a few crude bins: likely, unlikely, maybe. Superforecasters use fine-grained numbers because Brier scoring measures continuously and they learn across rounds that granularity earns points.

The precision is not false rigor. It reflects finer discrimination of evidence and forces notice of small confidence shifts that words would hide. Before reforms, intelligence communities used phrases such as 'serious possibility' with no agreed numeric meaning, so different readers of the same report inferred wildly different odds—a problem visible in pre-Iraq War assessments.

Whenever a group leaves uncertainty in words, people hear what they already expect. Forcing a number onto the table, even an imperfect one, exposes disagreement and lets you track accuracy later. Vague language makes both impossible.

In practice: The same report calling something a 'serious possibility' left one reader near 20 percent and another near 80 because the words carried no shared scale.

6

Revise in tiny increments

Frequent small updates as evidence trickles in beat infrequent large revisions that arrive only when the outcome is nearly obvious.

1New fragment arrivesA small piece of news or data 2Shift a few pointsMove the probability in proportion to its strength 3Keep the estimate liveReady for the next fragment ↻ and round again

Superforecasters move a probability by only a few points at a time as small pieces of news appear. They do not wait for a dramatic event that forces a wholesale rethink. Each fragment shifts the running estimate in proportion to its strength, the Bayesian habit.

Underreaction dismisses disconfirming evidence to protect a cherished view. Overreaction lets one dramatic report swing the number wildly. The accurate path sits between: treat every update as incremental evidence inside a live estimate, not a verdict.

Brier scoring punishes both clinging and late leaps. Points go to those already close when the outcome resolves.

In practice: Forecasters tracking the missing Malaysia Airlines jet who adjusted near-daily as fragments appeared outscored those who held their first judgment and jumped only when the end was clear.

7

Structure the crowd carefully

Well-run teams that share reasoning and invite dissent beat skilled individuals, but groupthink teams lose to simple independent averages.

Groupthink team Early convergence, silenced dissent, worse than independent average

Structured team Shared reasoning, invited critique, weighted by track record, further gains

When Good Judgment Project organizers placed top forecasters into small teams that explained reasoning and challenged assumptions, accuracy rose beyond the same people working alone. Averaging independent estimates cancels random error—the pattern Galton saw in 1906 when a country-fair crowd's average ox weight beat nearly every individual, including expert cattle judges.

Not every team helps. Groups that converge early and stop voicing dissent fall into groupthink and underperform even a plain average of independent views. The format that worked treated disagreement as informative signal, not a threat to smooth over.

An algorithm that weighted individuals by track record and then slightly extremized the aggregate toward 0 or 1 beat a straight average. Collective benefit depends on structure and incentives for honest pushback, not headcount alone.

In practice: Galton's fair crowd average beat the cattle experts; GJP teams that kept dissent alive beat their own members' solo scores.

8

Temperament beats credentials

Superforecasters were only modestly smarter than average and often lacked deep subject credentials; a cluster of intellectual habits separated them.

Across the tournament the standouts scored higher on intelligence than average but not dramatically. Many lacked deep expertise in the domains they forecast well. What marked them was active open-mindedness—the willingness to seek and seriously weigh contradictory information—plus comfort with probabilistic numbers, a habit of treating beliefs as testable hypotheses rather than identities, and a growth mindset that treats skill as improvable through practice and feedback.

They also showed grit: hours spent researching low-stakes questions purely to tighten calibration. That profile matters because it means forecasting skill can be taught and selected for rather than reserved for a rare cognitive elite.

The tournament's design proved the point. It recruited volunteers with no special credentials and found genuine, replicable skill among them across years and question sets.

In practice: Volunteers without subject-matter titles still outperformed credentialed analysts once the right habits and feedback loops were in place.

Key terms Superforecasters The top roughly 2 percent of Good Judgment Project volunteers whose accuracy pulled far ahead of the rest Brier score A measure of the gap between forecast probabilities and actual outcomes; lower is better Outside view The base rate of how often similar cases produced the outcome, taken before case-specific details Inside view Focus on the unique features of the situation at hand, which often fuels overconfidence Fermi estimate Breaking a hard quantity into a chain of rough sub-estimates that combine into an answer Hedgehog and fox Hedgehogs force events into one big theory; foxes draw from many sources and adjust—and foxes forecast better Active open-mindedness Seeking out and seriously weighing information that contradicts your current view Extremizing Nudging an aggregated probability slightly further toward 0 or 1 after weighting by track record

What to do

Write down a base rate from similar past cases before you touch the specifics of your next forecast. State the prediction as a precise percentage with a date so you can score it later. Set a recurring reminder to move that number in small increments as new information arrives. In a team forecast, assign one person to argue the opposite case before you lock the number. Keep a running log of past predictions against outcomes so confidence has to earn its place.

Questions to think with When did you last change a public prediction by only a few percentage points instead of holding or flipping? What base rate would you use for the next big call on your desk, and where would you find it? Does your team treat disagreement as signal to explore or as friction to remove? Which of your current beliefs are you treating as identity rather than as a testable hypothesis? How would your incentives change if every forecast you make were scored and posted?

The other side

The strongest evidence covers short-to-medium-term geopolitical and economic questions with clear resolution dates—the kind IARPA could score cleanly. The book is thinner on transfer to genuinely novel black-swan events that lack a reference class, a limit Tetlock partly concedes by separating clocklike from cloudlike systems without a full rule for telling them apart in advance. Tournament horizons mostly under a year say little about decade-scale forecasts where feedback is too slow to train rapid updating. Superforecasters were self-selected volunteers willing to spend unpaid hours, so it is unclear how much of the edge survives paid, high-stakes, politically pressured work; some corporate and government replications have shown smaller gains. Brier scores reward calibrated probabilities on well-defined questions, not wisdom about which questions matter or how to frame them.

Who it's for

Analysts, strategists, and executives who make judgment calls under uncertainty will get the most, along with anyone tired of unaccountable punditry who wants a concrete method for updating beliefs as evidence arrives.

Read the full summary in Library →