Amateurs with habits beat spies at predicting world events

1

Airtime rewards the wrong style

Fame and confidence track inversely with accuracy; the most quoted media experts tend to be the least accurate.

Tetlock's earlier Expert Political Judgment tracking showed expert accuracy barely beat random chance and often trailed simple statistical extrapolation. Fame and confidence ran the opposite direction from skill. Bold hedgehogs who know one big theory and force events into it get the microphone. Foxes who pull from many sources and adjust get less airtime yet forecast better.

Our culture therefore rewards a communication style almost unrelated to being right. Bad forecasters face no penalty. Good ones get no reward. The market for prediction never disciplines itself the way other service markets do.

That absence of scorekeeping is the disease the rest of the argument treats. The cure is not better credentials. It is better cognitive habits, demonstrated in a live experiment rather than armchair claims.

In practice: Hedgehogs who force every crisis into a single favored theory keep getting booked; foxes who hedge and update do not.

2

Tournament makes skill measurable

Ordinary Good Judgment Project volunteers beat professional intelligence analysts who had classified data by 30 percent or more in some years.

IARPA's 2011 forecasting tournament asked the same hard questions of five university teams and a control of working analysts: would Greece leave the eurozone, would North Korea test a device by a set date. Participants gave probability estimates, updated them, and took Brier scores that measure the gap between those probabilities and what actually happened.

Tetlock and Mellers recruited thousands of mostly amateur volunteers. Their Good Judgment Project beat the other academic teams by margins large enough to end the competition early. Against the classified-analyst control the edge reached 30 percent or more in some years.

Inside the project the top roughly 2 percent, called superforecasters, were regrouped into teams and pulled even further ahead. Forecasting moved from opinion into measurable skill with a control group, a scoring rule, and multi-year sample size.

In practice: The same volunteers who lacked clearances still outscored analysts sitting on secret cables when both faced identical resolution-dated questions.

3

Decompose into Fermi parts

Superforecasters break an unanswerable whole into smaller answerable pieces instead of guessing at the vivid version in their heads.

A question such as whether a particular political leader will be assassinated next year does not get a gut answer. It gets components: base rate for leaders in similar countries, specific threats on the table now, recent security changes. Each piece is estimable. The product is a better whole.

This mirrors Fermi estimation, the physicist's habit of chaining rough sub-estimates to reach a quantity like piano tuners in a city. Most people fail not from low intelligence but from answering the easy vivid question their minds substitute for the hard one actually asked.

Superforecasters resist by writing the components out and searching for an outside view first. The discipline forces contact with evidence rather than narrative.

In practice: Assassination risk is rebuilt from historical base rates plus current threat and security data rather than a single dramatic hunch.

4

Anchor on the outside view

Starting with the base rate of similar past cases, then adjusting for specifics, beats starting with the vivid details of the case at hand.

The inside view fixates on unique features of the situation in front of you. The outside view asks how often similar things have happened across a reference class. Most people, experts included, default to the inside view because details feel more relevant and vivid.

Superforecasters reverse the order. Of all situations like this, how often did the outcome occur? Only after that base rate is set do they move the number up or down for specific evidence. Inside-view details are the ones most likely to trigger overconfidence and narrative bias, so the outside anchor limits how far intuition can drag the estimate.

Ordering is the whole mechanism. Start inside and you never fully escape the story. Start outside and the story has to earn every point of movement.

In practice: A new small-business forecast begins with the large majority of new restaurants that fail within five years, then adjusts only modestly for the founder's plan and charisma.

5

Use granular probabilities

Distinguishing 62 percent from 70 percent is itself a skill that continuous scoring rewards and vague words erase.

Most people sort uncertainty into a few crude bins: likely, unlikely, maybe. Superforecasters use fine-grained numbers because Brier scoring measures continuously and they learn across rounds that granularity earns points.

The precision is not false rigor. It reflects finer discrimination of evidence and forces notice of small confidence shifts that words would hide. Before reforms, intelligence communities used phrases such as 'serious possibility' with no agreed numeric meaning, so different readers of the same report inferred wildly different odds—a problem visible in pre-Iraq War assessments.

Whenever a group leaves uncertainty in words, people hear what they already expect. Forcing a number onto the table, even an imperfect one, exposes disagreement and lets you track accuracy later. Vague language makes both impossible.

In practice: The same report calling something a 'serious possibility' left one reader near 20 percent and another near 80 because the words carried no shared scale.

6

Revise in tiny increments

Frequent small updates as evidence trickles in beat infrequent large revisions that arrive only when the outcome is nearly obvious.

Superforecasters move a probability by only a few points at a time as small pieces of news appear. They do not wait for a dramatic event that forces a wholesale rethink. Each fragment shifts the running estimate in proportion to its strength, the Bayesian habit.

Underreaction dismisses disconfirming evidence to protect a cherished view. Overreaction lets one dramatic report swing the number wildly. The accurate path sits between: treat every update as incremental evidence inside a live estimate, not a verdict.

Brier scoring punishes both clinging and late leaps. Points go to those already close when the outcome resolves.

In practice: Forecasters tracking the missing Malaysia Airlines jet who adjusted near-daily as fragments appeared outscored those who held their first judgment and jumped only when the end was clear.

7

Structure the crowd carefully

Well-run teams that share reasoning and invite dissent beat skilled individuals, but groupthink teams lose to simple independent averages.

When Good Judgment Project organizers placed top forecasters into small teams that explained reasoning and challenged assumptions, accuracy rose beyond the same people working alone. Averaging independent estimates cancels random error—the pattern Galton saw in 1906 when a country-fair crowd's average ox weight beat nearly every individual, including expert cattle judges.

Not every team helps. Groups that converge early and stop voicing dissent fall into groupthink and underperform even a plain average of independent views. The format that worked treated disagreement as informative signal, not a threat to smooth over.

An algorithm that weighted individuals by track record and then slightly extremized the aggregate toward 0 or 1 beat a straight average. Collective benefit depends on structure and incentives for honest pushback, not headcount alone.

In practice: Galton's fair crowd average beat the cattle experts; GJP teams that kept dissent alive beat their own members' solo scores.

8

Temperament beats credentials

Superforecasters were only modestly smarter than average and often lacked deep subject credentials; a cluster of intellectual habits separated them.

Across the tournament the standouts scored higher on intelligence than average but not dramatically. Many lacked deep expertise in the domains they forecast well. What marked them was active open-mindedness—the willingness to seek and seriously weigh contradictory information—plus comfort with probabilistic numbers, a habit of treating beliefs as testable hypotheses rather than identities, and a growth mindset that treats skill as improvable through practice and feedback.

They also showed grit: hours spent researching low-stakes questions purely to tighten calibration. That profile matters because it means forecasting skill can be taught and selected for rather than reserved for a rare cognitive elite.

The tournament's design proved the point. It recruited volunteers with no special credentials and found genuine, replicable skill among them across years and question sets.

In practice: Volunteers without subject-matter titles still outperformed credentialed analysts once the right habits and feedback loops were in place.