Book finds expert judgments differ far more than leaders expect

1

Bias versus noise

Organizations that correct only bias ignore unwanted scatter among judgments that should be identical—a second error source that leaves no direction to blame.

Judgment, for the authors, is any call under uncertainty where reasonable people could disagree: a sentence, a diagnosis, a hiring choice, an insurance premium.

Bias is systematic tilt—the average of many judgments misses in one direction, like a rifle sight set too far left. Noise is variability among judgments that ought to match. Four shooter groups at one bullseye split the idea: tight and centered is accurate; tight but off-center is biased; wide but centered is noisy; wide and off-center is both.

Most organizational talk targets bias because it has a story—a stereotype, an incentive, a blind spot. Noise has none. Two competent, well-intentioned professionals reading the identical file simply land apart, and the same person may reverse an earlier verdict later.

Because scatter points nowhere obvious, leaders underestimate how much they carry until someone deliberately measures side-by-side disagreement.

In practice: Four groups aim at one bullseye; the wide, centered spray is pure noise, and that is the pattern professional systems quietly produce when no one compares files.

2

What a noise audit shows

Giving identical real cases to many professionals independently reveals disagreement no one inside believes exists until the spread appears.

The book’s central evidence comes from a large insurer where Kahneman advised. Executives thought two experienced underwriters pricing the same case would differ by a tolerable margin.

The audit handed identical files to 828 underwriters without saying consistency was the point. The median difference between randomly paired quotes hit 55 percent—five times what those leaders had guessed. Claims adjusters showed large gaps on valuations too.

A noise audit needs no ground truth. Strip identifying cues, give the same case to many judges alone, and measure disagreement rather than the average. The design generalizes to forecasters on one economic figure, radiologists on one scan, judges on one hypothetical defendant, and patent examiners on one application.

The lesson is systemic, not personal incompetence: without a consistency check, the premium a customer pays or the sentence a defendant receives depends heavily on which employee drew the file.

In practice: 828 underwriters, same stripped cases, independent quotes: median pairwise gap of 55 percent against leadership’s rough 10 percent expectation.

3

Level versus pattern noise

Most disagreement comes from idiosyncratic weighting of case features, not from a few people being globally harsher—so recalibrating outliers leaves the bulk untouched.

Once an audit surfaces scatter, the authors split its sources. Level noise is the simple kind: some judges sit higher or lower across every case. A sentencer whose average runs above peers on equivalent crimes adds level noise.

Pattern noise is subtler and, in the insurance and sentencing decompositions, typically accounts for far more total noise. Two judges can share the same overall average yet reverse each other case by case—one soft on first offenses and hard on repeats, the other the opposite.

Average their averages and nothing looks wrong. Compare defendant by defendant and inconsistency is huge. Ordinary supervision almost never catches it, because each live case goes to one person alone.

Remedies aimed only at consistently harsh or lenient outliers therefore miss most of the problem. Later fixes in the book target how features get weighed, not merely where a global dial sits.

In practice: Two judges with matching average sentences still create a lottery if one is lenient on first offenses while the other is tough on them and softer on repeats.

4

When you disagree with yourself

Hiring better people cannot lock in reliability, because even one skilled professional shifts with mood, fatigue, and the order information arrives.

A third layer is noise inside a single person across occasions. The same doctor, judge, or underwriter given an identical case on two days often produces two answers though nothing in the file changed.

Fingerprint examiners re-shown a print they earlier called a match sometimes reverse the call when they do not recognize the repeat, especially once irrelevant context tags along. Order effects work similarly: material seen first anchors harder than the same material seen later, so two readers of one personnel file can diverge from sequence alone.

Some headline field findings on incidental schedule factors are contested; the authors still treat mood, fatigue, and context leaking into judgment as well supported by controlled experiments.

Selection and training therefore cannot finish the job. An excellent professional remains a slightly different instrument tomorrow morning, which pushes the argument toward process design over better headcount.

In practice: Fingerprint examiners reverse prior match calls on unmarked repeats when surrounding context changes, even though the print itself is unchanged.

5

Groups amplify scatter

Putting qualified people in a room does not average out idiosyncrasies; cascades and polarization often yield a more extreme, falsely confident shared verdict.

Replacing solo calls with group talk looks like an obvious fix. The authors argue it frequently makes noise worse.

Informational cascades do early damage. Whoever speaks first pulls later speakers toward that opening view, so the outcome tracks seniority or volume more than independent judgment and then dresses up as consensus.

Group polarization compounds it. People who already lean one way typically finish more extreme after discussion than the average of their private priors. A jury mildly inclined toward harshness often grows harsher, not milder.

Genuine averaging requires independence first: private, simultaneous estimates combined statistically. A show of hands after open talk is not that. The practical line is structured collection before any discussion.

In practice: A committee’s shared verdict often mirrors the first senior voice in the room rather than the mean of uninfluenced private views.

6

The illusion of agreement

Noise persists because judgment never feels arbitrary from the inside, so each professional assumes a colleague seeing the same facts would reach essentially the same conclusion.

If scatter is this large, why do organizations tolerate it unnoticed? Workflow is half the answer: each file usually goes to one person, so side-by-side disagreement never appears.

Psychology supplies the rest. The authors name the illusion of agreement. A diagnosis or hiring impression does not feel like one draw among many plausible ones; it feels like seeing what is there. The judge therefore never tests the peer match quietly assumed.

That confidence links to overconfidence patterns the authors invoke: people poorly sense how much history, mood, and values shape a call, and judgment-heavy fields rarely score outcomes against a clear right answer that would force the lesson.

Without feedback—what another judge would have sentenced, what a rejected applicant would have done—noise lasts until someone runs a deliberate audit. Measurement, not intention, is the required first step.

In practice: A loan officer almost never learns how a rejected applicant would have performed, so nothing challenges the sense that the call was simply the facts speaking.

7

Decision hygiene practices

Breaking a complex call into independently scored sub-assessments before any overall verdict removes much noise from first impressions and order effects without erasing expertise.

The practical program is decision hygiene, borrowed from the public-health idea of handwashing: steps that cut error without naming the specific disaster avoided in any one case.

Core moves include splitting judgment into separate sub-assessments recorded before a global conclusion forms; sequencing information so subjective or overall impressions arrive last; and aggregating multiple independent estimates statistically rather than through open discussion.

The mediating assessments protocol makes this concrete for interviews and evaluations. Raters score several specific attributes one at a time on scales anchored to concrete examples, and only after every sub-rating is locked do they form an overall judgment. Structured hiring interviews and formal grant reviews are working versions of the same idea.

None of this removes the need for skill. It aims to stop irrelevant order, mood, and early global impressions from driving variability, while leaving genuine expertise intact on well-defined sub-questions.

In practice: In a hiring process using mediating assessments, interviewers finish every trait rating on anchored scales before they may record an overall candidate score.

8

Formulas without bad days

Even crude statistical rules often outperform unaided experts on prediction tasks because they never tire or play favorites, yet people distrust algorithms and punish their errors more than identical human ones.

One strong path is replacing unaided judgment with a rule or formula. The evidence line runs through Paul Meehl’s 1954 review: in the great majority of roughly one hundred studies, simple statistical formulas matched or beat clinical experts on parole violation, academic success, and psychiatric diagnosis.

A mechanical rule carries zero occasion noise and zero pattern noise among judges, because there is only one consistent judge. Even equal weights on a handful of relevant variables tend to win—not from deeper insight, but from never tiring and never letting one salient feature overrun the rest.

Resistance stays strong. The book describes algorithm aversion: people still prefer the human after seeing the formula is more accurate, and they judge a formula’s errors more harshly than the same error from a person.

The authors treat the pushback as understandable—wanting individualized consideration, unease delegating parole or treatment to a machine, fear that biased training data will scale bias. Their narrower claim is that documented accuracy gains mean resistance to using rules even as one input deserves harder scrutiny than it usually receives.

In practice: Meehl’s review found simple formulas matched or beat expert clinical judgment in the great majority of about one hundred prediction studies across parole, grades, and diagnosis.

9

Consistency is not fairness

Driving noise out leaves underlying bias intact and can sacrifice local knowledge, mercy, adaptability, and the legitimacy some decisions draw from visibly human judgment.

The authors refuse to sell noise reduction as free. Heavy structure and rules can flatten the adaptability that helps with unusual cases—the applicant who fits no checklist box yet is still a sound risk.

Excessive standardization can demoralize skilled professionals who feel procedure is replacing craft, and can dull the independent noticing that sometimes catches what a formula would miss.

Consistency is also not fairness. An identical rule can still be a bad rule; removing scatter from a biased system only applies the bias more evenly. Noise work and bias work remain separate projects.

Some domains may want limited noise on purpose. In sentencing, individualized mercy can be part of what justice is supposed to offer, and not every judicial disagreement is pure error rather than a values clash. The authors’ answer is deliberate weighing by domain: killing order effects is nearly free; binding algorithms need a real stakes-and-values conversation.

In practice: A rigid loan checklist yields consistent yes-or-no decisions while missing the unusual applicant whose circumstances fit no box but who would still repay.