Noise

Why do judges, doctors, and even admissions officers make wildly different decisions in similar scenarios, and how can we reduce these inconsistencies?

1. Unseen Factors Can Skew Judgment

Human decisions are affected by unrelated factors that often go unnoticed but change outcomes in ways that defy logic. Take weather, for instance. A study found that admissions officers lean on academic performance on cloudy days but focus on extracurricular skills on sunny days. This random influence can unfairly alter critical life opportunities.

Not only weather, but other trivial factors like hunger, temperature, or mood can tilt decisions. A hungry judge is more likely to deny parole, and even something as irrelevant as their favorite sports team losing can affect rulings. These seemingly small things create unpredictable changes in judgment.

Academics call this kind of random change “occasion noise.” It’s a form of variability that happens when the same person acts inconsistently in similar situations. While invisible, its consequences ripple through the justice system, job hiring, healthcare, and countless industries.

Examples

• Admission officers prioritizing grades on cloudy versus sunny days
• Judges denying parole when hungry versus after eating
• Physicians making different diagnoses when affected by mood shifts

2. The Difference Between Noise and Bias

When we talk about making mistakes, bias gets much of the blame, but noise is just as significant. Bias operates systematically – like consistently shooting below a target. Noise, on the other hand, is random; like having your shots scattered all over the target without any pattern.

Imagine a Miami immigration court where asylum rulings swing wildly between 5% and 88%, depending on the judge. While part of this is due to bias (the judges’ personal stances), a large part is random noise – differences in individual decisions that shouldn’t exist for similar cases.

This kind of noise at a system-wide level is “system noise.” Whether in business, healthcare, or the justice system, unchecked system noise erases fairness and predictability, sometimes overshadowing even the effects of bias.

Examples

• Two judges in the same city differ drastically in asylum approvals
• A hiring team gives inconsistent feedback on the same candidate
• Doctors misinterpreting the same scans differently

3. Predicting Outcomes Is a Weak Spot for Humans

When predicting someone’s behavior or future performance, human intuition often fails. A judge deciding whether to grant bail has to predict what a defendant would do if released. Yet studies reveal that even simple math outperforms this decision-making.

For example, an algorithm used to judge bail dramatically outperformed human judges by simultaneously reducing crime and the number of people detained. Similarly, a simple formula using age and court attendance history was better at predicting flight risk than most experienced legal minds.

The reason is our tendency to trust what feels intuitively right. But this “feels right” signal often clouds judgment. It blinds us to our incomplete knowledge or biases, creating noise in predictions.

Examples

• Algorithms improving bail judgments more than experienced judges
• Simple formulas outperforming professionals in predicting outcomes
• People overestimating success probabilities based on their gut instincts

4. Our Love for Stories Hides Noise

Humans are wired to believe in stories. We crave cause-and-effect explanations. So, noise – which is random and resists simple narratives – often gets overlooked or mistaken for bias.

For example, when a qualified candidate doesn’t get hired, they might attribute the rejection to discrimination. While bias could play a part, when we zoom out, noise often reveals itself in the messy, chaotic patterns of human decision-making. We simply don’t intuitively notice randomness.

Psychological tendencies like “attribution error” push us to assign outcomes to personal failings or triumphs rather than luck or randomness. Noise lacks drama or coherence, making it harder for our minds to grasp, and easier to ignore completely.

Examples

• Students explaining bad grades based on “unfair” teachers rather than randomness
• People spotting bias instead of randomness in time spent on similar projects
• Fundamental attribution error skewing how we explain both success and failure

5. Combining Multiple Opinions Reduces Noise

One way to cut through the randomness of individual decisions is by averaging independent judgments. The key is to make sure that each decision-maker is working alone without influence from others. This cancels out errors and provides a clearer picture.

Francis Galton’s famous ox-weighing experiment showed this beautifully. While each individual guess for the ox’s weight was often wildly wrong, the average of the guesses was almost perfectly accurate. Unrelated errors cancel each other when you aggregate multiple independent inputs.

But beware – if biases infect the group (like sexism in a hiring panel), averaging won’t help. Also, groupthink can ruin independent judgment. For the technique to work, independence is critical.

Examples

• Villagers’ noisy ox weight guesses average out to near perfection
• Group decisions improving when members give input separately
• Hiring committees becoming biased if influenced by a loud voice

6. Audits Help You See Noise

Noise is often invisible until measured. A noise audit systematically reveals how inconsistent decisions really are, showing variability in black and white. This is the first step in addressing it.

Judge Marvin Frankel conducted such an audit in the legal system. He asked judges to review identical case descriptions and found massive variations in sentencing. Some judges recommended 30 days for a crime that others penalized with 15 years.

By exposing these random inconsistencies, organizations can start reducing noise. Auditing lets institutions quantify what’s tolerable before building a system to address it.

Examples

• Frankel’s audit exposed that similar crimes yielded wildly different sentences
• Variability in customer claim payouts at an insurance firm
• Medical diagnoses differing widely between two doctors reviewing identical symptoms

7. Clean Up Decisions with Discipline

Decision hygiene involves adopting disciplined ways of thinking to reduce noise. Like handwashing prevents infections, these methods prevent randomness from getting into decisions.

A useful method is stepping back to look at a case statistically instead of diving into irrelevant details. For example, instead of narrowly analyzing a CEO candidate’s resume, compare track records of executives in similar companies. This helps make decisions based on patterns rather than a gut feeling.

Another approach is breaking complex decisions into smaller, independent judgments. Example: evaluating job candidates based separately on skills, experience, and culture fit, rather than trying to form an overall impression.

Examples

• Taking a statistical approach when forecasting results
• Breaking hiring decisions into independent scores for different categories
• Delaying emotional reactions until data aligns with trends

8. Decision Rules Are Better When Judges Agree

Noise reduction works best when decision-makers agree on goals and methods. If a group lacks consensus or resists structured guidelines, noise soon returns.

Take the US Sentencing Reform Act, which drastically reduced sentencing variability by setting strict rules. After it was struck down, sentencing disparity doubled as personal values crept back into decisions.

Judge Frankel’s lesson? Before implementing any system to reduce noise, get buy-in by showing people why consistency beats personal discretion in judgment.

Examples

• Sentencing variability collapsing under guidelines and resurging without them
• Workplace pay structures becoming more uniform with standardized evaluations
• Medical protocols cutting variations between diagnoses for identical cases

9. Noise Has Steep Costs We Can No Longer Ignore

The random fluctuation in decisions isn’t just inefficient; it leads to injustices, wasted resources, and personal tragedy. It undermines public trust in institutions like the law, medicine, and education.

The judicial system is a stark example. A criminal facing wildly different potential sentences depending on their judge can feel victimized by a biased system. But the same issue spreads far wider than the courtroom, stretching into hiring, loans, healthcare procedures, and everyday decision-making.

Reducing noise may seem technical, but it’s about fairness, accuracy, and trustworthiness. Recognizing the costs makes tackling it a top priority.

Examples

• Massive economic losses when insurance underwriters misjudge claims
• Public frustration with perceived corruption due to inconsistent rulings
• Hiring the wrong person based on random interview variations

Takeaways

1. Run noise audits in your workplace or processes to uncover inconsistent decisions. Compare judgments on similar cases to reveal variability.
2. Use the wisdom of multiple opinions – but only if they’re independent – to distill accurate conclusions for complex questions.
3. Slow down decisions to avoid knee-jerk feelings. Frame your thinking through data or comparisons, not just immediate intuition.