Everybody Lies Summary

People lie in surveys, but their internet searches reveal the truth.

1. Big Data is Big in Scale, but Intuitive in Nature

Big data refers to a massive amount of information, so large that it takes computational power to detect patterns. But despite its size, the process of analyzing it can actually feel quite instinctive. It’s akin to what people do when they try to learn from experience. Data scientists use this information to confirm or challenge intuition and make predictions.

For example, the author’s grandmother often analyzed years of relationship patterns to predict suitable partners. She believed shared friendships made a relationship stronger. However, a Facebook study revealed the opposite. Couples with many mutual friends were more likely to break up, proving that data often refines imperfect personal assumptions.

This approach means that big data doesn’t only capture massive patterns—it can also challenge individual beliefs. When sound methods are used with vast datasets, people can see past their biases and arrive at surprising yet accurate conclusions.

Examples

• The grandmother’s belief about shared friendships in relationships clashed with Facebook’s large-scale study.
• Lars Backstrom and Jon Kleinberg’s research highlighted how data could prove common assumptions wrong.
• Data can pinpoint how certain variables, like mutual friends, affect other outcomes, like relationship status.

2. Google Transforms Data Into Practical Predictions

What sets big data apart isn’t just size—it’s finding practical applications in real time. Google transformed search engine algorithms to prioritize links and relevance, producing helpful results instead of keyword-heavy spam.

This application of big data allows researchers to generate immediate, actionable insights. For instance, tracking flu-related searches like “flu symptoms” demonstrates disease spread across geographies. Google data replaced slower methods of collecting health statistics, bringing faster and sometimes more precise outcomes.

These real-time applications of big data have revolutionized predictions in various fields, from market trends to public health emergencies. Unlike long surveys or delayed reports, dynamic data analysis creates immediate, useful results.

Examples

• Google search uses links, not keywords, to rank websites’ relevance.
• Flu-related searches identified and tracked influenza hotspots globally.
• Businesses use traffic and trend data from Google tests to make quick decisions.

3. Big Data Exposes the Lies We Tell

Humans often distort the truth to seem better when asked to self-report. Surveys can fail to capture authentic behavior. People tend to embellish things like grades, income, or habits to paint a flattering picture. But big data, collected from anonymous online searches, doesn’t have this problem.

Search data strips away social pressures. When seeking answers in private—like typing “how to lose weight fast”—people reveal their true concerns. In unexpected examples, browsing adult content databases even revealed peculiar searches like “anal apple,” showing unconventional interests people might deny if asked publicly.

This shows that the honesty in big data creates a clearer understanding of human preferences and behaviors, offering a raw, unfiltered lens into society.

Examples

• 11% of students had GPAs under 2.5, though only 2% admitted it in surveys.
• Search engine data uncovers taboo tastes people hide in daily life.
• Private searches trump self-reported accounts in exposing personal truths.

4. Large-Scale Data Helps Zero In on Micro Triggers

Big data isn't just about identifying general trends—it also provides granular detail. Researchers can isolate subsections of large datasets to focus on smaller, more specific questions. This ability enables detailed examinations that weren’t possible before.

One notable case is a study by Professor Raj Chetty, who examined the “American dream” through tax records. While overall mobility in the U.S. seemed weaker than in other countries, big data revealed a more complex picture. Certain cities, like San Jose, offered better opportunities for poor citizens to climb the economic ladder compared to others like Charlotte.

Using data to zoom in enables researchers to measure phenomena across various scales, providing more context and nuance to broader figures.

Examples

• Chetty’s tax data study showed differences in regional economic mobility.
• San Jose fostered upward mobility more effectively than Charlotte.
• Granular data helps compare specific states or cities against nationwide trends.

5. Experimentation Gets Easier and Quicker With Big Data

Making sense of correlations and causation often requires controlled experiments. A/B testing, or randomized trials where groups experience different variables, is one tool for understanding causal relationships. Big data automates this once tedious process.

Barack Obama’s 2008 campaign used A/B tests for optimizing donation solicitations. By changing images and text on web pages and analyzing these versions' performance, they identified what worked best to increase sign-ups and donations. Traditional experiments would have required expensive setups, but technology cut the cost and time dramatically.

With big data, constant experimentation helps companies refine their strategies and improve outcomes without investing heavily.

Examples

• A/B testing determined Obama's campaign website’s most effective layout.
• Companies like Netflix analyze what menu designs increase viewing hours.
• Algorithms speed up experiments, replacing manual control groups with live data.

6. Data May Be Big, But it Needs Context and Care

Sometimes, big datasets with complex variables make accurate patterns hard to spot. Random coincidences might appear meaningful when they’re not, leading to flawed conclusions. An example is a 1998 study linking the gene IGF2r to higher IQs. Later attempts to repeat this study found no correlation—randomness caused the earlier false alarm.

Another issue arises when topics aren’t quantifiable. Facebook collects billions of clicks and likes but these metrics won’t explain user happiness or satisfaction. That’s where smaller approaches, like user surveys and psychological studies, become relevant.

Big data works best when mixed with smaller, targeted studies that frame its conclusions to human contexts.

Examples

• Robert Plomin’s IQ-gene study derailed due to random noise in his dataset.
• Facebook measures user emotions through targeted surveys, not just clicks.
• Some patterns are too abstract to measure without human input, like happiness.

7. Ethical Boundaries Exist When Governments Use Big Data

Big data brings up tough questions about ethics and privacy. Should governments act on personal searches like “I want to kill myself”? While actionable patterns show up across regions, applying those insights to individual cases crosses an ethical line.

Resource allocation also matters. Monitoring search data for suicidal intent isn’t plausible—3.5 million searches per month occur, far outweighing actual suicides. Instead, governments use this regional information for public awareness campaigns, such as broadcasting helpline numbers in high-risk areas.

Big data might find correlations, but acting responsibly requires restraint and a balance between public good and personal freedoms.

Examples

• Researchers connected suicide-related searches to state-by-state statistics.
• Police responding to individual searches would drain significant resources.
• Suicide prevention ads have been targeted at communities flagged by search trends.

8. Generational and Cultural Biases Appear in Data

People are shaped by their environments, and data reflects these influences. By analyzing generational trends in names, preferences, and habits, fascinating biases emerge. A search for popular baby names, for instance, reveals preferences rooted in cultural and social contexts.

Data harnessed from massive digital sources can reveal how traditions or taboos shift over time. It also explains how different kinds of information or online searches vary across countries based on regional norms.

Acknowledging these biases ensures interpretations stay sensitive to underlying cultural narratives.

Examples

• Baby-name databases mapped changing preferences through decades.
• Regional preferences for entertainment or consumer choices varied widely.
• Cultural norms influence what people search for, shaping macro-level data trends.

9. Big Data Lets Us See Ourselves More Honestly

Through anonymous aggregation of our collective habits, big data provides a mirror into true human nature. This transparency clarifies myths and offers valuable lessons about what’s universal versus individual quirks.

From disarming taboos to normalizing private preferences, widespread data allows a fuller embrace of human diversity.

Examples

• The stigma around uncommon interests diminishes when normalized.
• Honest patterns in search represent collective fears or fantasies.
• Big data erases individual hesitations, painting broader truths.

Takeaways

1. Use private digital behavior data for making accurate predictions instead of relying on self-reported surveys.
2. Apply A/B testing techniques regularly to assess and streamline decisions in work, creative projects, or campaigns.
3. Balance large-scale data with ethical considerations, especially avoiding unnecessary individual intrusions.