Data Literacy & Critical Thinking
Topic 3 of 11 · In Data Driven Decisions · September 9, 2026
Read data critically, detect statistical fallacies, recognize Simpson's paradox and survivorship bias, and learn to evaluate research and data sources before acting on them.
Overview
Data literacy without critical thinking is dangerous. You can read every chart perfectly and still reach the wrong conclusion — because the data is misleading, the sample is biased, or your brain is playing tricks on you. The world is full of convincing-looking statistics that are flat-out wrong.
This topic builds the defensive layer between data and decisions. You will learn to read data critically rather than passively, recognize the statistical fallacies that trip up even experienced professionals, understand why some of the most famous analyses in history were wrong, and develop a habit of asking the right questions before acting on any data-driven claim. This is the immune system of data-driven decision making.
Before You Start
- Re-read the cognitive-bias table from DDDM Fundamentals — today's statistical fallacies build directly on confirmation and survivorship bias.
- Be able to compute a percentage change and to state what 'from what baseline' means for three metrics you track.
- Bring one news article or company report that makes a statistical claim; you will dismantle and rebuild it.
- Recall the three reasons correlation appears without causation and give an example of each from memory.
- Practice stating a claim literally — the numbers, not your interpretation — before offering any opinion.
Learning Objectives
- Read and interpret data presentations critically, questioning assumptions and methodology
- Identify and explain Simpson's paradox with a real-world example
- Recognize survivorship bias and its effect on business conclusions
- Understand base rate neglect and why it distorts risk assessment
- Distinguish correlation from causation in data analysis
- Evaluate the quality and credibility of a data source
- Ask the right questions before accepting any data-driven claim
- Detect common statistical fallacies in business reports and media
Prerequisites
Start with Data-Driven Decision Making Fundamentals for the basics of DDDM and cognitive biases. KPI Design & Metric Frameworks is helpful but not required.
Reading Data Critically
Most data presentations are persuasion tools, not neutral information. Every chart, table, and statistic has been selected, framed, and designed by someone with a perspective. Reading critically means asking:
- What is being measured? Is the metric well-defined, or could it be interpreted multiple ways?
- What is the baseline? A 20% increase means nothing without knowing 20% from what?
- What is the sample? Who was included, who was excluded, and how were they selected?
- What is the time frame? Cherry-picked dates can make anything look good or bad.
- Who benefits from this interpretation? Follow the incentive.
- What is not shown? The most dangerous data is the data that was collected but not presented.
Simpson's Paradox
Simpson's Paradox occurs when a trend that appears in aggregated data reverses when the data is split into subgroups. It is one of the most counterintuitive phenomena in statistics.
Classic example: Hospital A has a higher overall survival rate than Hospital B. But when you split patients by severity (mild vs severe), Hospital B has a higher survival rate in both groups. How? Hospital B treats more severe cases — the case mix is different. The aggregation hides the true pattern.
Business example: Your company's overall conversion rate improved from 3% to 4%. But when you split by channel, conversion dropped in every single channel. The improvement came from a shift in traffic mix — more traffic from a high-conversion channel diluted the data. The "improvement" was an artifact, not real progress.
The lesson: always ask "are there subgroups that might tell a different story?" before trusting an aggregate number.
Survivorship Bias
Survivorship bias is the logical error of focusing on things that survived a process while ignoring those that did not — because the failures are invisible.
Famous example: During WWII, the military studied returning bombers and found bullet holes concentrated on the wings and fuselage. They wanted to add armor there. Statistician Abraham Wald pointed out they were only looking at planes that survived. The engines and cockpit, where returning planes had no holes, needed armor — because planes hit there did not return.
Business examples:
- Studying successful startups and concluding "all founders drop out of college" — you cannot see the millions who dropped out and failed
- Analyzing your best customers and concluding "long trial periods lead to high LTV" — you do not see the people who tried a long trial and still left
- Reading survivorship-biased business books ("7 Habits of Highly Effective People") and assuming those habits caused success
Base Rate Neglect
Base rate neglect is the tendency to ignore the general prevalence of something (the base rate) in favor of specific, vivid information.
Example: A drug test is 99% accurate. Only 0.1% of the population uses the drug. If you test positive, what is the probability you actually use the drug? Most people guess ~99%. The actual answer is about 9%. Why? Of 1,000 people tested, 1 is a true positive (0.1% × 1 × 99% ≈ 1) and 10 are false positives (99.9% × 1%). So of 11 positives, only 1 is real. The base rate (0.1%) overwhelms the test accuracy.
Business application: If 2% of leads convert and your predictive model is 90% accurate, a "high probability" lead is still more likely to not convert than to convert. Base rates matter — always.
Correlation ≠ Causation
This is the most important rule in data analysis: correlation is not causation. Two things can move together without one causing the other.
Three reasons correlation appears without causation:
- Confounding variable — a third factor causes both. Ice cream sales and drowning rates are correlated — not because ice cream causes drowning, but because both increase in summer (temperature is the confounder).
- Reverse causation — the arrow points the other way. "Hospitals have more deaths" does not mean hospitals cause death — sick people go to hospitals.
- Coincidence — with enough variables, some will correlate by pure chance. "US cheese consumption correlates with deaths by bedsheet entanglement" is real data — and pure coincidence.
When can you infer causation? Only from a well-designed randomized controlled experiment (A/B test) where the only difference between groups is the treatment. Observational data alone cannot prove causation, no matter how strong the correlation.
Evaluating Data Sources
Not all data is created equal. Before trusting any data-driven claim, evaluate the source:
| Question | What to look for | Red flag |
|---|---|---|
| Who collected the data? | Independent researchers, reputable institutions | The company selling the product collected the data |
| What was the methodology? | Random sampling, adequate sample size, controlled conditions | Self-selected survey, tiny sample, no control group |
| Is it peer-reviewed? | Published in a reputable journal or vetted by experts | Blog post, press release, or self-published report |
| Can it be replicated? | Methods are transparent and data is available | Proprietary methodology, no data access |
| What is the funding source? | Independent funding or disclosed conflicts | Funded by a party that profits from the result |
| Is it current? | Data is recent enough to be relevant | Using 2010 data for a 2025 decision |
Asking the Right Questions
Critical thinking is not about knowing all the answers — it is about knowing which questions to ask. Before acting on any data claim, run through this checklist:
- What is the claim? State it in one sentence.
- What data supports it? Ask for the specific numbers, not the conclusion.
- What is the sample size? A claim based on 12 people is much weaker than one based on 12,000.
- What is the margin of error? A ±5% margin on a 3% effect means the effect might not exist.
- Is there a control group? Without one, you cannot attribute cause.
- Could survivorship bias apply? Are you only seeing the survivors?
- Could Simpson's Paradox apply? Would subgroups tell a different story?
- What would the opposite finding look like? If someone found the reverse, would the methodology be equally valid?
- Who benefits from this conclusion? Follow the incentive.
- Is this statistically significant AND practically significant? A tiny effect can be statistically significant with a large enough sample — but it may not matter in practice.
Hands-On Practice
These exercises build your critical thinking muscle with real data.
- Find a news article that makes a statistical claim. Apply all 10 critical questions above and write a one-paragraph assessment of the claim's validity.
- Search for "Simpson's Paradox real examples" and find one in a business context. Explain what the aggregate data shows versus the subgroup data.
- Find an advertisement or press release that uses data. Identify the most likely bias (survivorship, base rate neglect, cherry-picking, or confounding) and explain why.
- Given the drug test example from the topic, calculate the actual probability of a true positive for different base rates (0.1%, 1%, 5%, 10%). How does the answer change?
- Write a short memo to a manager who says "Our data shows X caused Y." Point out every possible reason the conclusion might be wrong.
Read each scenario and identify the primary statistical fallacy at play:
- "We hired five people from Company X and they all performed excellently. Company X produces the best talent."
- "Countries with more hospitals have higher death rates. Hospitals must be dangerous."
- "Our app has a 4.8-star rating with 10,000 reviews. It must be excellent."
- "Sales increased 15% after we changed the logo. The new logo is working."
Answers: (1) Survivorship bias — you only see the hires who worked out. (2) Confounding variable — sicker people go to hospitals. (3) Survivorship bias + selection bias — only satisfied users leave reviews. (4) Post hoc ergo propter hoc — correlation (timing) does not equal causation.
Deeper Worked Examples
Why your headline 'conversion improvement' may be an illusion
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Common Mistakes
- Treating correlation as causation — the single most common error in business analytics
- Ignoring base rates when interpreting probabilities or test results
- Generalizing from a tiny or self-selected sample
- Falling for Simpson's Paradox — trusting aggregate data without checking subgroups
- Assuming that because data looks clean and professional, the methodology is sound
- Ignoring survivorship bias — only studying winners or survivors
- Accepting data from a source with a clear conflict of interest without skepticism
Real-World Usage
In 2014, a widely-cited study claimed that the Apollo moon landing footage showed anomalies suggesting it was faked. The "evidence" relied on survivorship bias (only examining frames that looked suspicious), misunderstanding of physics (base rate neglect of how light behaves in a vacuum), and cherry-picked comparisons. Every claim was debunked by applying the exact critical thinking skills in this topic. In business, Netflix faced a version of Simpson's Paradox when analyzing viewer satisfaction: overall ratings looked great, but when split by genre, satisfaction was declining in every category — the improvement was driven entirely by a shift in content mix. Recognizing this early prevented Netflix from declaring victory on a metric that was masking a real problem.
Mini Project
Business problem: Your company's quarterly report claims "customer satisfaction increased 25% this quarter" and attributes it to a new chatbot. The CEO wants to double down on the chatbot investment.
Your task: Act as the critical thinker who audits this claim before the company commits resources.
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
| Deliverable | What it proves |
|---|---|
| Claim breakdown | You can deconstruct a data-driven claim precisely |
| Bias checklist | You can apply multiple critical thinking lenses |
| Confounding analysis | You can identify alternative explanations |
| Audit memo | You can communicate critical findings to leadership |
| Better study design | You can propose rigorous methodology |
Key Takeaways
- Read data critically: always ask about baselines, samples, time frames, and incentives
- Simpson's Paradox shows that aggregate data can hide or reverse subgroup trends
- Survivorship bias causes us to study only winners and ignore invisible failures
- Base rate neglect makes us overweight specific evidence and ignore general prevalence
- Correlation never implies causation — only randomized controlled experiments can
- Evaluate data sources for methodology, sample size, peer review, funding, and currency
- The 10-question critical thinking checklist catches problems before they become decisions
Key Terms & Glossary
| Term | Meaning |
|---|---|
| Simpson's paradox | A trend visible in aggregated data that reverses when the data is split into subgroups, usually because the composition of the groups changes. |
| Survivorship bias | Drawing conclusions only from cases that survived a process while the failures are invisible, such as studying winning startups and ignoring the failed ones. |
| Base rate neglect | Ignoring how common something is in general and over-trusting vivid specific evidence, such as a single impressive testimonial. |
| Confounding variable | A third factor that causes both of the correlated measures, producing an association that is not causal. |
| Post hoc ergo propter hoc | The fallacy of assuming that because B happened after A, A caused B; a campaign followed by rising sales does not prove the campaign did it. |
| Margin of error | How far a survey result may reasonably be from the true population value; a plus-or-minus 5% margin on a 3% effect means the effect might not exist. |
Learning Resources
How to Lie with Statistics — Darrell Huff
A classic (and short) book on how data can be used to mislead — essential reading for critical consumers of data.
Thinking, Fast and Slow — Daniel Kahneman
The cognitive science behind why our brains misinterpret data — covers base rate neglect, anchoring, and more.
Spurious Correlations (Tyler Vigen)
A humorous and educational site showing absurd correlations — perfect for teaching correlation ≠ causation.
Data Literacy Fundamentals (Qlik)
A free course covering the core skills of reading, working with, and analyzing data critically.
The Signal and the Noise — Nate Silver
How to separate meaningful signals from noise in data, covering prediction, uncertainty, and real-world case studies.
Chapter Quiz
Chapter Quiz — Data Literacy & Critical Thinking
Five questions on critical reading, Simpson's paradox, survivorship bias, base rates, causal claims, and source evaluation.
Question 1 of 5
Overall conversion improved from 3% to 4%, yet conversion dropped in every single channel. This is:
Knowledge Check
End-of-Topic Exam
Hint: Work with 1,000 people in a table: true positives versus false positives.
Model answer: In 1,000 people, one actually has the condition and is detected (about one true positive). The 999 healthy people produce about 10 false positives (1% of 999). So of roughly 11 positive results, only 1 is real — a probability of about 9%. People get it wrong because they judge the 99% accuracy alone and ignore the base rate of 0.1%; this is exactly base rate neglect, and Bayes' theorem exists to correct it.
Hint: Who is missing from the 90%?
Model answer: Survivorship and selection bias: the statistic may count only graduates who completed the course, ignoring dropouts who quit before completion, and may rely on students who respond to the survey. You would need to know the full cohort (all starters), their completion rate, how graduates were surveyed (self-selected or tracked), the sample size, the time frame, and whether the jobs are related to the training — otherwise the 90% is unverifiable.
Hint: Check who funded it, the methodology, and whether the conclusion matches the data.
Model answer: Three points: (1) Conflict of interest — the funder profits from a positive result, so independence is doubtful. (2) Methodology — a self-selected sample of 40 with no control group cannot support a causal uplift claim, and the margin of error on 40 respondents is very wide. (3) Replicability — without transparent methods or access to data the finding cannot be verified. Verdict: do not act on it; treat it as a hypothesis and run your own controlled test, ideally on your own traffic.