Exercises: Evaluate Reports Based on Data
Each report below makes a claim from data. Work through the four lenses:
(1) the sample — who was studied and how were they chosen; (2) the design —
does it license association only, or causation; (3) the uncertainty — is a
sample size, margin of error, or significance disclosed; and (4) the display
— is any graph honest. Write explanation answers in complete sentences. A
causal verb ("causes," "raises") needs a randomized experiment; an
observational study supports association only.
Warm-Up: Taking a Report Apart
These problems review the parts of a report and the IC vocabulary you already know.
A headline reads: "New Study: Students Who Drink Coffee Score 15% Higher on Exams." A report has three separable parts — the claim, the evidence, and the source. Which of the following is the claim?
Drinking coffee raises exam scores.
The actual statistic, the sample of students, and the study behind the 15%.
The researcher or company that produced and funded the study.
A study must be trustworthy because it appears in a headline and reports a number.
In the same coffee report, the "15% higher" came from comparing students who chose to drink coffee with students who chose not to. No one was assigned to a coffee or no-coffee group. What type of study is this?
An observational study, because the groups were self-selected, not randomly assigned.
A randomized experiment, because two groups were compared.
A sample survey with random selection.
A simulation of exam scores.
The four evaluation lenses are sampling, design, uncertainty, and display. Match each report weakness to its lens. "The report never says how many people were surveyed" is a problem with the ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ lens. "A bar chart's y-axis starts at 90 instead of 0" is a problem with the ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ lens. (Use one word for each: sampling, design, uncertainty, or display.)
Fluency: One Lens at a Time
Apply a single lens to each short report.
A report states: "In a survey of shoppers leaving Brand X's flagship store, 92% said Brand X is the best clothing brand." Name the one-word source concern this raises about who paid for or stands to benefit from the result. (One word, starting with a letter — think about conflict of ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ .) ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲
A website reports: "Our poll found 78% of people support the new policy." It does not say how many people were polled or how they were chosen. Applying the sampling lens, what is the best response?
The undisclosed sample size and selection method are themselves a red flag; the 78% cannot be judged as evidence about the population.
A 78% majority is large, so the sample must have been big and fair.
The result is fine as long as at least 30 people were polled.
Percentages from polls are always representative of everyone.
A report observes that "teens who spend more hours on social media report lower happiness" and headlines it "Social Media Lowers Teen Happiness." The data come from an observational survey. Applying the design lens, what is the precise problem?
The headline uses a causal verb ("lowers"), but an observational study supports association only; a confounder could drive both.
There is no problem; a clear pattern in the data proves causation.
The only problem is that the sample was probably too small.
The headline should have reported a higher percentage to be convincing.
An ad claims a supplement makes you "40% more alert." Applying the uncertainty lens, which two pieces of information do you most need before this percentage can be judged as evidence?
The sample size (how many people) and the baseline/measure the 40% is computed from.
The brand name and the price of the supplement.
Nothing more; "40% more alert" is precise enough to trust.
Only the color of the packaging used in the study.
A report passes the sampling lens (large random sample) and the design lens (randomized experiment with a significance check), but its only chart uses a y-axis starting at 95 to make a 1-point difference look enormous. Write a one-sentence calibrated verdict that names the strength AND the weakness, rather than declaring the whole report worthless.
You're viewing 2 of 6 sections.
Create a free account to continue the full exercise set and save your progress.
Create free account