Building the Distribution and Making the Decision

Lesson 2 of 2: Run It and Decide

In this lesson:

• Build a simulated distribution and read its center and spread
• Decide whether a real result is typical or surprising

Run the Coin Sim 200 Times

The coin model: one trial = 10 spins, record the number of heads.

Now run it 200 times.

• What does the pile of 200 head-counts look like?

Guess the shape. That pile is the key to today's decision.

A Few Trials by Hand

• Each dot = one trial of 10 spins
• After a few trials: jagged, with gaps

Many More Trials Pile Up

• 200 trials: a clear mound, peak near 5
• Extremes (0 or 10 heads) are rare

Reading the Distribution: Center and Spread

• Center: near 5 heads — the model's expected count
• Spread: most trials land between 3 and 7
• Tails: 0 or 10 heads happen, but rarely

Center = what's normal. Spread = natural wobble. Tails = extreme.

Name It: The Reference Distribution

This picture is the reference distribution.

• The full menu of what the model produces, and how often
• It's the yardstick for judging real data

Without it, "surprising" is opinion. With it, it's a measurable position.

More Repetitions Give a Steadier Picture

• 20 trials: jagged → 200: a mound → 2,000: smooth
• Same center and spread — just sharper

This Is A.1 Variability, Made Visible

The spread of this distribution is sampling variability.

• Each dot is one trial's statistic
• The spread = how much the statistic naturally wanders

Last unit said variability was measurable. This is us measuring it.

Read Center and Range Off a Plot

A spinner's reference distribution: mound peaks at 8, most dots between 5 and 11.

• What is the center?
• Outside what range would a result look unusual?

Commit to both before advancing.

We Have the Menu — Now Judge It

We have the reference distribution. Now bring in the real data.

• Drop the observed value onto the distribution
• Estimate the fraction of trials at least as extreme

Tiny fraction → surprising. Large fraction → typical.

The Decision Rule: Locate and Read the Tail

Locate the observed result → shade "at least as extreme" → read the fraction

Coin Case: Five Tails Is Uncommon

Simulate many sets of 5 spins; count all-tails.

• Uncommon — but the kind of thing that happens
• Decision: mild evidence at most; keep the fair-coin model

Free Throw: The Result Sits Deep

Simulate 20-shot sessions under : makes cluster near 16.

• Her real result, 11, sits deep in the lower tail
• Very few simulated sessions made 11 or fewer

Decision: genuine evidence to question the 80% claim

Decide and Justify From the Tail

Both decisions came from tail position, not gut feeling.

• Coin: 1 in 32 → not rare enough → keep the model
• Free throw: deep tail → rare → doubt the claim

Same procedure, two answers, both defensible from the distribution.

A Fair Coin Gave Nine Heads

In our 200 fair-coin trials, a few landed on 9 heads — extreme.

But these came from a coin we built to be fair.

• Does 9 heads prove the coin is unfair?

Commit to yes or no before advancing.

Surprising Is Evidence to Doubt, Not Disproof

That 9-heads trial came from a fair coin — the sim's own counterexample.

• An extreme result can't prove the model false
• It gives reason to doubt, proportional to rarity

Evidence, never proof. This is the core of the standard.

A Fitting Result Never Proves the Model True

A result in the middle is typical — but that doesn't prove the model.

• Many models produce the same unsurprising result
• and both make "6 heads" ordinary

"Consistent with" means "not contradicted" — weaker than "true."

Three Cautions About Surprise and Proof

Surprising → reason to doubt, not proof of false
Fits the model → not proof the model is true
Run-to-run wobble is the variability we measure, not a defect

Each caution blocks a different overclaim. Together they keep you honest.

Decide and Justify Two Cases

Spinner model: red of the time; reference centered at 4 reds in 12.

1. You observe 8 reds — surprising?
2. You observe 4 reds — surprising? (careful!)

Decide and justify both. "Not surprising" ≠ "proven."

Key Takeaways From Lesson Two

✓ Build a reference distribution; read center and spread
✓ Decide by tail position: tiny fraction → surprising
✓ Many reps make the decision robust

Surprising = evidence to doubt, not proof of false
A fitting result never proves the model true

Coming Up Next: The Same Engine, Reused

The tail-reasoning you just learned powers the rest of the unit.

It returns to build a margin of error, and to decide whether a difference between two treatments is real or chance.