When to Doubt a Model, Designing a Simulation

Lesson 1 of 2: From Doubt to Design

In this lesson:

• Decide when a surprising result should make you doubt a model
• Design a simulation of a model's data-generating process

Learning Objectives for This Unit

By the end of this two-lesson unit, you should be able to:

1. Explain that surprise is evidence, not disproof
2. Design a simulation by its four choices
3. Build and read a simulated distribution
4. Locate a result; judge typical or surprising
5. Decide whether data questions the model

Five Tails: Rigged or Ordinary?

A coin is supposed to be fair: .

You spin it and get 5 tails in a row.

• Is the coin rigged?
• Or did you just see something ordinary?

Right now, all you can do is shrug. Let's do better than that.

How Much Doubt? Watch It Grow

• 3 tails? 5? 10? 20? Your doubt climbs
• Yet every one is possible under a fair coin

Surprising Is Not the Same as Impossible

• Consistent with the model ≠ the only allowed result
• Surprising ≠ impossible

A fair coin can give 20 tails — it just almost never does.

Your doubt tracks how often, not whether. That's what we'll measure.

The Logic Behind the Whole Standard

Assume the model → ask how often this extreme → let rarity govern doubt

Even True Models Make Extreme Data

A genuinely fair coin will sometimes give 5 — even 10 — tails.

• So a surprising result is evidence to doubt
• It is never proof the model is false

Surprise questions the model. It never rejects it outright.

Rank These Results by Surprise

Under a fair coin (10 spins), rank from least to most surprising:

• A. 6 heads B. 9 heads C. 10 heads

Reason about how often each happens — don't calculate. Commit before advancing.

From Gut Feeling to a Machine

You can feel surprise — but feeling isn't a method.

• Two people disagree; no one can defend a threshold

So we build a simulation: make the model produce results hundreds of times.

Now we can see how often the model makes data this extreme.

The Free-Throw Claim: 80%, but 11 of 20

A player claims she makes 80% of free throws.

In one session of 20 shots, she makes only 11.

• Is 11 of 20 surprising if the 80% claim is true?

We'll design a simulation that assumes 80% and counts how often 11 happens.

Design Choice 1: The Chance Device

The device must reproduce the model's probability (80% make).

• A spinner with 80% of its area shaded "make"
• Random digits: 0–7 = make, 8–9 = miss

The device is interchangeable — any one that gives 80% works.

Design Choice 2: One Trial

One trial = one full imitation of the real data collection.

• She really took 20 shots → one trial is 20 simulated shots
• Not a single shot

This is the choice everyone gets wrong. Here's why it matters.

One Trial Is 20 Shots, Not One

• Ask: what number did I observe? 11 out of 20
• One trial must produce a count out of 20

Design Choices 3 and 4

• Statistic recorded each trial: the number of makes out of 20
• Repetitions: many — 500 or 1,000 — so the pattern stabilizes

A handful of trials is too jagged. Many trials give a trustworthy picture.

Three Devices, All One Model

• Spinner, digit table, random function — all give 80%
• Pick any one; the model is identical

The Full Design, Now Assembled

For the 80% free-throw claim:

• Device: random digits, 0–7 = make
• One trial: 20 digits (20 shots)
• Statistic: count makes out of 20
• Repetitions: 1,000

A complete, runnable design — built, not yet run.

Your Turn: Design a Simulation

A website claims 60% of visitors return. In 30 visitors, only 14 returned.

Specify all four choices:

• Device? One trial? Statistic? Repetitions?

Careful with "one trial" — the real data was 30 visitors. Do all four first.

Key Takeaways From Lesson One

✓ Surprising ≠ impossible; doubt tracks how often
✓ A simulation needs four choices: device, trial, statistic, reps

Watch out: "5 tails" is very unlikely, not impossible
Watch out: one trial = the whole sample (20 shots), not one shot

Coming Up Next: Run the Machine

In Lesson 2, you'll run the simulation hundreds of times and build the picture of what the model produces.

Then you'll drop the real data on it and decide: typical, or surprising?