The Observed Difference and the No-Effect Model

Lesson 1 of 2: Setting Up the Test

In this lesson:

• Compute the observed difference between two treatment groups
• State the no-effect assumption as the model to test

Learning Objectives for This Unit

By the end of this two-lesson unit, you should be able to:

1. Compute a difference in summary statistics between two groups
2. State the no-effect assumption as the model to test
3. Build a re-randomization distribution by shuffling labels
4. Locate the observed difference and decide significance
5. Interpret a result causally, given random assignment

A 4-Point Gap: Real Effect or Luck?

40 volunteers randomly assigned: 20 to a new method (T), 20 to the usual (C).

• New method improved by 12 points on average
• Usual method improved by 8 points

A 4-point gap. Real effect — or the luck of who landed where?

The Experiment: Two Groups, 20 Each

• Treatment mean 12, control mean 8 — and the groups overlap

The Observed Difference: 12 − 8 = 4

The observed difference is the gap between the group means.

• This is the headline number
• But alone, it can't separate a real effect from luck

Computing the gap is step one. Testing it is the rest.

The Chance Explanation Is Live

Maybe the stronger students just happened to land in the new-method group.

• If so, that group scores higher even if the method does nothing
• Random splits routinely produce gaps

Before crediting the method: could this be a lucky draw?

A Proportion Version: 70% vs 55%

Instead of means, suppose we measured recovery:

• Treatment: 70% recovered; Control: 55%
• Observed difference: 15 percentage points

Same question — real effect or luck? The method handles means and proportions alike.

Assignment Licenses Cause — If More Than Chance

Random assignment balanced the groups — so a real gap is caused, not just associated.

• This is the experiment's power over observation
• But only if the gap is more than chance

Assignment earns the causal reading; the test cashes it in.

Compute the Difference; Name T and C

40 patients randomly assigned to a new or standard painkiller.

• New: 6-point average reduction; Standard: 3.5
• Observed difference? Which is T? Which is C? Response variable?

Do all three before advancing.

To Rule Out Chance, Assume No Effect

To test the gap, we'll assume the treatment does nothing.

• Then ask: how surprising is a 4-point gap under that?
• Same logic as model-fitting — assume a model, test it

If "no effect" rarely makes a 4-point gap, we doubt "no effect."

Stating the No-Effect Assumption Precisely

Assume the treatment has no effect at all.

• Then each student's score is the same in either group
• Their improvement was going to be what it was, regardless

This is the model we'll test — and it's surprisingly powerful.

Outcomes Are Fixed, Labels Are Arbitrary

• The outcome on each card is fixed; only the label is arbitrary
• The gap is about which students got the T label

The Gap Is "Which Students Got Labeled T"

Under no effect, the 4-point gap exists only because of how labels fell.

• A few higher scorers landing in T → a gap appears
• Nothing about the treatment caused it

If labels are arbitrary, we can re-deal them — that's next lesson.

Same Logic as Model-Fitting, New Model

This is the same logic as testing whether data fits a model.

• Before: assume a fair coin; is the data surprising?
• Now: assume no effect; is a 4-point gap surprising?

Same machine, new model. You've seen this shape before.

Why Exchangeable Labels Matter So Much

Under no effect, the labels are exchangeable — swapping them is just as valid.

• That exchangeability is what lets us shuffle the labels
• Miss this, and the next lesson feels like a ritual

Exchangeable labels are the permission slip for shuffling.

State the No-Effect Assumption Yourself

A gardener randomly assigns 30 seedlings to a new fertilizer or plain water.

• State the no-effect assumption in your own words
• What does it imply about the labels?

Don't compute — articulate. Write it before advancing.

Key Takeaways From Lesson One

✓ Observed difference = gap between group means (here, 4)
✓ A gap alone can't separate effect from luck
✓ No effect → outcomes fixed, labels arbitrary

A nonzero gap is not proof the treatment worked
The no-effect model centers at zero, not at 4

Coming Up Next: Shuffle the Labels

In Lesson 2, you'll shuffle the arbitrary labels hundreds of times.

That reveals exactly how big a gap chance alone makes — so you can judge whether your 4 is more than chance.