Instrument Resolution and Precision Propagation

Lesson 2 of 2: Resolution, Context, and Multi-Step Calculations

In this lesson:

• Identify the resolution limit of a measuring instrument
• Apply appropriate precision in context — neither over- nor under-precise
• Track how precision limits propagate through multi-step calculations

Deck 2 Learning Objectives Today

By the end of this lesson:

1. Match reported precision to instrument resolution — not to calculator output
2. Recognize false precision and under-precision in reported quantities
3. Track precision propagation through multi-step calculations using the weakest-link rule

What Can a Ruler Actually Detect?

Ruler marked in millimeters:

• 23 mm vs. 24 mm → Yes (one tick mark apart)
• 23.4 vs. 23.5 mm → Maybe (visual estimate between marks)
• 23.41 vs. 23.42 mm → No (0.01 mm — ruler can't detect)

The smallest detectable difference is the instrument's resolution.

Instrument Resolution: The Upper Limit

Reported precision cannot exceed the instrument's resolution.

Over-Precision: More Digits Than the Instrument

A bathroom scale reads to the nearest pound. A student steps on it and gets 143 lbs.

After calculation, they report: body mass index = 22.4871

This implies the scale was accurate to 0.0001 pounds. It wasn't.

Over-precision occurs when reported digits exceed what the measurement can support.

Ruler Reads 23 mm: What to Report?

• 23 mm: readable directly ✓
• 23.5 mm: achievable by visual estimation ✓
• 23.41 mm: cannot be read — false precision ✗

Report: 23.5 mm (or 23 mm without estimating)

Match reported precision to the instrument's capability.

When Fewer Digits Are More Honest

A survey of 1,000 households finds average income = $47,325.42

Sampling variability far exceeds ±$1 — so the penny-level precision is false.

Report $47,300 (or ~$47,000) to reflect actual confidence.

Context-appropriate precision reflects real uncertainty.

Match Precision to the Audience and Purpose

Same distance — three reporting contexts:

• GPS app: 4.74 km (3 sig figs — needed for routing)
• Geography textbook: 4.7 km (2 sig figs — general scale)
• Trail sign: "About 5 km" (rough guidance)

Same physical value, different appropriate precision.

Misconception: More Decimal Places Means Better

Wrong: "8.937246 g/cm³ is more accurate than 8.9 g/cm³."

If the mass had ~2 significant figures, the density has ~2 significant figures. The extra digits are the calculator's precision, not the measurement's.

Principle: Garbage in, garbage out.

Quick Check: Is This Precision Appropriate?

Classify each: over-precise, under-precise, or appropriate?

1. Pharmacist reports "about 500 mg" — medication requires ±1 mg
2. Classroom length reported as 8.34729 m using a tape measure
3. Temperature as 37.2°C using a 0.1°C resolution thermometer

Report at the Right Precision Level

A digital scale has a resolution of 0.1 g.

A student weighs a sample and records: 45.6 g

After mixing with another substance, the combined mass is: 45.6 + 12.34 = 57.94 g

What should the student report?

Apply the addition rule — and consider the instrument's resolution.

Single Step vs. Multi-Step Precision

Multi-step calculations create a new challenge: rounding error accumulates.

• Round early → errors compound
• Round at the end → one rounding, minimal error

Carry full precision through intermediate steps. Round the final answer only.

Weakest Input Governs the Final Result

Weakest Link Sets the Precision Ceiling

Final precision = precision of the least precise input.

(still 2 sig figs — 4.2 is the weakest)

More precise length measurements cannot rescue an imprecise width measurement.

Round at the End, Not at Each Step

m, m, m

Round early: , then m³

Round at end: m³

Early rounding gave 61 instead of 60 — an avoidable error.

How Precision Is Reported Across Fields

Full Workflow: Tile Cost with Precision Tracking

Room: 4.2 m × 6.8 m; tile costs 28.50 per m²

1. Identify sig figs in each measurement
2. Calculate area — identify the weakest link
3. Calculate total cost — carry precision through
4. Report the final answer, rounded once at the end

Show all steps.

Before Reporting: Ask How Precisely You Know

The meta-question: Before you write any computed quantity, ask:

"How precisely were my input measurements made?"

• More digits than the inputs support → false precision
• Fewer digits than the inputs support → you're hiding real information

Report what you actually know — no more, no less.

Precision Applies to Every Model You Build

You can now:

• Match reported precision to instrument resolution
• Recognize false precision and under-precision
• Track the weakest link through multi-step calculations

Every modeling task in future mathematics, science, and data work starts with measured inputs — and their precision limits every result.