Sample size and power
🎯You'll be able to
Explain significance level, power and minimum detectable effect, and calculate the visitors a test needs.
🏠The everyday picture
To tell whether a coin is slightly unfair, ten tosses aren't enough: you'd need hundreds. The smaller the difference you want to detect, the more tosses it takes.
📘What it really is
- Baseline: the current rate (for example 3.0% of visitors order).
- Minimum detectable effect (MDE): the smallest change worth detecting, here in percentage points (0.6 points: from 3.0% to 3.6%).
- Significance level (alpha): how often you'd accept a false win when there's no real effect; usually 5%, two-sided.
- Power: the chance of detecting the MDE if it's real; usually 80%.
-
The formula for each variant's sample size, for comparing two proportions p1 and p2: n = (z₁₋α/₂ × √(2p̄(1 − p̄)) + z₁₋β × √(p1(1 − p1) + p2(1 − p2)))² ÷ (p2 − p1)², where p̄ is the average of p1 and p2, z₁₋α/₂ ≈ 1.96 and z₁₋β ≈ 0.84 for 80% power.
-
For MittiMart: baseline 3.0%, MDE 0.6 points → 13,914 visitors per variant.
- Halving the MDE roughly quadruples the sample. Small effects on low rates need a lot of traffic.
- Duration: visitors needed ÷ daily visitors, rounded up to whole weeks so every weekday is included.
- If you can't reach the sample size, test a bigger change, a higher-rate metric, or don't test and decide another way.
📝A worked example
Input: sampleSize({ baseline: 0.03, mde: 0.006 }), with about 1,500 visitors a day entering the test.
Steps:
- Calculate: 13,914 per variant, 27,828 in all.
- Days: 27,828 ÷ 1,500 = 18.6, so 19 days.
- Round up to whole weeks: 21 days.
Expected output: the Sample size and Duration sections of the plan, and Project 2's test 1 passing when sampleSize returns 13,914 (and
10,921 for a 5% baseline, a 1-point MDE and 90% power).
🌍Where it's used
- Experimentation teams run a sample size check before any test starts.
- Product managers decide whether a test is feasible with current traffic.
- Agencies explain to small clients why some tests aren't worth running.
📝Case scenario
A small online bookshop with 300 visitors a day wanted to detect a 5% relative lift on a 2% conversion rate. The calculation said more than 3 lakh visitors per variant: almost six years of traffic. They tested a much bolder change instead, and used customer interviews for the small ideas.
🖱️Try it
In the kit, call sampleSize (after fixing it) with a baseline of 10% and an MDE of 2 points, then with an MDE of 1 point. Success: the
second needs roughly four times as many visitors (about 3,841 and about 14,750).
⚠️Common mistakes
- No sample size. Sign: "Let's run it for a week." Fix: calculate first.
-
Mixing relative and absolute changes. Sign: "a 20% lift" meaning 3.0% to 3.6% on one slide and 3.0% to 23% on another. Fix: say "percentage points" for absolute changes.
-
One-sided tests by default. Sign: z = 1.645 with no reason. Fix: two-sided unless a one-sided test was justified beforehand.
🛠️For your project
Project 2's test 1 checks your sampleSize; test 8 checks the plan's visitors per variant and days.
✅Quick check
What happens to the sample size if you halve the minimum detectable effect?
Show the answer
It roughly quadruples.
Why round a test's duration up to whole weeks?
Show the answer
So every day of the week, with its different visitors, is included equally.