Hypotheses and metrics
🎯You'll be able to
Write a testable hypothesis, choose one primary metric and guardrail metrics, and decide the unit of randomisation.
🏠The everyday picture
A farmer who wants to know whether a new seed grows better plants it in some fields and the old seed in others, chosen by coin toss, and decides beforehand to measure the harvest in kilograms, not how green the leaves look.
📘What it really is
-
An A/B test shows version A (the control, the current page) to some visitors and version B (the change) to others, assigned at random, and compares an outcome.
-
A hypothesis names the change, the expected effect and why: "Showing 'Arrives by …' above Add to cart will raise the share of visitors who order, because customers ask about delivery times."
-
One primary metric decides the test. Choose it before starting: for MittiMart, the share of visitors with at least one order.
- Guardrail metrics must not get worse: average order value, refunds, page speed, complaints.
- The unit of randomisation is usually the visitor (so each person sees one version throughout), not the page view.
- Is a test worth it? You need enough traffic (Lesson 2), a change you can ship either way, and a question that matters.
- Write the plan first (MittiMart:
experiment/next-test-plan.md): hypothesis, metrics, sample size, duration, stop rule.
📝A worked example
Input: an idea: "Free delivery messages are good. Let's try it and see what happens." (the starter's next-test plan).
Steps:
- Name the change: "Add ₹… more for free delivery" in the cart.
- Name the effect and reason: more cart visitors order, because people are surprised by the delivery charge at checkout.
- Choose the primary metric (cart visitors who order in the same visit) and guardrails (order value, refunds, cart page speed).
Expected output: the Hypothesis, Primary metric and Guardrail sections of experiment/next-test-plan.md.
🌍Where it's used
- E-commerce teams test product pages, delivery messages and checkout steps.
- Fintech apps test onboarding screens with guardrails on fraud and complaints.
- Publishers test headlines with a guardrail on how long people read.
📝Case scenario
A food app tested a bigger "Order now" button and measured clicks. Clicks rose 15%, but orders didn't: people tapped it by accident and backed out. Choosing orders as the primary metric, with clicks as background, would have avoided a misleading win.
🖱️Try it
Write a hypothesis for one change to a site you use: the change, the expected effect and why. Add one primary metric and one guardrail. Success: someone else could tell whether the test supported it.
⚠️Common mistakes
- "Let's see what happens." Sign: no expected effect. Fix: a hypothesis with a reason.
- Several primary metrics. Sign: "We'll ship if clicks, time on page or orders rise." Fix: one metric decides; others inform.
- Randomising page views. Sign: one person sees A then B. Fix: randomise visitors.
🛠️For your project
Project 2's test 8 checks that experiment/next-test-plan.md has a Hypothesis, Primary metric, Guardrail, Sample size, Duration and Stop
rule.
✅Quick check
What's a guardrail metric?
Show the answer
A metric that must not get worse, such as order value or refunds, while the primary metric is tested.
Why randomise by visitor rather than by page view?
Show the answer
So each person sees one version throughout, and results aren't mixed.