Running the test and checking the data
🎯You'll be able to
Check randomisation with a sample ratio test, remove bots and broken assignments, and know when a test's data can't be trusted.
🏠The everyday picture
If a coin toss is meant to split people into two queues, and one queue ends up much longer, something is wrong with the coin or someone is jumping queues. You'd fix that before comparing the queues.
📘What it really is
-
Randomise properly: assign each visitor by a hash of their ID or a random draw, store the assignment, and keep it for the whole test.
-
Sample ratio mismatch (SRM): the split differs from the plan (for example 50/50) by more than chance allows. A chi-square test gives a p-value; a very small one (below 0.001 is a common alarm) means the assignment or tracking is broken.
-
Common causes: bots hitting one variant, redirects that lose visitors in one variant, tracking that fails on one version, visitors assigned twice.
-
Clean, then check again:
- remove rows flagged as bots;
- remove visitors who appear in both variants (they saw both, so they tell you about neither);
- count per visitor, not per order or page view;
-
re-run the SRM check on the cleaned data.
-
Don't change the test mid-way: no new variants, no changed traffic split, no edits to B.
- Run full weeks, and avoid launching in an unusual period (a sale, a festival, an outage) unless that's what you're testing.
📝A worked example
Input: MittiMart's experiment/assignments.csv: 24,168 rows; raw split A 12,718 and B 11,450.
Steps:
- SRM on the raw split: p-value below 0.001: broken.
- Remove 1,260 bot rows (all in A, a crawler loading only the control page) and 154 visitors who appear in both variants.
- SRM again on the 22,600 visitors left (A 11,304, B 11,296): p-value 0.958.
Expected output: the Data cleaning section of experiment/report.md (Rows, Bot rows removed, Visitors in both variants removed, Visitors
analysed), which Project 2's test 4 checks, and an SRM p-value of 0.958 for test 6.
🌍Where it's used
- Large experimentation platforms run SRM checks automatically on every test.
- Analysts filter known bots and internal staff before analysis.
- Engineers fix assignment bugs found by SRM alerts.
📝Case scenario
A travel site's test showed B winning by 8%. An SRM check found B had 6% fewer visitors than A: B's new page redirected some mobile users, and many were lost before tracking started. The ones who remained were keener buyers. After the redirect was fixed and the test re-run, the "win" disappeared.
🖱️Try it
In the kit, after fixing srm, run it on 5,100 against 4,900 and on 51,000 against 49,000. Success: the first is about 0.046 and the
second is far below 0.001: the same 51/49 split is much less likely by chance with more visitors.
⚠️Common mistakes
- Skipping the SRM check. Sign: comparing rates on a broken split. Fix: check the split before the results.
- Keeping visitors seen in both variants. Sign: one visitor counted in A and B. Fix: remove them, and fix the assignment.
- Counting orders instead of visitors. Sign: "Converted: 735" from 694 people. Fix: visitors with at least one order.
🛠️For your project
Project 2's tests 2, 4 and 5 check your srm function, the cleaning counts and the per-visitor results table.
✅Quick check
What does a very small SRM p-value tell you?
Show the answer
The split is broken (for example by bots or a redirect), so the results can't be trusted yet.
Why remove visitors who appear in both variants?
Show the answer
They saw both versions, so they can't tell you about either.