Digital Marketing (Growth and Analytics) · Projects

Project 2: An A/B test reported honestly

Marking guide version 1 · Lessons: DM6 lessons 1, 2 and 5, DM7 lessons 1–5, DM10 lesson 3

The problem and who it's for

MittiMart tested showing a delivery date on product pages (B) against no date (A) for 14 days, from 7 to 20 September 2026. After three days, someone wrote a report saying "B wins! 62% uplift", from raw data, and recommended stopping the test. The raw data contains bot visits, all in A, and visitors who saw both versions, and its split between A and B is far from 50/50. The stats.js file meant to do the sums has mistakes and is missing the split check. The shop's owners want to know whether to show delivery dates, and how sure they can be.

Scope

• Fix stats.js: sampleSize, srm (the sample ratio check) and twoProportion (difference, two-sided p-value and 95% interval).
• Clean the data: remove bot rows and visitors who appear in both variants, and say how many of each were removed.
• Count visitors and converted visitors (visitors with at least one order, not orders) per variant, after cleaning.
• Check the split before and after cleaning, then run the test on the cleaned data.
• Rewrite experiment/report.md in the format in experiment/README.md, with a decision that matches the evidence and honest limits.
• Plan the next test in experiment/next-test-plan.md, with its size and duration worked out before it starts.

Not included

• Bayesian methods, sequential testing or tests on revenue per visitor (named in DM7 as next steps).
• A real experimentation tool or feature flag service (DM9 covers flags; optional extra).

Before you start

• Core: Module 21 (Arrays and Objects), Module 22 (Loops and Algorithms), Module 26 (Debugging and Testing), Module 55 (SQL Fundamentals).
• Path: DM6 and DM7 complete.
• npm run test:experiment running (all 8 of its tests fail at first).

Starter files

• mittimart/experiment/assignments.csv: 24,168 invented rows, one per visitor and variant, with the day, device and an is_bot flag.
• mittimart/experiment/conversions.csv: the orders, with the visitor, the day and the order value in paise.
• mittimart/experiment/stats.js: the statistics functions, with mistakes and no srm; each says what it should return.
• mittimart/experiment/report.md: the "B wins!" report written after three days.
• mittimart/experiment/next-test-plan.md: an idea, not a plan.
• mittimart/experiment/README.md: the data's columns and the headings and labels the report must use.
• mittimart/tools/generate.js: rebuilds the data files exactly (npm run generate).
• mittimart/tests/experiment.test.js: the 8 acceptance tests for this project.

Download everything at once: starter/digital-marketing-starter.zip.

Milestones

1. Run npm run test:experiment and read what each of the 8 tests expects.
2. Fix sampleSize, write srm and fix twoProportion (tests 1, 2 and 3).
3. Clean the data and write the Data cleaning section (test 4).
4. Count visitors and converted visitors per variant and write the Results table (test 5).
5. Write the Test, Decision and Limits sections (tests 6 and 7).
6. Write the next test plan (test 8), then run all 8 tests twice in a row.

Expected outputs

• 8 of 8 experiment tests passing, twice in a row.
• The fixed stats.js, the rewritten report.md and next-test-plan.md.
• A one-paragraph answer for the shop's owners: what the test found, how sure you are, and what you'd do next.

Acceptance tests

Failure and recovery exercises

1. Count orders instead of converted visitors for B and run test 5. Explain why a visitor who ordered twice shouldn't count twice in a conversion rate, then restore your counts.
2. Run twoProportion on the first three days of raw data (A 68 of 2,537, B 100 of 2,299) and then on the full cleaned data. Write two sentences on why the first answer looked so convincing and why it was wrong to stop there.
3. Skip the cleaning and run srm on the raw counts. Explain what a sample ratio mismatch tells you about a test before you look at any result.

Evidence to submit

• The deliverable link to your repository or folder with the experiment changes.
• The output of npm run test:experiment showing 8 passed, pasted into test evidence.
• Your report.md, next-test-plan.md and the paragraph for the shop's owners.
• Your explanation: why the bots and visitors in both variants were removed, what the interval says, and why "no clear difference" is a useful result.

Marking guide and self-check

Pass rule: every Safety-critical (S) and Mandatory (M) item must be met. A single unmet safety item means the project is not complete, whatever else is done. Quality items are for your own improvement.

Your answers are saved in this browser on this device. Self-checked means you checked your own work against this guide; it is not a reviewer's approval. Never paste passwords, keys, tokens or real customer data here.