Task results and small numbers
🎯You'll be able to
Count task results correctly, present them as counts, and explain what small numbers can and can't show.
🏠The everyday picture
If 2 of 5 cooks burn the same recipe, you know the recipe has a problem. You don't know that 40% of all cooks in India will burn it.
📘What it really is
-
Task results: for each task, count completed without help, completed with help and failed. Keep the three separate; "completed" hides the help.
-
Show counts, not percentages, for small groups: "2/5". With five people, one person changes the share by 20 points.
- Time on task can be useful, but think-aloud slows people down, so compare times only within the same kind of study.
-
What five sessions show: that a problem exists, roughly how serious it is, and often why. Studies of usability testing suggest that a handful of participants finds most of the biggest problems in a design, but not all, and not rare ones.
-
What they don't show: how common a problem is in the whole population, or whether one design is better than another by a small margin. Those need larger samples (surveys, analytics or A/B tests).
-
Confidence words: "all five", "most (4 of 5)", "some (2 of 5)", "one person". Avoid sweeping claims about "users" in general.
📝A worked example
Input: the result lines in research/sessions/.
grep -h '^Result' research/sessions/*.md | sort | uniq -c
Expected output:
2 Result T1: completed
2 Result T1: completed with help
1 Result T1: failed
3 Result T2: completed
2 Result T2: failed
2 Result T3: completed
1 Result T3: completed with help
2 Result T3: failed
So the task results table reads T1 2/5, 2/5, 1/5; T2 3/5, 0/5, 2/5; T3 2/5, 1/5, 2/5. Only 2 of 5 booked an appointment without
help. Watch the trap: grep -h 'Result T1: completed' research/sessions/*.md | wc -l also matches "completed with help", which would give 4 instead of 2.
🌍Where it's used
- Before-and-after testing: the same tasks with a new design, compared as counts.
- Benchmark studies: larger numbers of participants, used when teams need a rate they can track.
- Accessibility testing with disabled participants, where each result matters even though numbers are small.
📝Case scenario
A team reported that a redesign raised task success "from 60% to 80%". The numbers were 3/5 and 4/5: one more person. When the director asked for the redesign to be rolled out because of a "20-point improvement", the team explained what the counts could and couldn't show, and ran a larger unmoderated test before launch.
🖱️Try it
Using the result lines, write the task results table in the kit's format. Then write one sentence about T2 that is honest about the sample size. Success: your counts match the expected output, and your sentence uses "of 5".
⚠️Common mistakes
- Merging "with help" into "completed". Sign: T1 shows 4/5. Fix: three separate counts.
- Percentages from five people. Sign: "40% failed". Fix: "2 of 5 failed".
- Search that matches too much. Sign: counts don't add up to 5. Fix: match the whole line, and check that each task adds to 5.
🛠️For your project
Project 1's test 6 recomputes the three counts for each task from the notes and compares them with your table.
✅Quick check
Why keep "completed with help" separate?
Show the answer
Because help hides a design problem; counting it as completed would overstate how well the design works.
Can five sessions show how common a problem is?
Show the answer
No: they show that it exists and roughly how bad it is, not its rate in the whole population.