Uncertainty, calibration and honest reporting
🎯You'll be able to
Put a range around a score with the bootstrap, check whether probabilities are calibrated, and write an evaluation that others can trust.
🏠The everyday picture
A weather forecast that says "70% chance of rain" is useful only if it rains on about 70% of such days. And a cricket average from five innings tells you less than one from fifty. Good reports say how sure they are.
📘What it really is
- Every score has noise. June has 215 late parcels; a different month would give a somewhat different score.
-
The bootstrap estimates the range: resample the test rows with replacement many times (say 1,000), compute the score each time, and take the middle 95% of results. For June's PR-AUC the range is about 0.558 to 0.693.
-
Calibration asks whether predicted probabilities match reality: of parcels given about 30%, are about 30% late? Check by grouping rows by predicted probability and comparing the average prediction with the actual rate.
-
Recalibrating (for example with isotonic regression fitted on validation data) can fix a model that is consistently too high or too low. It can't fix a shift that only appears in a later month.
-
An honest report gives: the data and dates; the baseline; the main measure with its range; the threshold and its rule; precision and recall; results by group (ML7); what wasn't tested; and the test set used once.
📝A worked example
Input: June, the kit's model (reference figures, 2 November 2026).
- Bootstrap, 1,000 resamples, seed 0: PR-AUC 0.6235 with a 95% range of about 0.558 to 0.693.
- Calibration by fifths of predicted probability: the top fifth averages a prediction of 0.287 but 0.381 were late; the lower fifths are also below their actual rates. Overall the model predicts 0.0697 on average against a true 0.0968.
- Why: June had more late parcels than earlier months, and the model has no month feature. On May its average prediction (0.0799) was close to the truth (0.082).
- Isotonic recalibration fitted on May didn't help on June (Brier score, the average squared error of the probabilities, went from 0.0557 to 0.0566), because the gap comes from June itself.
The honest summary: "ranks well (PR-AUC 0.62, range about 0.56 to 0.69); probabilities ran low in June; threshold chosen on May".
🌍Where it's used
- Planning: when predicted probabilities are added up into an expected number (for example, extra vans), calibration matters.
- Risk communication: showing a customer "30% chance" needs calibrated scores.
- Reviews and audits: ranges and stated limits are what reviewers look for.
📝Case scenario
A team announced that a new model beat the old one by 0.02 PR-AUC. A reviewer ran the bootstrap on the difference itself and found the range ran from −0.01 to +0.05: probably better, but not certainly. They launched it as a trial on two hubs, with a plan to compare after a month, rather than a full switch.
🖱️Try it
Run a bootstrap for June's recall at threshold 0.3 with 1,000 resamples. Success: you report the recall (0.5349) and a range, and the range is wide enough to make you careful about small differences.
⚠️Common mistakes
- One number, no range. Sign: "PR-AUC 0.6235" as if exact. Fix: add a bootstrap range.
- Trusting probabilities without checking. Sign: planners add predictions up and get the wrong total. Fix: check calibration on recent data.
- Recalibrating on the test set. Sign: June used to fit the calibration. Fix: fit on validation; check on test.
🛠️For your project
Project 1's model card reports the June figures with the date ranges, the threshold rule and results by zone; your case study adds the range.
✅Quick check
What does the bootstrap give you?
Show the answer
A range for a score, by recomputing it on many resamples of the test rows.
What does "calibrated" mean?
Show the answer
Predicted probabilities match how often the event really happens.