ML2: Preparing data without fooling yourself
Leakage: when the answer slips into the inputs
🎯You'll be able to
Spot leakage from future information, from the target itself and from the way data was collected, and remove it.
🏠The everyday picture
A quiz where the answer sheet is printed on the back of the question paper gives perfect scores and teaches nothing. Leakage is an answer sheet hidden in your data: the model gets the answer from a column instead of learning the pattern.
📘What it really is
-
Leakage is any information in the inputs that wouldn't be available, in that form, at the moment of prediction. It makes test scores look better than real use.
-
Three common kinds:
- Future information: columns filled in after the moment of prediction, such as
delivered_atordelay_hours. - Target in disguise: a column that is the answer under another name, such as a "refund issued for lateness" flag.
- Collection leakage: the way rows were collected gives the answer away, such as "late" parcels coming from a complaints file with different formatting.
-
Warning signs: a score that is almost perfect, one feature that is far more important than all the others, or a big drop in real use.
-
Subtle cases need a decision. The kit's
hub_loadis the number of parcels at the hub on the booking day. In the kit it stands for the hub's planned load, published the evening before, so it is known at booking. If it were the final count at the end of the day, it would be future information for a 9 a.m. booking. Write down how each borderline feature is produced.
📝A worked example
Input: the starter kit's dakdost/features.py, which adds delay_hours to the features.
Steps and expected output (reference measurements, 2 November 2026):
- Train gradient boosting with
delay_hoursincluded: PR-AUC on June is 1.0, a perfect score. - Check the data:
((df['delay_hours'] > 0).astype(int) == df['late']).mean()returns1.0. The target is just "delay above zero". - Permutation importance (ML7 Lesson 2) on May shows
delay_hoursat 0.919 and every other feature at 0.0. - Remove
delay_hoursand train again: PR-AUC on June is 0.6235, a real result.
🌍Where it's used
- Credit: a "collections team assigned" column leaks whether a loan defaulted.
- Health: a test that is only ordered for very sick patients leaks the diagnosis.
- Churn: an "account closed date" column leaks whether a customer left.
📝Case scenario
A model to predict which online orders would be returned scored 0.98. Its most important feature was "return label printed". The label is printed after the customer asks for a return, so the model was predicting returns that had already started. Without it, the score was 0.71, which was still useful, and the team could finally trust it.
🖱️Try it
For each column of raw/shipments.csv, write when it becomes known: at booking, during delivery or after delivery. Success: you mark
delivered_at and delay_hours as after delivery, and you write one sentence on how hub_load must be produced for it to be safe.
⚠️Common mistakes
- Celebrating a perfect score. Sign: PR-AUC near 1.0 on a hard problem. Fix: look for the column that gives the answer away.
- Checking columns only by name. Sign: a harmless-sounding column leaks. Fix: ask how and when each column is filled in.
- Leaving leakage for later. Sign: models are compared while one has a leaky feature. Fix: remove leakage before any comparison.
🛠️For your project
Project 1's test 3 checks that no feature is known only after delivery or is personal; test 5 scores the model with those columns removed.
✅Quick check
What's the first thing to suspect when a score is almost perfect?
Show the answer
Leakage: a column that gives the answer away.
Why is delay_hours leakage?
Show the answer
It's only known after delivery, and it decides late exactly.