ML9: MLOps: tracking, monitoring and retraining
Drift: when the world moves
🎯You'll be able to
Measure input drift with the population stability index, tell input drift from a change in the relationship, and avoid false alarms.
🏠The everyday picture
A shop that stocks for summer finds in the monsoon that people want umbrellas, not sunglasses. Nothing about the shop changed; the customers' needs did. Models face the same thing.
📘What it really is
- Data drift (input drift): the spread of inputs changes, such as more remote parcels or more poor addresses.
-
Concept drift: the link between inputs and the outcome changes, such as the same address score now meaning more delay because roads are flooded.
-
The population stability index (PSI) compares the share of rows in each bin of a feature, in training and now:
PSI = Σ (now − then) × ln(now ÷ then). A common rule of thumb: below 0.1 little change, 0.1 to 0.2 a warning, above 0.2 an alert. The kit uses those levels. -
Bins matter. For numbers, the kit uses five bins at training-data quantiles. Values that come in clumps, such as
hub_load(one count per hub per day), can give false alarms with more bins. -
Drift isn't always bad. It's a reason to look: check results by group and the share flagged, and ask the business what changed.
- Concept drift needs outcomes. Only when the real results arrive can you see that the model is wrong more often.
📝A worked example
Input: python -m dakdost.drift raw/new/2026-07-normal.csv and python -m dakdost.drift raw/new/2026-08-monsoon.csv.
Expected output (reference solution, 2 November 2026): July shows no drift (every PSI at or below 0.022) and exits with code 0. August exits with code 1:
| Feature | August PSI | Level |
|---|---|---|
address_quality |
0.269 | alert |
dest_zone |
0.259 | alert |
promised_days |
0.135 | warn |
distance_km |
0.083 | ok |
Remote parcels rose from 9.7% of training to 28.8% in August, and poor addresses (score below 0.5) from 12.0% to 36.1%. With 20 bins
instead of 5, July's hub_load scores 0.567, a clear alert, for a month that hasn't really changed: a false alarm caused by clumped
values. With 10 bins it scores 0.079.
🌍Where it's used
- Seasonal businesses: festival sales, monsoon logistics, exam seasons.
- Credit: shifts in applicants after a marketing campaign.
- Health: a new hospital, machine or patient group.
📝Case scenario
A lender's model kept working well for years, then defaults rose sharply after an economic shock. Input drift was small at first: the applicants looked similar. The change was in the link between income and repayment, which only showed as outcomes arrived. The lender now watches results monthly, not only inputs.
🖱️Try it
Run the drift check on both new files, then change bins=5 to bins=20 in dakdost/drift.py and run it on July again. Success: you see
hub_load above 0.5 with 20 bins, change it back, and write one sentence on why clumped values cause it.
⚠️Common mistakes
- Treating every alert as a crisis. Sign: retraining after each alert. Fix: look first: is it real, and does it affect results?
- Only input drift. Sign: a model getting worse while inputs look stable. Fix: watch results when outcomes arrive.
- Bins that don't suit the data. Sign: alerts on months that didn't change. Fix: check the bins on a normal month.
🛠️For your project
Project 2's test 9 checks that July shows no drift and that August flags address_quality and dest_zone.
✅Quick check
What's the difference between data drift and concept drift?
Show the answer
Data drift is a change in the inputs; concept drift is a change in how inputs relate to the outcome.
What PSI level does the kit treat as an alert?
Show the answer
Above 0.2.