ML7: Responsible AI: fairness, explanations, privacy and security
Explaining models
🎯You'll be able to
Use permutation importance for an overview, "what if" changes for a single prediction, and explain both in plain words.
🏠The everyday picture
A cricket coach explains a loss two ways: overall ("we lost wickets in the middle overs") and for one moment ("that run-out happened because both batters ran for the same end"). Model explanations also come in overall and single-case forms.
📘What it really is
- Global explanations describe the model overall:
- permutation importance shuffles one feature's values on validation data and measures how much the score drops. A big drop means the model relies on it. It works for any model.
-
Partial dependence shows how the average prediction changes as one feature changes.
-
Local explanations describe one prediction:
- what if: change one input and see how the prediction moves;
-
SHAP values split one prediction into contributions from each feature (the
shaplibrary; not used in this kit). -
Explanations describe the model, not the world. "The model relies on address quality" doesn't mean better addresses cause on-time delivery, though here it's plausible.
-
Linked features share importance (ML3 Lesson 1). Shuffling one of two linked features may show a small drop because the other covers for it.
-
Say it plainly for the person affected: "This parcel was flagged mainly because the address check found it incomplete."
📝A worked example
Input: permutation importance on May for the kit's model, measured as the drop in PR-AUC, 5 repeats (reference figures, 2 November 2026).
| Feature | Drop in PR-AUC |
|---|---|
address_quality |
0.4234 |
dest_zone |
0.0684 |
pickup_slot |
0.0357 |
distance_km |
0.0146 |
payment |
0.0141 |
booked_hour |
−0.0103 |
Then a "what if" for one parcel (remote, evening pickup, cash on delivery, hub load 52, address score 0.31): probability 0.8533. With an
address score of 0.6 it's 0.5077; with 0.9 it's 0.4285. With a local destination instead of remote it's 0.7611. The hub team can be told:
"mostly the address; fixing it roughly halves the risk". A negative drop, like booked_hour's, means shuffling it made no real
difference; the small change is noise.
🌍Where it's used
- Credit decisions: reasons given to applicants who are refused.
- Model reviews: checking the model relies on sensible features and not on leakage (ML2 Lesson 2).
- Operations: telling staff what to fix, not only what to expect.
📝Case scenario
A model to spot pneumonia in chest X-rays scored well, but explanation maps showed it looked at a marker in the image's corner that differed between hospitals. One hospital had sicker patients, so the marker predicted the diagnosis. Removing the markers from the images, and testing on a new hospital, gave a more honest score.
🖱️Try it
Run sklearn.inspection.permutation_importance on May with scoring='average_precision', n_repeats=5 and random_state=0. Success: your
top three features match the table, and you can explain the top one in one sentence for the hub team.
⚠️Common mistakes
- Explaining with training data. Sign: importance reflects memorised rows. Fix: use validation data.
- Treating importance as cause. Sign: "zone causes lateness". Fix: say "the model relies on".
- Explanations nobody can read. Sign: a chart of numbers for a customer. Fix: one plain sentence and what they can do.
🛠️For your project
The capstone design notes ask how a hub team member can see why a parcel was flagged.
✅Quick check
What does permutation importance measure?
Show the answer
How much the score drops when one feature's values are shuffled.
What's a "what if" explanation?
Show the answer
Changing one input of a single case and seeing how its prediction moves.