ML3: Classic models with scikit-learn
Tuning and choosing a model fairly
🎯You'll be able to
Tune a model's settings on validation data, choose between candidates, and know when a difference is too small to matter.
🏠The everyday picture
Choosing a cricket team, selectors pick from players' recent form, not from one lucky innings. They also pick the steady player over the flashy one if the scores are close. Choosing a model works the same way.
📘What it really is
-
Hyperparameters are settings you choose before training: depth, number of trees, learning rate,
C. Parameters are what the model learns: weights and splits. -
Tuning tries combinations on validation data. Grid search tries every combination from a list; random search tries random combinations and often finds good ones faster.
-
Choosing fairly:
- every candidate gets the same training rows, the same validation rows and the same measure;
- write the rule before you look, for example "highest validation PR-AUC";
-
the test set is used once, for the chosen model only.
-
When close, prefer simpler. If two candidates are within the normal spread (Lesson 4), pick the faster, smaller or easier-to-explain one, and say so.
-
Budget: each extra try is a small chance of a lucky winner. Many tries on one validation set start to overfit it too.
📝A worked example
Input: a small grid for the kit's gradient boosting, scored on May (reference measurements, 2 November 2026):
| Learning rate | 7 leaves | 15 leaves | 31 leaves |
|---|---|---|---|
| 0.03 | 0.5786 | 0.5631 | 0.547 |
| 0.05 | 0.5664 | 0.5617 (kit) | 0.5317 |
| 0.1 | 0.5585 | 0.5524 | 0.5363 |
The best cell is 0.5786 (rate 0.03, 7 leaves), 0.017 above the kit's setting. A bootstrap estimate (ML4 Lesson 5) of May's score runs from about 0.485 to 0.638, so this difference is well inside the noise. You could choose either; the honest report gives the grid, the rule you used and the test result of the one you chose, once.
🌍Where it's used
- Product teams: a tuning budget is agreed (for example 50 tries) and recorded.
- AutoML tools: search many models and settings automatically; the same fairness rules apply.
- Research: papers report tuning budgets so others can compare fairly.
📝Case scenario
A team reported that their new model beat the old one by 2 points. The new model had been tuned with 200 tries; the old one with its defaults. Given the same tuning budget, the old model was within half a point. They changed their review checklist to require equal budgets.
🖱️Try it
Run the grid above with random_state=0, then write a three-line decision: which setting you choose, the rule you used, and why the
difference is or isn't meaningful. Success: your decision quotes May, not June.
⚠️Common mistakes
- Tuning on the test set. Sign: June used to pick settings. Fix: tune on May, test once on June.
- Unequal effort. Sign: the favourite model gets more tries. Fix: the same budget for each candidate.
- Chasing tiny gains. Sign: a 0.005 improvement drives the choice. Fix: compare with the spread from Lesson 4.
🛠️For your project
Project 1 records the validation scores used to choose (validation_pr_auc in metrics.json) and the choice made.
✅Quick check
What's the difference between a parameter and a hyperparameter?
Show the answer
Parameters are learned in training; hyperparameters are settings you choose before training.
When two models are within the noise, which should you choose?
Show the answer
The simpler, faster or easier-to-explain one, and say why.