ML3: Classic models with scikit-learn
Ensembles: random forests and gradient boosting
🎯You'll be able to
Explain how random forests and gradient boosting combine many trees, and train both.
🏠The everyday picture
Asking one friend for a restaurant tip is hit and miss. Asking twenty friends and going with the most common answer is more reliable. That's a random forest. Gradient boosting is more like a teacher marking drafts: each new draft fixes what the last one got wrong.
📘What it really is
-
A random forest trains many deep trees, each on a random sample of rows and with a random subset of inputs at each split, and averages their predictions. The randomness makes the trees different, so their mistakes partly cancel out.
-
Gradient boosting trains small trees one after another. Each new tree focuses on the errors the previous trees still make. It usually scores best on tables but has more settings to tune.
-
scikit-learn's
HistGradientBoostingClassifiersorts numbers into bins first, which makes it fast on thousands to millions of rows. Its main settings:learning_rate(how big each correction is),max_iter(how many trees),max_leaf_nodes(how big each tree is) andl2_regularization. -
Other libraries: XGBoost, LightGBM and CatBoost are popular gradient boosting libraries with the same idea. They aren't needed for this path.
-
Trade-offs: ensembles are harder to explain than one tree or a logistic regression; ML7 shows ways to explain them.
📝A worked example
Input: three candidates trained on January to April, scored on May (reference measurements, 2 November 2026):
| Model | Settings | Training PR-AUC | Validation PR-AUC |
|---|---|---|---|
| Random forest | 300 trees, min_samples_leaf=5 |
0.8695 | 0.5215 |
| Logistic regression | scaled, default C |
— | 0.5513 |
| Gradient boosting | 200 trees, rate 0.05, 15 leaves, L2 1.0 | 0.8236 | 0.5617 |
Gradient boosting is best on May, so the kit chooses it. Every model scores much higher on training rows than on May; Lesson 4 is about that gap.
🌍Where it's used
- Tables everywhere: credit, insurance pricing, delivery time estimates and demand forecasting often use gradient boosting.
- Search ranking: boosted trees rank results in many search systems.
- Quick strong baselines: a random forest with default settings is a reasonable first try on a new table.
📝Case scenario
A team ran a gradient boosting model with a high learning rate and 2,000 trees. It looked good on training data and poor on new data. With a lower rate, fewer leaves and early stopping on validation data, it scored a little lower in training and better on new data.
🖱️Try it
Train a random forest with n_estimators=50 and then 300, both with random_state=0. Compare May's PR-AUC and the time each takes.
Success: you can say whether more trees helped enough to be worth the extra time.
⚠️Common mistakes
- Assuming the fanciest model wins. Sign: no comparison on validation. Fix: compare candidates fairly (Lesson 5).
- Ignoring run time and size. Sign: a 300 MB forest is too slow for the API. Fix: weigh speed and size against small gains.
- No seed. Sign: forests change between runs. Fix:
random_state.
🛠️For your project
Project 1 trains logistic regression and gradient boosting, records both on May and June, and keeps the one chosen on May.
✅Quick check
How does a random forest reduce the mistakes of single trees?
Show the answer
It averages many different trees, so their mistakes partly cancel out.
What does each new tree do in gradient boosting?
Show the answer
It focuses on the errors the trees before it still make.