ML5: Neural networks and deep learning
How networks learn: loss, gradients and optimisers
🎯You'll be able to
Explain a loss function, gradient descent, a learning rate, batches and epochs.
🏠The everyday picture
Finding the lowest point of a valley in fog: you feel which way the ground slopes under your feet and take a step downhill, again and again. Step too far and you overshoot; step too little and you never arrive.
📘What it really is
-
The loss measures how wrong the predictions are. For yes/no targets, binary cross-entropy punishes confident wrong answers heavily. Training means making the loss smaller.
-
The gradient says, for each parameter, which way and how strongly the loss changes if you nudge it. Backpropagation computes all the gradients at once by working backwards through the layers; frameworks do this automatically.
-
Gradient descent moves each parameter a little against its gradient. The learning rate sets the step size. Too high and the loss jumps around; too low and training crawls.
-
Batches and epochs: instead of all rows at once, take a batch (say 256 rows), compute the loss and update. One pass over all training rows is an epoch.
-
Optimisers improve on plain steps. Adam adapts the step size for each parameter and is a common default (learning rate 0.001 is a usual start).
-
Numbers to watch: the training loss going down, and the validation score (Lesson 4).
📝A worked example
Input: a single parameter w, with loss (w − 3)². The slope of the loss is 2 × (w − 3).
Steps with learning rate 0.1, starting at w = 0:
- Slope at 0 is −6. New
w= 0 − 0.1 × (−6) = 0.6. - Slope at 0.6 is −4.8. New
w= 0.6 + 0.48 = 1.08. - Slope at 1.08 is −3.84. New
w= 1.08 + 0.384 = 1.464.
Expected output: w moves towards 3, with smaller steps as the slope flattens. With learning rate 1.1 instead, the first step goes to 6.6,
the next to −1.32, then 8.18, and the steps grow: the learning rate is too high and training diverges. A network does the same thing for every one of
its parameters at once.
🌍Where it's used
- Every neural network: from a 1,249-parameter model to large language models, training is gradient descent on a loss.
- Other models: logistic regression and gradient boosting also follow gradients, in their own ways.
- Debugging training: most "it won't learn" problems are the learning rate, the loss or the data.
📝Case scenario
A team's image model's loss went to "nan" (not a number) after a few hundred steps. The learning rate was too high for their setup, and the numbers had grown until they overflowed. A lower rate and inputs scaled to 0–1 fixed it. They added a check that stops training when the loss stops being a number.
🖱️Try it
Repeat the worked example with learning rate 0.5. Success: the first step lands exactly on 3 (0 − 0.5 × (−6) = 3), which shows why the learning rate matters so much.
⚠️Common mistakes
- Learning rate too high. Sign: the loss jumps or becomes "nan". Fix: lower it, by a factor of 3 to 10 at a time.
- Unscaled inputs. Sign: slow or unstable training. Fix: scale inputs, as for logistic regression.
- Watching only the training loss. Sign: it keeps falling while new-data results get worse. Fix: watch validation too (Lesson 4).
🛠️For your project
Not needed for the projects; it explains what "training" means in the network you compare in the assignment.
✅Quick check
What does the learning rate control?
Show the answer
How big each step is when the parameters are updated.
What's an epoch?
Show the answer
One full pass over all the training rows.