ML5: Neural networks and deep learning
From a neuron to a network
🎯You'll be able to
Describe a neuron, a layer and an activation function, and count the parameters in a small network.
🏠The everyday picture
A factory line where each station takes the parts from the station before, combines them and passes on something more finished. Early stations make simple shapes; later ones assemble them into a product. A neural network passes numbers through layers in the same way.
📘What it really is
-
A neuron takes inputs, multiplies each by a weight, adds them up with a bias, and passes the result through an activation function. Without the activation it would be logistic regression's weighted sum (ML3 Lesson 1).
-
Activation functions add bends, so the network can learn curves and combinations. ReLU (
max(0, x)) is the common default in hidden layers. Sigmoid squeezes the final output to a probability for a yes/no target. -
A layer is many neurons side by side, each with its own weights. A dense (fully connected) layer links every input to every neuron.
- A network stacks layers: input → hidden layers → output. "Deep" just means several hidden layers.
- Parameters are the weights and biases. A dense layer with
ninputs andmneurons hasn × m + mof them. - Special shapes suit special data: convolutional networks for images (they look at small patches), transformers for text and sequences (they let each word look at every other word; ML6).
📝A worked example
Input: DakDost's 21 prepared inputs (14 one-hot columns and 7 scaled numbers, ML2 Lesson 3), and a small network: 21 → 32 → 16 → 1.
Steps:
- First layer: 21 × 32 weights + 32 biases = 704.
- Second layer: 32 × 16 + 16 = 528.
- Output: 16 × 1 + 1 = 17.
Expected output: 704 + 528 + 17 = 1,249 parameters. PyTorch confirms it with
sum(p.numel() for p in model.parameters()), which printed 1249 with torch 2.15.0. The kit's gradient boosting model has far more
numbers in its 200 trees, which is a reminder that "neural" doesn't mean "bigger".
🌍Where it's used
- Phones: face unlock and photo search use convolutional networks.
- Speech: voice assistants turn sound into text with networks trained on many hours of speech.
- Recommendations: networks learn from what people watched or bought.
📝Case scenario
A team added three more hidden layers to a network for credit risk, expecting a better score. It trained more slowly and scored the same on validation. The data was a table of 20 columns, where boosted trees were already as good; the extra layers added nothing but cost.
🖱️Try it
Count the parameters of a network 21 → 64 → 1 by hand, then check it with code if you have PyTorch. Success: 21 × 64 + 64 + 64 + 1 = 1,473.
⚠️Common mistakes
- No activation between layers. Sign: a deep network behaves like logistic regression. Fix: put ReLU (or similar) after hidden layers.
- Assuming more layers is better. Sign: slower training, no gain. Fix: compare on validation with smaller networks and with trees.
- Forgetting the output activation's job. Sign: outputs outside 0 to 1 used as probabilities. Fix: sigmoid for yes/no (or use a loss that includes it, Lesson 2).
🛠️For your project
The projects don't need a network; the assignment asks you to compare one with the kit's model and decide.
✅Quick check
What does an activation function add?
Show the answer
Bends, so the network can learn curves and combinations instead of only weighted sums.
How many parameters has a dense layer with 10 inputs and 5 neurons?
Show the answer
10 × 5 + 5 = 55.