ML1: Thinking in machine learning
The workflow: train, validate, test, and keep records
🎯You'll be able to
Describe the steps of an ML project, explain why data is split three ways, and keep a record that lets someone else repeat your result.
🏠The everyday picture
Students practise with past papers (training), take a mock exam to see where they stand and adjust (validation), and then sit the real exam once (test). If they saw the real exam paper while practising, their result would say nothing about what they know.
📘What it really is
-
The steps: frame the problem → collect and check data → split it → baseline → train candidates → choose on validation → test once → write it up → serve → monitor. Expect to loop back: most time goes into data and checking, not training.
-
Three sets:
- training set: the model learns from it;
- validation set: you compare models and settings, and choose a threshold, on it;
-
test set: used once at the end, to report how the chosen model does on data it has never influenced.
-
Why three: each time you choose using a set, your choice is fitted a little to that set. The test set stays untouched so its result is an honest estimate.
-
In DakDost: training is January to April (8,883 parcels), validation is May (2,306) and test is June (2,221). Splitting by date matches how the model is used: trained on the past, used on the future (ML2 Lesson 1).
-
Records: for every result, save the data version, the code version, the settings, the random seed and the figures. The kit writes
models/metrics.jsonandmodels/model_card.md; ML9 adds experiment tracking.
📝A worked example
Input: in the kit, run:
python -m dakdost.train
Expected output (reference solution, measured on 2 November 2026): test PR-AUC 0.6235 (baseline 0.0968), recall 0.5349, precision
0.6647, threshold 0.3. Open models/metrics.json: it records the date ranges of the three sets, the validation scores used to choose
the model (gradient boosting 0.5617 against logistic regression 0.5513), the threshold rule and the June figures. Run it again: the
figures are identical, because the seed is fixed. In the starter, the figures change on every run and look far better, for reasons the
projects uncover.
🌍Where it's used
- Every ML team: the train/validation/test discipline is the basic quality rule.
- Competitions: organisers keep a hidden test set so nobody can tune to it.
- Regulated work: banks and health providers keep records so an auditor can repeat a result.
📝Case scenario
A team tried 40 settings and reported the best test score in a presentation. When the model went live, it did noticeably worse. They had used the test set as a validation set 40 times, so the best result was partly luck. The next version kept a test month aside and touched it once.
🖱️Try it
Open models/metrics.json from the reference run figures above (or your own run once Project 1 works) and write three sentences: what
each set covers, what the validation set was used for, and the one June figure you would quote. Success: you quote a June figure, not a
May one, and you name the threshold rule.
⚠️Common mistakes
- Choosing on the test set. Sign: the test set was used more than once. Fix: choose on validation; test once at the end.
- No seed. Sign: the same code gives different figures. Fix: set
random_state(or the equivalent) everywhere it applies. - No record of the data. Sign: nobody can say which file a model was trained on. Fix: save the data version with the figures.
🛠️For your project
Project 1's tests check the split dates, the threshold chosen on May, figures that match a recomputation, and reproducible training.
✅Quick check
What is the validation set for?
Show the answer
Comparing models and settings, and choosing a threshold, without touching the test set.
Why use the test set only once?
Show the answer
Each use fits your choices a little to it, so its result stops being an honest estimate.