ML9: MLOps: tracking, monitoring and retraining
Experiment tracking and reproducibility
🎯You'll be able to
Record each training run's code, data, settings, seed and results, so that any result can be found and repeated.
🏠The everyday picture
A good cook trying new recipes writes down each attempt: the amounts, the oven temperature, the time and how it tasted. Without notes, the best version is lost the moment they try another.
📘What it really is
- An experiment is one training run. Record for each:
- the code version (a git commit);
- the data version (a file hash or dataset version);
- the settings (hyperparameters, threshold rule) and the random seed;
- the library versions;
- the results (validation and, for the final model, test);
-
the files produced (model, metrics, card).
-
Tools: MLflow, Weights & Biases and similar tools store runs with a web page to compare them; a simple JSON file per run, like the kit's
metrics.json, plus git, already gets you far. MLflow isn't used in this kit (not verified here). -
Reproducible means another person, with the same code, data and versions, gets the same figures. Fixed seeds (ML1 Lesson 5) and pinned libraries (ML8 Lesson 3) make this possible.
-
Notebooks are good for exploring but easy to run out of order. Move the final training into a script (
python -m dakdost.train).
📝A worked example
Input: the reference solution, run twice.
python -m dakdost.train
md5sum raw/shipments.csv
Expected output (2 November 2026): both runs print test PR-AUC 0.6235 (baseline 0.0968), recall 0.5349, precision 0.6647, threshold
0.3, and the data's MD5 hash starts 422fcfa7ef9f. metrics.json records the model version (2026.07.1), the features, the split dates,
the validation scores used to choose (0.5617 against 0.5513), the threshold rule and all June figures. Adding the git commit and the hash
to metrics.json would complete the record.
🌍Where it's used
- Research and product teams: comparing hundreds of runs without losing track.
- Audits: showing exactly how a deployed model was made.
- Handover: a new team member can repeat last year's model.
📝Case scenario
A team's best fraud model came from "the run on Thursday". Nobody could repeat it: the notebook had been edited, the data file overwritten and the seed not set. They retrained, got a worse result, and spent two weeks trying to recover the old one. From then on, every run wrote its commit, data hash, settings and results to a tracking table.
🖱️Try it
Add two fields to metrics.json in your Project 1 solution: data_md5 (the hash of raw/shipments.csv) and git_commit (from
git rev-parse HEAD, or "not in git"). Success: two runs give identical files apart from any timestamp.
⚠️Common mistakes
- Results without inputs. Sign: a score nobody can repeat. Fix: record code, data, settings, seed and versions with it.
- Final models from notebooks. Sign: cells run out of order. Fix: a script for training.
- Overwriting runs. Sign: only the latest result exists. Fix: one record per run.
🛠️For your project
Project 1's test 9 checks that training twice gives the same PR-AUC.
✅Quick check
What must be recorded to repeat a training run?
Show the answer
The code version, data version, settings, seed, library versions and results.
Why move final training out of a notebook?
Show the answer
Notebook cells can be run out of order, so the result may not be repeatable.