My progress
Menu

AI / ML Engineer · Focus area 2: MLOps

ML9: MLOps: tracking, monitoring and retraining

Goal: treat models like software that changes: record every experiment, version data and models, watch a live model's inputs, outputs and results, detect drift, and retrain and roll back safely.

You'll need: ML1 to ML8, Module 71 (CI/CD), Module 72 (Production Operations), Python 3.11 or newer, and the DakDost practice kit.

Where this is used: a model that was good at launch gets worse as the world changes: a monsoon, a new city, a price change, a new app screen. MLOps is the set of habits and tools that notice this early and let a team fix it without guesswork.

How it differs: in ordinary software, code changes cause most problems. In ML, the data changes too, even when nobody touches the code. So monitoring has to watch the data and the model's behaviour, not only errors and speed, and releases include model versions as well as code versions.

Time: reading about 2 hours · practice (the Try it tasks) about 3 hours · assignment about 3 hours.

Lessons

  1. 1Experiment tracking and reproducibilityReading 15–25 minutes, plus practice✓
  2. 2Versioning data and modelsReading 15–25 minutes, plus practice✓
  3. 3Monitoring a live modelReading 15–25 minutes, plus practice✓
  4. 4Drift: when the world movesReading 15–25 minutes, plus practice✓
  5. 5Retraining, rollback and keeping the card currentReading 15–25 minutes, plus practice✓

🛠️Assignment: Keeping DakDost's model healthy

This is the second part of Project 2.

  1. Write dakdost/drift.py with PSI for every feature, five quantile bins for numbers, levels of 0.1 and 0.2, a JSON report and exit codes.
  2. Run it on July and August and record the results.
  3. Add dakdost_requests_total, dakdost_request_seconds and dakdost_flagged_total to the API's /metrics.
  4. Score July and August with the saved model and compare the share flagged and, using the outcomes in the files, precision and recall.
  5. Write a one-page monitoring and retraining plan: what is watched, the alert levels, who acts, the gate and the rollback.

Self-check before you finish:

✅Module check

Explain in a few sentences, in your own words:

How do you keep a model working well after launch?

Show a good answer

A good answer mentions: recording code, data, settings, seeds, versions and results for every run; versioned data and models, a registry and prediction logs with model versions so outcomes can be joined later; monitoring the service, inputs, outputs and results, with the share flagged as an early signal and alerts that have owners; PSI for input drift, bins checked on a normal month, and concept drift seen in results; and retraining on a policy, gated against the current model on the same recent data, released through shadow and canary with a way back, with the model card regenerated each time.

📚Official references

Official documentation and guidance for this module's topics, chosen on 2 November 2026. Tools, laws and standards change, so check the current version before you rely on a detail. If a link has moved, search the same site.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. A team's best model came from a notebook run last month. They can't reproduce it. What should they have recorded?

Why each option is right or wrong

  • “Only the final accuracy, so it can be compared with later runs” (Incorrect): A score alone can't be repeated.
  • “A screenshot of the notebook's last cell” (Incorrect): A screenshot doesn't capture the data, settings or order of cells.
  • “The names of the people who ran it” (Incorrect): Useful to know, but it doesn't make the run repeatable.
  • “Commit, data version, settings, seed and results” is correct.

    With all of these, anyone can rerun the same training and get the same model.

    • Tool: a tracking tool or one JSON record per run.
    • Also: train from a script.

📖 Revise: Lesson 1: Experiment tracking and reproducibility

Question ID sp-ai-ml-09-q1 · Report a mistake in this question
2. A customer complains about a flag from three weeks ago. The logs hold only the request and the probability. What's missing?

Why each option is right or wrong

  • “The customer's phone number in each log line” (Incorrect): That adds personal data without helping trace the model.
  • “A longer retention period for the logs” (Incorrect): The logs exist; they lack the key field.
  • “A screenshot of the app at the time” (Incorrect): That shows the screen, not which model answered.
  • “The model version that made the prediction” is correct.

    Without it, the team can't tell which model, data and settings were behind the decision.

    • Fix: log the version with every prediction.
    • Then: use the registry to find its training run.

📖 Revise: Lesson 2: Versioning data and models

Question ID sp-ai-ml-09-q2 · Report a mistake in this question
3. A credit model's API is fast and has no errors, but approvals fall from 40% to 15% overnight. Which layer of monitoring catches this

Why each option is right or wrong

  • “Service metrics such as latency and error rate” (Incorrect): Those are all normal here.
  • “Results monitoring, once repayments come in months later” (Incorrect): Too late to catch an overnight change.
  • “The model card's metrics section” (Incorrect): The card describes training-time results, not today's.
  • “Output monitoring, such as the share approved” is correct.

    A sudden change in outputs shows something changed today, before any outcomes are known.

    • Then: check inputs for drift or a broken feed.
    • Alert: on large jumps.

📖 Revise: Lesson 3: Monitoring a live model

Question ID sp-ai-ml-09-q3 · Report a mistake in this question
4. A feature's PSI is 0.31 between training and this month. What does that tell you?

Why each option is right or wrong

  • “The model's accuracy has fallen by 31%” (Incorrect): PSI measures a change in inputs, not accuracy.
  • “The feature must be removed from the model” (Incorrect): It's a reason to look, not an automatic removal.
  • “Nothing, because PSI only matters above 1.0” (Incorrect): On the usual rule of thumb, above 0.2 is already an alert.
  • “Its spread has shifted a lot; look into why” is correct.

    PSI above 0.2 is a common alert level. Check what changed, results by group and the share flagged.

    • Kit: August's address_quality was 0.269.
    • Ask: the business what changed.

📖 Revise: Lesson 4: Drift: when the world moves

Question ID sp-ai-ml-09-q4 · Report a mistake in this question
5. A shop's nightly retraining job trained on a day where half the data feed was missing, and the new model went live automatically. What

Why each option is right or wrong

  • “A faster training job” (Incorrect): Speed doesn't stop bad data being used.
  • “Retraining every hour instead of nightly” (Incorrect): More frequent retraining spreads the problem faster.
  • “Deleting the old models so that only the newest one remains in the registry” (Incorrect): That removes the way back.
  • “A gate: data checks and a match against the current model” is correct.

    Row counts and rates within expected ranges, and a match on a recent month, stop a bad model reaching users.

    • Plus: shadow before switching.
    • Keep: the previous version ready.

📖 Revise: Lesson 5: Retraining, rollback and keeping the card current

Question ID sp-ai-ml-09-q5 · Report a mistake in this question
6. A loan model's inputs look stable month to month, but defaults among approved customers have doubled. What kind of change is this most

Why each option is right or wrong

  • “Data drift in the applicants' inputs” (Incorrect): The inputs look stable, so this isn't mainly input drift.
  • “A bug in the PSI calculation” (Incorrect): Nothing points to the calculation.
  • “Training/serving skew in the API” (Incorrect): Skew is possible, but stable inputs and worse outcomes point to a change in the world.
  • “Concept drift: inputs now mean something else” is correct.

    The same inputs now lead to different outcomes, which shows only in results.

    • Watch: results monthly as outcomes arrive.
    • Then: retrain on recent outcomes.

📖 Revise: Lesson 4: Drift: when the world moves

Question ID sp-ai-ml-09-q6 · Report a mistake in this question
7. A data team edits last quarter's training file to fix some typos, keeping the same name. What problem does this cause?

Why each option is right or wrong

  • “The file becomes too large to store” (Incorrect): A few edits don't change the size much.
  • “Models trained on it will become more accurate automatically” (Incorrect): Old models don't change, but you can no longer tell what they were trained on.
  • “Nothing, as long as the fixes are correct” (Incorrect): Even correct fixes break the link between old models and their data.
  • “Old models can't be traced to their exact data” is correct.

    A dataset used for training should be frozen. Fixes go into a new version.

    • Tool: a content hash or DVC.
    • Record: the version with each run.

📖 Revise: Lesson 2: Versioning data and models

Question ID sp-ai-ml-09-q7 · Report a mistake in this question
8. A delivery model's flag share triples during the monsoon. The hub team can't make that many calls. What's a sensible immediate step?

Why each option is right or wrong

  • “Switch the model off until the monsoon ends” (Incorrect): That removes help when it's most needed.
  • “Raise the threshold to 0.99 so almost nothing is flagged” (Incorrect): That throws away most of the useful flags.
  • “Ask customers to check their own parcels” (Incorrect): That moves the work to customers without fixing the overload.
  • “Call the top-scoring parcels up to the team's capacity” is correct.

    Ranking still works, so focusing calls on the highest scores uses the team well while a retrained model is prepared.

    • Record: the temporary rule.
    • Then: retrain with monsoon outcomes.

📖 Revise: Lesson 5: Retraining, rollback and keeping the card current

Question ID sp-ai-ml-09-q8 · Report a mistake in this question
9. A drift check with 20 bins gives hub_load a PSI of 0.57 for a month the team knows was normal. The feature is one count per hub per

Why each option is right or wrong

  • “The hubs changed the way they count their parcels that month” (Incorrect): The month was normal; the counting didn't change.
  • “PSI can't be used for numeric features” (Incorrect): It's widely used for numbers, with bins.
  • “The model has concept drift” (Incorrect): Concept drift is about outcomes, not this input check.
  • “Clumped values make many bins noisy” is correct.

    With few distinct values per month, small shifts move whole groups between bins. Check bins on a normal month.

    • Kit: 5 bins give 0.022 for July.
    • Record: the bin choice.

📖 Revise: Lesson 4: Drift: when the world moves

Question ID sp-ai-ml-09-q9 · Report a mistake in this question
10. Which set of metrics best covers a live model's health?

Why each option is right or wrong

  • “CPU use and memory only” (Incorrect): Those show the server, not the model's answers.
  • “The training PR-AUC, checked once a year” (Incorrect): Training figures don't show what's happening now.
  • “The number of people who have used the app” (Incorrect): That's a product metric, not model health.
  • “Service, inputs, outputs and results” is correct.

    Each layer catches different problems: outages, drift, sudden output changes and falling accuracy.

    • Kit: /metrics, the drift report and monthly results.
    • Alert: where someone will act.

📖 Revise: Lesson 3: Monitoring a live model

Question ID sp-ai-ml-09-q10 · Report a mistake in this question

ProLearners Assist