ML9: MLOps: tracking, monitoring and retraining
Monitoring a live model
🎯You'll be able to
Choose what to watch for a live model (service health, inputs, outputs and results) and expose it as metrics.
🏠The everyday picture
A doctor checks a patient's temperature and pulse every day, not only once a year. Monitoring checks the model's vital signs regularly, so problems show up before people complain.
📘What it really is
- Four layers to watch:
- service: requests, errors, latency (p50, p95), as for any API;
- inputs: the spread of each feature compared with training (Lesson 4), and the share of rejected requests;
- outputs: the share flagged and the average probability, by hub or zone;
-
results: once outcomes arrive, precision, recall and PR-AUC on recent weeks.
-
Metrics: the kit's API exposes counters and a latency histogram at
/metricsin Prometheus format:dakdost_requests_totalby endpoint and status,dakdost_request_secondsanddakdost_flagged_total. A tool such as Prometheus collects them and Grafana draws dashboards (the Cloud / DevOps path). -
Alerts: alert on what someone will act on: the service failing, the flag share jumping, a feature's drift crossing a level, recall falling below the agreed minimum.
-
Outputs are early signals. Results need outcomes, which arrive days later; a sudden change in the share flagged shows something changed today.
📝A worked example
Input: the reference API after one batch of 100 parcels, 100 single calls and one batch that was too large (2 November 2026).
dakdost_requests_total{endpoint="batch",status="200"} 1.0
dakdost_requests_total{endpoint="predict",status="200"} 100.0
dakdost_requests_total{endpoint="batch",status="413"} 1.0
Then the outputs for two later months, scored with the saved model:
| Month | Parcels | Share flagged | Average probability | Real late rate |
|---|---|---|---|---|
| July (normal) | 2,398 | 9.0% (217) | 0.0808 | 8.88% |
| August (monsoon) | 2,393 | 28.0% (670) | 0.2145 | 26.24% |
In August, the share flagged triples. The hub team can't make three times as many calls, so this needs a decision on the first day it shows, long before August's outcomes are known.
🌍Where it's used
- Any live model: service metrics plus model-specific ones.
- Operations teams: dashboards by region or hub.
- Product reviews: monthly results against the targets in the model card.
📝Case scenario
A recommendation model's service metrics were all green, but sales from recommendations fell by a third. An upstream change had started sending empty product categories; the model still answered quickly, just badly. An input check on the share of missing categories, added afterwards, would have alerted on the first day.
🖱️Try it
Call the solution API five times and read /metrics. Success: you find the request counter for predict and see it increase by five, and
you find the latency histogram's buckets.
⚠️Common mistakes
- Only service metrics. Sign: a fast, error-free model giving bad answers. Fix: watch inputs, outputs and results too.
- Waiting for outcomes. Sign: problems found weeks late. Fix: watch outputs as an early signal.
- Alerts nobody acts on. Sign: alerts ignored. Fix: alert only on what has an owner and an action.
🛠️For your project
Project 2's test 7 checks that /metrics counts requests and records time.
✅Quick check
Why watch the share flagged?
Show the answer
It changes as soon as something changes, long before outcomes arrive.
Name the four layers of monitoring.
Show the answer
Service, inputs, outputs and results.