My progress
Menu

AI / ML Engineer · Focus area 2: MLOps

ML8: Serving models

Goal: put a trained model behind a dependable API: choose batch or online predictions, validate every input, keep training and serving consistent, meet a speed target, and package the model with its versions.

You'll need: ML1 to ML7, Module 25 (APIs and JSON), Module 47 (REST API Design), Module 69 (Docker), Python 3.11 or newer with FastAPI (tested with 0.162.0), and the DakDost practice kit.

Where this is used: a model only helps when other software can use it: a booking screen asking "is this parcel likely to be late?", a nightly job scoring all of tomorrow's parcels, an app showing a delivery estimate. Serving is where data science meets backend engineering.

How it differs: the Backend path builds APIs whose rules are written by people. A model API has extra failure modes: inputs the model has never seen, features computed differently from training, answers that are probabilities, and a model file that has its own version.

Time: reading about 2 hours · practice (the Try it tasks) about 3 hours · assignment about 3 hours.

Lessons

  1. 1Batch or online predictionsReading 15–25 minutes, plus practice✓
  2. 2An API with validation, using FastAPIReading 15–25 minutes, plus practice✓
  3. 3Training/serving skew and versionsReading 15–25 minutes, plus practice✓
  4. 4Speed, batching and limitsReading 15–25 minutes, plus practice✓
  5. 5Packaging and deploying a model serviceReading 15–25 minutes, plus practice✓

🛠️Assignment: DakDost's late-parcel API

This is the main part of Project 2.

  1. Write a pydantic Shipment model with types, allowed categories, ranges and extra='forbid', and use it for /predict.
  2. Return probability, decision, threshold, model version and a request id; log the same without any personal data.
  3. Use the shared build_frame and saved pipeline, and add a test that compares the API with the offline model at four booking times.
  4. Add /predict/batch with a limit of 100 and a clear 413 above it.
  5. Measure p50 and p95 for 200 single calls and one batch of 100, and record where you measured.

Self-check before you finish:

✅Module check

Explain in a few sentences, in your own words:

How do you serve a model so that its answers can be trusted in use?

Show a good answer

A good answer mentions: choosing batch or online from when inputs are known and answers are needed, with a fallback; a schema that rejects missing, unknown, impossible and extra inputs with clear errors; responses with probability, decision, threshold, version and a request id; one shared feature function and a saved pipeline to avoid training/serving skew, checked by comparing the API with the offline model; pinned library versions; latency reported as percentiles, batching for many rows and size limits; and packaging with versioned model files chosen by a setting, health checks, and shadow or canary releases with a way back.

📚Official references

Official documentation and guidance for this module's topics, chosen on 2 November 2026. Tools, laws and standards change, so check the current version before you rely on a detail. If a link has moved, search the same site.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. A property website wants price estimates shown on each listing page. Listings change about once a day. Which serving approach fits best?

Why each option is right or wrong

  • “An online API called every time anyone views a page” (Incorrect): It works, but repeats the same calculation thousands of times for no benefit.
  • “A streaming system that scores every page view as an event” (Incorrect): Page views don't change the estimate; streaming adds cost for nothing.
  • “Ask an analyst to estimate prices by hand once a month” (Incorrect): That's slow, inconsistent and out of date.
  • “Score listings in a nightly batch and store the estimates” is correct.

    Inputs change daily and the page needs an instant answer, so precomputed estimates are simplest and cheapest.

    • Refresh: when a listing is edited, re-score it.
    • Fallback: hide the estimate if it's missing.

📖 Revise: Lesson 1: Batch or online predictions

Question ID sp-ai-ml-08-q1 · Report a mistake in this question
2. A model API accepts {"age": -5, "city": "Atlantis"} and returns a confident probability. What's the missing piece?

Why each option is right or wrong

  • “A bigger model that understands unusual inputs” (Incorrect): Any model will score nonsense; it can't tell it's nonsense.
  • “A lower threshold for unusual inputs” (Incorrect): The threshold doesn't detect bad inputs.
  • “A retry when the probability is very high” (Incorrect): Retrying the same input gives the same answer.
  • “Input validation with ranges and allowed values” is correct.

    A schema with limits rejects nonsense values with a clear 422 before the model sees them.

    • Tool: pydantic Field and Literal.
    • Also: forbid extra fields.

📖 Revise: Lesson 2: An API with validation, using FastAPI

Question ID sp-ai-ml-08-q2 · Report a mistake in this question
3. A recommendation model was trained on prices in rupees. The new app sends prices in paise. Nothing errors, but suggestions are odd.

Why each option is right or wrong

  • “Overfitting to the training data” (Incorrect): The model is fine; its inputs changed.
  • “Data drift caused by customers' changing tastes” (Incorrect): Drift is a real change in the world; this is a change in preparation.
  • “A cold start for new users” (Incorrect): It affects every user, not only new ones.
  • “Training/serving skew from a changed unit” is correct.

    The model gets values 100 times larger than it learned from. A schema with named units and a comparison test catch it.

    • Name fields: price_rupees.
    • Check: ranges from training.

📖 Revise: Lesson 3: Training/serving skew and versions

Question ID sp-ai-ml-08-q3 · Report a mistake in this question
4. An API's average response time is 60 ms, but users complain it "sometimes hangs". What should you measure?

Why each option is right or wrong

  • “The average again, measured over a longer period of time” (Incorrect): The average hides the slow requests.
  • “The model's PR-AUC on the latest month” (Incorrect): Accuracy isn't about speed.
  • “The size of the model file on disk” (Incorrect): Size matters at start-up, not usually per request.
  • “p95 and p99, and what slow requests share” is correct.

    Percentiles show the slow tail users notice. Then find the cause: large requests, cold starts or a slow dependency.

    • Kit: p50 36.8 ms, p95 45.3 ms in-process.
    • Fix ideas: limits, batching, warm-up.

📖 Revise: Lesson 4: Speed, batching and limits

Question ID sp-ai-ml-08-q4 · Report a mistake in this question
5. A team deploys a new model by copying it over the old file on the server. The new model misbehaves. What practice would have helped

Why each option is right or wrong

  • “Testing the new model on the training data before copying” (Incorrect): Training data doesn't show how it behaves on new requests.
  • “Deleting the logs so the problem isn't visible” (Incorrect): That hides the problem and loses evidence.
  • “Copying the file at night when traffic is low” (Incorrect): Timing reduces impact but leaves no way back.
  • “Versioned files picked by a setting, old one kept” is correct.

    Switching back is then a one-line change, and you know which version served each request.

    • Plus: shadow or canary first.
    • Report: version in /health.

📖 Revise: Lesson 5: Packaging and deploying a model service

Question ID sp-ai-ml-08-q5 · Report a mistake in this question
6. A partner sends 5,000 rows in one request to a model API, and other users' requests slow down. What's the best change?

Why each option is right or wrong

  • “Make the server bigger so it can handle any request” (Incorrect): It moves the limit without setting one.
  • “Remove batch requests and accept only single rows” (Incorrect): 5,000 single calls would be even slower overall.
  • “Ask the partner politely to send fewer rows” (Incorrect): Without a limit in code, the next caller may do the same.
  • “Cap the batch size and give large jobs their own route” is correct.

    A clear limit protects other users. Bulk jobs can use a batch endpoint or a scheduled job.

    • Kit: 1 to 100 per request, 413 above.
    • Plus: per-caller rate limits.

📖 Revise: Lesson 4: Speed, batching and limits

Question ID sp-ai-ml-08-q6 · Report a mistake in this question
7. A model was saved with joblib using scikit-learn 1.9. The server installs the newest scikit-learn without a version. What's the risk?

Why each option is right or wrong

  • “None; joblib files work with any version” (Incorrect): Saved models depend on the library's internal classes, which can change.
  • “The server will run faster with the new version” (Incorrect): Speed isn't the risk; loading and behaviour are.
  • “The model will retrain itself on the new version” (Incorrect): Loading doesn't retrain anything.
  • “The model may fail to load or behave differently” is correct.

    A saved model depends on the library version it was saved with. Pin versions and retrain to upgrade.

    • Kit: requirements.txt pins scikit-learn 1.9.1.
    • Record: library versions with each model.

📖 Revise: Lesson 3: Training/serving skew and versions

Question ID sp-ai-ml-08-q7 · Report a mistake in this question
8. Which response is most useful from a model API to the app calling it?

Why each option is right or wrong

  • “{"late": true}” (Incorrect): The caller can't tell how sure it is, which model answered or how to trace it.
  • “The full list of model parameters” (Incorrect): That's large, meaningless to the caller and may reveal the model.
  • “The rows of training data that look most like this request” (Incorrect): That may expose other people's data.
  • “Probability, decision, threshold, version, request id” is correct.

    The caller can show or log the decision, trace a complaint and tell which model made it.

    • Log: the same, without personal data.
    • Kit: test 2 checks these fields.

📖 Revise: Lesson 2: An API with validation, using FastAPI

Question ID sp-ai-ml-08-q8 · Report a mistake in this question
9. A team wants to compare a new model with the current one on real traffic without affecting users. Which release approach fits?

Why each option is right or wrong

  • “Switch all traffic to the new model and watch complaints” (Incorrect): Users would be affected immediately.
  • “Test the new model only on last year's data” (Incorrect): That doesn't show behaviour on current traffic.
  • “Ask users to choose which model they prefer” (Incorrect): Users can't judge model quality from a single prediction.
  • “A shadow release, scoring the same requests silently” is correct.

    Both models score the same requests; only the current one's answers are used. Compare flag rates now and accuracy once outcomes arrive.

    • Then: a small canary.
    • Keep: the old version ready.

📖 Revise: Lesson 5: Packaging and deploying a model service

Question ID sp-ai-ml-08-q9 · Report a mistake in this question
10. A delivery-time model is called from a checkout page. It sometimes takes five seconds when the model server restarts. What should the

Why each option is right or wrong

  • “Wait as long as it takes, so the estimate is shown” (Incorrect): Customers leave when checkout hangs.
  • “Show an error and block the checkout” (Incorrect): Losing the sale is worse than a missing estimate.
  • “Call the model five times in parallel and use the fastest” (Incorrect): That multiplies load on a struggling server.
  • “Use a time limit and a simple fallback estimate” is correct.

    Decide in advance: after a short wait, show a rule-based estimate (or none) and log it.

    • Example: the usual days for the zone.
    • Track: how often the fallback is used.

📖 Revise: Lesson 1: Batch or online predictions

Question ID sp-ai-ml-08-q10 · Report a mistake in this question

ProLearners Assist