My progress
Menu

Cloud / DevOps Engineer · Focus area 3: CI/CD and reliability

CD9: Monitoring, alerts and scaling

Goal: measure what users experience, set objectives for it, collect metrics with Prometheus, show them on Grafana dashboards, write alerts that fire for real problems and say what to do, and scale the service up and down with load without failing requests.

You'll need: CD2, CD3, CD8, Module 72 (Production Operations), Module 95 (Scaling), Module 96 (Reliability Engineering), and the RouteRani kit.

Where this is used: during an exam-results release, a cricket final or a festival sale, teams watch dashboards of requests, errors and response times, get paged when something breaks, and rely on autoscaling to add capacity. Site Reliability Engineering (SRE) teams at large companies are built around these practices.

How it differs: an app developer checks whether their feature works. A DevOps engineer checks whether the whole service works for users right now, finds out before users complain, and makes sure the right person is woken up for the right reasons, and nobody for the wrong ones.

Time: reading about 2½ hours · practice (the Try it tasks) about 3 hours · assignment about 4 hours.

Lessons

  1. 1What to measure, and how good is good enoughReading 15–25 minutes, plus practice✓
  2. 2Prometheus and PromQLReading 15–25 minutes, plus practice✓
  3. 3Dashboards that answer questionsReading 15–25 minutes, plus practice✓
  4. 4Alerts people trustReading 15–25 minutes, plus practice✓
  5. 5Scaling with loadReading 15–25 minutes, plus practice✓

🛠️Assignment: See it, hear it, scale it

This is the core of Project 3.

  1. Make Prometheus find every API copy with service discovery, and add Grafana with a provisioned data source and dashboard.
  2. Build the dashboard: request rate, failed share, p95, copies answering, version running, with units.
  3. Rewrite the alert rules (error share with rate(), p95 from the histogram, a stopped copy), each with for, severity, summary and runbook, and write alerts.test.yml with cases that fire and cases that don't.
  4. Fix tools/autoscale.js: rounding up, minimum and maximum, a stabilisation window for scaling down, and no change when Prometheus can't be read.
  5. Record a load test with scaling in docs/observe.md: copies over time, failures, and the dashboard during the test.

Self-check before you finish:

✅Module check

Explain in a few sentences, in your own words:

How do you know a service is healthy, find out quickly when it isn't, and keep it healthy as load changes?

Show a good answer

A good answer mentions: measuring rate, errors and duration (with percentiles) as SLIs, with SLOs and an error budget; Prometheus scraping every copy through service discovery, and PromQL with rate(), ratios and histogram_quantile; low-cardinality labels; a provisioned Grafana dashboard that answers the main questions at a glance; alerts on symptoms with for durations, severities, summaries and runbooks, tested with promtool and routed by Alertmanager; and horizontal scaling on a matching signal with the ceil formula, bounds, a stabilisation window, no action on missing data, and a load test to prove it.

📚Official references

Official documentation and guidance for this module's topics, chosen on 2 November 2026. Tools, laws and standards change, so check the current version before you rely on a detail. If a link has moved, search the same site.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. A dashboard shows an average response time of 140 ms, yet users complain the booking page is slow at peak times. What should the team look

Why each option is right or wrong

  • “The server's CPU temperature during the busy hours” (Incorrect): Temperature is a cause signal, far from what users feel.
  • “The total number of requests since the server started” (Incorrect): A running total says nothing about speed.
  • “The average again, measured over a longer period” (Incorrect): A longer average smooths the slow peaks even more.
  • “The p95 and p99 times for booking requests” is correct.

    Averages hide slow tails. A small share of requests can take seconds while the average looks fine.

    • Query: histogram_quantile(0.99, ...).
    • Then: an SLO on the percentile.

📖 Revise: Lesson 1: What to measure, and how good is good enough

Question ID sp-cloud-devops-09-q1 · Report a mistake in this question
2. A team sets an SLO of 99.9% successful requests over 30 days. Halfway through the month, they've had 40 minutes of failures. What does the

Why each option is right or wrong

  • “Nothing, because SLOs are only reviewed at the end of the month” (Incorrect): The budget is a live guide, not a monthly report.
  • “Raise the SLO to 99.99% so the team tries harder” (Incorrect): A stricter target doesn't undo the failures, and is harder still to meet.
  • “Release faster to make up for the lost time” (Incorrect): More change means more risk, with almost no budget left.
  • “Most of the budget is gone: slow down and fix reliability” is correct.

    About 43 minutes of failure are allowed in 30 days. With 40 used, more risk would probably break the objective.

    • Typical response: focus on reliability work.
    • Next month: the budget resets.

📖 Revise: Lesson 1: What to measure, and how good is good enough

Question ID sp-cloud-devops-09-q2 · Report a mistake in this question
3. An alert rule says http_requests_total{code=~"5.."} > 100. After a few days it fires constantly, even when nothing is wrong. Why?

Why each option is right or wrong

  • “Prometheus double-counts errors after each scrape” (Incorrect): Counters aren't double-counted.
  • “The rule needs severity: page to work properly” (Incorrect): Labels affect routing, not when the rule fires.
  • “Errors are counted per copy, so three copies triple the number” (Incorrect): That makes it worse, but the core problem is the raw counter.
  • “A counter only rises, so once past 100 it stays above it” is correct.

    The total of all errors since start-up keeps growing. The rule needs a rate, compared with all requests.

    • Fix: sum(rate(... 5..[1m])) / sum(rate(...[1m])) > 0.05.
    • Test: promtool cases that fire and stay quiet.

📖 Revise: Lesson 4: Alerts people trust

Question ID sp-cloud-devops-09-q3 · Report a mistake in this question
4. Three RouteRani copies run, but Prometheus shows only one target. Its configuration has static_configs: targets: ["app:3000"]. What's the

Why each option is right or wrong

  • “List app:3000 three times in the targets” (Incorrect): The same name three times still gives one target.
  • “Scrape the load balancer instead, which sees all copies” (Incorrect): The load balancer forwards each scrape to one copy, so results jump between them.
  • “Restart Prometheus after each scaling change” (Incorrect): Restarts lose nothing here, but still scrape just one address.
  • “Use dns_sd_configs so every copy is found” is correct.

    A static target is one address. DNS service discovery asks for every address behind app and scrapes each copy.

    • Check: count(up{job="routerani"} == 1) shows 3.
    • Test: Project 3 test 3.

📖 Revise: Lesson 2: Prometheus and PromQL

Question ID sp-cloud-devops-09-q4 · Report a mistake in this question
5. A developer adds a user_id label to the request counter. A week later, Prometheus is using all its memory. What happened?

Why each option is right or wrong

  • “User IDs are too long, so each sample takes more space” (Incorrect): Label length isn't the main cost.
  • “Counters with labels can't be compressed by Prometheus” (Incorrect): Labelled counters compress normally.
  • “Prometheus keeps user data forever for privacy reasons” (Incorrect): Retention isn't tied to label content.
  • “Every user created a new series, so series grew into millions” is correct.

    Each label combination is its own series. Unbounded values such as IDs multiply them beyond what Prometheus can hold.

    • Fix: remove the label; use logs or traces for per-user detail.
    • Protect: limits on series per scrape.

📖 Revise: Lesson 2: Prometheus and PromQL

Question ID sp-cloud-devops-09-q5 · Report a mistake in this question
6. During an incident, engineers open eleven dashboards and argue about whether errors are rising. What's the best long-term improvement?

Why each option is right or wrong

  • “More dashboards, one for each engineer's favourite view” (Incorrect): More views make agreement harder.
  • “Remove all dashboards and use logs only” (Incorrect): Logs are good for detail but slow for "is it healthy right now?".
  • “Dashboards with every metric the service exposes” (Incorrect): Too many panels hide the few that matter.
  • “One provisioned health dashboard per service” is correct.

    A shared dashboard with the main questions at the top gives everyone the same picture fast, and lives in Git.

    • Link from: runbooks and alerts.
    • Detail: further down or on other dashboards.

📖 Revise: Lesson 3: Dashboards that answer questions

Question ID sp-cloud-devops-09-q6 · Report a mistake in this question
7. An on-call engineer gets about 300 alerts a night, mostly "CPU above 70%" on healthy servers. A real outage was noticed by customers first.

Why each option is right or wrong

  • “Raise the CPU threshold to 90% and keep the rest” (Incorrect): Fewer alerts, but still about a cause, not what users feel.
  • “Hire a second on-call person to share the 300 alerts” (Incorrect): Two tired people still miss the real one.
  • “Turn off alerts at night and check dashboards each morning” (Incorrect): Outages at night would go unnoticed for hours.
  • “Alert on user symptoms, with runbooks, and drop noise” is correct.

    Pages should mean users are affected. Symptom alerts, each with first steps, catch real outages without burying them in noise.

    • Keep CPU: on a dashboard, or as a ticket.
    • Review: alerts regularly.

📖 Revise: Lesson 4: Alerts people trust

Question ID sp-cloud-devops-09-q7 · Report a mistake in this question
8. An autoscaler targets 150 requests per second per copy, with a minimum of 2 and a maximum of 6. Traffic is 1,000 requests per second. How

Why each option is right or wrong

  • “7, from ceil(1,000 ÷ 150)” (Incorrect): That's the raw result; the maximum caps it.
  • “2, the minimum, because traffic is above the target” (Incorrect): The minimum is the floor, used when load is low.
  • “1, because the formula rounds down” (Incorrect): The formula rounds up, so there's enough capacity for the measured load.
  • “6, because ceil(1,000 ÷ 150) is 7 and the maximum is 6” is correct.

    The formula wants 7; the bound keeps it at 6 to protect the database and cap cost.

    • Watch: if it stays at the maximum, raise it or find the bottleneck.
    • Same formula: Kubernetes' HPA.

📖 Revise: Lesson 5: Scaling with load

Question ID sp-cloud-devops-09-q8 · Report a mistake in this question
9. An autoscaler removes copies the moment traffic dips, then adds them again a minute later, all afternoon. What's missing?

Why each option is right or wrong

  • “A higher maximum number of copies” (Incorrect): The maximum isn't reached; the problem is flapping.
  • “A lower target per copy for the requests” (Incorrect): That shifts the numbers but doesn't stop the flapping.
  • “Faster scale-up, so new copies arrive sooner” (Incorrect): Faster scale-up helps a little, but copies still flap.
  • “A stabilisation window before scaling down” is correct.

    Scaling down only after the recommendation stays lower for a while ignores short dips, so copies aren't removed and re-added.

    • RouteRani: --down-delay=60.
    • Kubernetes default: 5 minutes.

📖 Revise: Lesson 5: Scaling with load

Question ID sp-cloud-devops-09-q9 · Report a mistake in this question
10. Prometheus goes down for maintenance. The starter autoscaler treats the failed query as 0 requests per second. What happens?

Why each option is right or wrong

  • “Nothing, because the autoscaler waits for Prometheus to return” (Incorrect): The starter doesn't wait; it acts on the 0.
  • “It scales to the maximum to be safe while blind” (Incorrect): The starter has no such rule; it reads 0 as no traffic.
  • “It crashes and must be restarted by hand” (Incorrect): It carries on running, which is what makes the mistake dangerous.
  • “It removes copies as if traffic stopped, possibly during real load” is correct.

    With no data treated as no traffic, copies are removed while users may still be busy.

    • Fix: change nothing when the signal can't be read.
    • Test: Project 3 test 8.

📖 Revise: Lesson 5: Scaling with load

Question ID sp-cloud-devops-09-q10 · Report a mistake in this question

ProLearners Assist