CD9: Monitoring, alerts and scaling
What to measure, and how good is good enough
🎯You'll be able to
Choose the signals that describe a service's health, and set service level objectives with an error budget.
🏠The everyday picture
A bus company doesn't judge a route by how shiny the buses are. It asks: did buses come (availability), were they on time (latency), and how many passengers rode (traffic)? It also accepts that one bus in a thousand will be late, and plans around that rather than promising perfection.
📘What it really is
- Three kinds of telemetry:
- metrics: numbers over time (requests per second, error count, memory), cheap to store and good for dashboards and alerts;
- logs: records of individual events, good for finding out what happened to one request;
-
traces: the path of one request across several services, with timings. OpenTelemetry is the common standard for producing all three.
-
The RED method for request-driven services: Rate (requests per second), Errors (failed requests), Duration (how long they take). Google's "four golden signals" add saturation (how full the service is: CPU, memory, connections).
-
Percentiles, not averages: an average response time of 120 ms can hide 5% of users waiting 3 seconds. p95 is the time that 95% of requests beat; p99 shows the slowest 1%.
-
SLI, SLO, SLA:
- a service level indicator is a measurement, such as "the share of booking requests that succeed in under 500 ms";
- a service level objective is the target for it, such as "99.5% over 30 days";
-
a service level agreement is a promise to customers with consequences (refunds), usually looser than the internal SLO.
-
Error budget: the failures the SLO allows. 99.9% over 30 days allows about 43 minutes of failure (30 × 24 × 60 × 0.001 = 43.2). If the budget is nearly used, the team slows releases and works on reliability; if there's plenty left, they can take more risk.
-
100% is the wrong target. It's impossible to reach, extremely expensive to approach, and users can't tell 99.99% from 100% through their own phone network's failures.
📝A worked example
Input: RouteRani's API exposes /metrics. Run curl -s localhost:8080/metrics | grep -E '^(http_requests_total|http_request_duration_seconds_bucket)' | head.
Expected output: lines such as http_requests_total{method="GET",route="/api/routes",code="200"} 1520 (a counter for each method, route and
status code) and http_request_duration_seconds_bucket{...,le="0.05"} 1498 (how many requests finished within 50 ms). These give you Rate and
Errors (from the counter) and Duration (from the histogram buckets). An SLI for RouteRani could be "the share of /api/ requests that don't
return 5xx", with an SLO of 99.5% over 30 days.
🌍Where it's used
- SRE teams set SLOs for each service and use error budgets to balance speed against reliability.
- Payment companies track payment success rate and p99 latency as their main health signals.
- Status pages report availability against published targets.
📝Case scenario
A food delivery company's dashboard showed average order-API latency of 150 ms, so the team believed it was fast. Complaints said otherwise. A p99 panel showed 4 seconds during lunch for the slowest 1% of requests, which turned out to be orders with many items. They set an SLO on p99 and fixed the slow query, and complaints dropped.
🖱️Try it
Work out the error budget in minutes for 99.5%, 99.9% and 99.95% over 30 days. Then choose one SLI and SLO for RouteRani's booking endpoint and write why you chose that target. Success: 216, 43.2 and 21.6 minutes, and a target with a reason that isn't "as high as possible".
⚠️Common mistakes
- Measuring causes instead of symptoms. Sign: alerts on CPU while users see errors unnoticed. Fix: measure what users experience first: rate, errors, duration.
- Averages for latency. Sign: "fast on average" with angry users. Fix: p95 or p99.
- Aiming for 100%. Sign: endless effort for tiny gains, and fear of every release. Fix: an SLO users would accept, and an error budget.
🛠️For your project
Project 3's dashboard shows request rate, failed share and p95, and its alerts are based on errors and p95.
✅Quick check
What do RED's three letters stand for?
Show the answer
Rate, Errors, Duration.
How much failure does a 99.9% SLO over 30 days allow?
Show the answer
About 43 minutes.