CD9: Monitoring, alerts and scaling
Alerts people trust
🎯You'll be able to
Write alert rules on user-facing symptoms with sensible waiting times, severities and runbooks, test them before they're used, and avoid alert fatigue.
🏠The everyday picture
A smoke alarm that goes off every time you make toast gets its battery taken out. Then it's silent on the day of a real fire. A good alarm reacts to real smoke, and the instructions for what to do are on the back.
📘What it really is
- Alert on symptoms users feel (errors, slowness, unavailability), not on every cause (CPU at 80% may be perfectly fine).
- Wait before firing:
for: 1mmeans the condition must hold for a minute, so a single blip doesn't wake anyone. - Severity decides who's told and how:
page: wake someone now (users are affected and it's getting worse);-
ticket: someone looks during working hours (one copy down while others serve). -
Every alert says what to do: a
summaryin plain words and arunbook(a link or short steps): what to check first, how to roll back. - RouteRani's error alert: ```yaml
-
alert: RouteRaniHighErrorRate expr: | sum(rate(http_requests_total{route=~"/api/.", code=~"5.."}[1m])) / sum(rate(http_requests_total{route=~"/api/."}[1m])) > 0.05 for: 1m labels: severity: page annotations: summary: "More than 5% of booking API requests are failing" runbook: "Check the latest release first (routerani_info shows the version); roll back if it started after a deploy." ```
-
Test alert rules like code:
promtool check rules alerts.ymlchecks syntax;promtool test rules alerts.test.ymlfeeds made-up series through the rules and checks which alerts fire, so you know a rule fires at 10% errors and stays quiet at 1%. -
Alertmanager receives firing alerts from Prometheus, groups related ones, routes them (pager for
page, chat or tickets forticket), and supports silences during planned work. -
SLO burn-rate alerts (an advanced form) page when the error budget is being used up fast, and open a ticket when it's being used slowly.
- Alert fatigue: if most pages need no action, people stop responding. Review alerts regularly; delete or downgrade the noisy ones.
📝A worked example
Input: in the finished kit, run docker run --rm -v "$PWD/monitoring:/w" -w /w --entrypoint promtool prom/prometheus:v3.15.1 test rules alerts.test.yml.
Then, with monitoring on, deploy the practice fault with terraform apply -var monitoring=true -var release_fault=errors and send traffic.
Expected output: promtool prints SUCCESS for the three test cases (10% errors fires the error alert, 1% doesn't, a stopped copy fires the
copy-down alert after a minute). After the faulty release, Prometheus's Alerts page shows RouteRaniHighErrorRate as pending, then firing about
a minute later, because the release fails about 25% of requests. Project 3 test 5 checks this happens within 150 seconds and that no alert fires
under normal traffic. Apply again without the fault afterwards.
🌍Where it's used
- On-call rotations: engineers take turns carrying the pager, and good alerts keep that bearable.
- Incident tools (PagerDuty, Opsgenie, Grafana OnCall) route alerts from Alertmanager.
- Post-incident reviews ask "did an alert fire, and did it say what to do?"
📝Case scenario
A bank's operations team received about 300 alerts a night, mostly "CPU above 70%" on busy but healthy servers. A real database failure was buried among them and noticed by customers first. They deleted alerts that never led to action, alerted on payment errors and latency instead, added runbooks, and nights dropped to a handful of meaningful pages.
🖱️Try it
In alerts.test.yml, add a case where 4% of requests fail for 5 minutes and expect no RouteRaniHighErrorRate alert, then run promtool. Then
change the expected result to an alert and run it again. Success: the first run passes and the second fails, so you've seen the tests catch a
wrong expectation.
⚠️Common mistakes
- Alerts on raw counters. Sign: an alert that fires forever once a total passes a number. Fix: rates and ratios.
- No
for. Sign: pages for 10-second blips. Fix: a duration that matches how quickly someone must act. - No runbook. Sign: the person paged at 3 am doesn't know where to start. Fix: a summary and first steps on every alert.
🛠️For your project
Project 3 tests 1, 2 and 5 check the rules and their tests, that each alert has for, severity, summary and runbook, and that the error alert
fires for a bad release and stays quiet otherwise.
✅Quick check
Why add for: 1m to an alert?
Show the answer
So the condition must last a minute before anyone is paged, ignoring blips.
What does promtool test rules check?
Show the answer
Which alerts fire, given made-up input series.