CD10: Capstone: three DevOps projects for RouteRani
Project 3: dashboards, alerts and scaling
🎯You'll be able to
Make the service visible and self-scaling until all 8 observability tests pass.
🏠The everyday picture
A bus depot has a board showing every bus, a bell that rings only when a bus is really late, and a supervisor who sends extra buses when queues build, and stands them down only after the rush has clearly passed.
📘What it really is
Project 3 asks you to fix:
- Prometheus (CD9 Lesson 2): a fixed target
app:3000. Usedns_sd_configsso every copy is scraped; -
Grafana (CD9 Lesson 3): not there at all. Add it in
monitoring.tfwith the provisioned data source, dashboard provider and a dashboard showing request rate, failed share and p95; -
alerts (CD9 Lesson 4): one rule on a raw counter, no
for, no runbook, and an empty test file. Write error-share, p95 and copy-down alerts withfor, severity, summary and runbook, and promtool tests that fire and stay quiet; -
the autoscaler (CD9 Lesson 5): rounds instead of rounding up, has no minimum or maximum, no stabilisation window, and treats a failed Prometheus query as zero traffic. Fix all four.
📝A worked example
Input: after your changes, run npm run test:observe.
Expected output: # pass 8 after about 9 minutes. Test 5 deploys a release that fails 25% of requests and waits for
RouteRaniHighErrorRate to fire; tests 6 and 7 run load, watch the copies reach 4 and come back to 2; test 8 stops Prometheus for 30 seconds
and checks nothing changes. In a longer run on the test machine (1 November 2026), the reference autoscaler went 2 → 4 → 6 → 2 copies with no
failed requests.
🌍Where it's used
- SRE onboarding: new engineers build exactly this for a service.
- Event readiness: before a sale or exam-results day, teams test dashboards, alerts and scaling under load.
- Incident reviews that ask "why didn't we see it sooner?".
📝Case scenario
A learner's autoscaler passes test 6 but fails test 7: copies drop from 4 to 2 straight after the load stops, before the window. Their window only looked at recommendations made so far, and at the first sample after start-up the list held a single low value. Requiring a full window of samples since the scaler started (as well as all recent recommendations being lower) fixed it.
🖱️Try it
Run the autoscaler by hand with --down-delay=60 while tools/load.js runs for 90 seconds, and watch its log. Success: you see it scale up
within a few intervals, wait at least 60 seconds after the load ends, and then scale back down, never below the minimum.
⚠️Common mistakes
- Testing alerts only in production. Sign: an alert that never fires or always fires. Fix: promtool tests for both cases.
- Dashboard queries without a data source. Sign: panels show "no data" though Prometheus has it. Fix: provision the data source with a fixed
uidand use it in panels. - Scaling without bounds. Sign: 20 copies on a laptop, or 0 at night. Fix: a minimum and maximum.
🛠️For your project
Save the dashboard during a load test, the alert firing, and the autoscaler's log on the Project 3 page.
✅Quick check
Why use dns_sd_configs instead of a fixed target?
Show the answer
So Prometheus scrapes every copy, however many there are.
What should the autoscaler do when Prometheus can't be reached?
Show the answer
Change nothing.