CD9: Monitoring, alerts and scaling
Dashboards that answer questions
🎯You'll be able to
Build a Grafana dashboard from code, with panels that answer "is the service healthy?" at a glance, and provision its data source automatically.
🏠The everyday picture
A car's dashboard shows speed, fuel and warning lights, not the temperature of every bolt. A glance tells you whether to keep driving. Mechanics have deeper tools for when a light comes on.
📘What it really is
- Grafana shows data from Prometheus (and many other sources) as dashboards of panels. It doesn't store metrics itself.
- Provisioning: instead of clicking, files tell Grafana its data sources and dashboards at start-up, so they're reviewed and versioned like other code:
monitoring/grafana/datasource.ymlnames Prometheus (url: http://prometheus:9090,uid: prometheus);dashboards.ymltells Grafana to load dashboards from a folder;-
routerani-dashboard.jsonis the dashboard itself, with each panel's query. -
A good first dashboard for a service answers, top to bottom:
- Is traffic normal? (requests per second)
- Are requests failing? (failed share, not count)
- Are they slow? (p95, perhaps p99)
-
Is it all running? (copies answering, version running) Then, lower down or on another dashboard: saturation (CPU, memory, database connections) and detail by route or copy.
-
Design habits: units on every panel (
reqps,percentunit,s), the same time range for all panels, thresholds that match your SLO, and a short description saying what "bad" looks like. -
Avoid dashboard sprawl: dozens of dashboards nobody owns slow everyone down in an incident. Start with one per service and add only what answers a real question.
📝A worked example
Input: in the finished kit with monitoring on, open http://localhost:3001 (anonymous viewing is on for practice), find "RouteRani: booking
API", and run node tools/load.js http://localhost:8080/api/routes 60 5.
Expected output: five panels: "Requests per second (API)" rising to a few hundred, "Failed requests (share of API traffic)" at 0, "p95 response time (API)" in milliseconds, "Copies answering" showing your copy count, and "Version running". Project 3 test 4 checks the dashboard has rate, failed-share and p95 queries, and that Grafana can query Prometheus through the dashboard's data source.
🌍Where it's used
- Incident rooms: the service dashboard is the first screen everyone opens.
- Release checks: engineers watch the dashboard during and after a deploy.
- Business reviews: product teams use the same tools for bookings or sign-ups per minute.
📝Case scenario
During a payments outage, engineers at a fintech company opened eleven different dashboards, each showing something slightly different, and spent 20 minutes agreeing on whether errors were rising. Afterwards, each service got one provisioned "health" dashboard with the same four questions at the top, and incident runbooks linked to it directly.
🖱️Try it
Add a sixth panel to your dashboard JSON: p99 response time, with the same query as p95 but 0.99. Restart Grafana with
terraform apply -replace='docker_container.grafana[0]' and check the panel appears. Success: the new panel shows data, and the change is in Git.
⚠️Common mistakes
- Counts instead of rates or shares. Sign: "errors: 15,203" with no idea if that's bad. Fix: errors as a share of requests.
- Dashboards made by clicking only. Sign: they vanish when Grafana is rebuilt. Fix: provision data sources and dashboards from files.
- No units. Sign: nobody knows if 0.5 means seconds or milliseconds. Fix: set the unit on every panel.
🛠️For your project
Project 3 test 4 looks for a dashboard with "RouteRani" in its title and rate, 5xx and histogram_quantile queries, with live data.
✅Quick check
Why provision Grafana from files?
Show the answer
So data sources and dashboards are reviewed, versioned and recreated identically.
What four questions should a service's main dashboard answer first?
Show the answer
Is traffic normal, are requests failing, are they slow, is everything running.