Lesson 2 of 5 · reading 15–25 minutes; practice and project time is on the module page

Prometheus and PromQL

🎯You'll be able to

Collect metrics with Prometheus, find every copy automatically, and write PromQL queries for request rate, error share and percentiles.

🏠The everyday picture

A meter reader visits every house on the list every month and writes down the reading. The reading on its own means little; what matters is the difference since last month. And if a new house is built, it has to be added to the list.

📘What it really is

Prometheus pulls ("scrapes") metrics from each target's /metrics page on a schedule (RouteRani: every 5 seconds) and stores them as time series: a metric name, labels and values over time.
Metric types:
counter: only goes up, resetting to 0 when the process restarts (http_requests_total);
gauge: goes up and down (memory in use, copies running);
histogram: counts observations into buckets (le="0.05", le="0.1", ...), so percentiles can be estimated across all copies.
Finding copies: a fixed static_configs target such as app:3000 scrapes whatever one address that name gives at the time, so most copies are missed. Service discovery finds them all: RouteRani uses dns_sd_configs with the name app, which returns every copy; Kubernetes and clouds have their own discovery.
PromQL essentials:
rate(http_requests_total[1m]): per-second increase over the last minute, which handles restarts. Raw counters are almost never what you want on a graph or in an alert;
sum(rate(http_requests_total{route=~"/api/.*"}[1m])): total API requests per second across all copies;
error share: promql sum(rate(http_requests_total{route=~"/api/.*", code=~"5.."}[1m])) / sum(rate(http_requests_total{route=~"/api/.*"}[1m]))
p95: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{route=~"/api/.*"}[2m])));
up{job="routerani"} is 1 for each target scraped successfully, 0 for each that failed.
Labels and cardinality: each distinct combination of labels is a separate series. Labels like route and code are fine; a label with user IDs or full URLs creates millions of series and can overload Prometheus.

📝A worked example

Input: in the finished kit, terraform apply -var monitoring=true -var app_count=3, then open http://localhost:9090 and run count(up{job="routerani"} == 1) and, while node tools/load.js http://localhost:8080/api/routes 60 5 runs, the error-share query above.

Expected output: the count is 3, one per copy. The error share is 0 (or shows "no data" until requests arrive). With the starter's static_configs target, the count is 1 however many copies run. That's Project 3 test 3.

🌍Where it's used

Kubernetes clusters almost always run Prometheus or a compatible system.
Managed services: Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus and Grafana Cloud accept the same queries.
Long-term storage: Thanos, Cortex, Mimir and VictoriaMetrics keep Prometheus data for months.

📝Case scenario

A team added user_id as a label to their request counter to see which users had errors. Within a week, Prometheus held 4 million series, used all its memory and crashed during an incident, when it was needed most. They removed the label, kept per-user detail in logs and traces, and added a limit that rejects scrapes with too many series.

🖱️Try it

In Prometheus, compare http_requests_total (the raw counter) with rate(http_requests_total[1m]) on a graph while load runs, then stop the load. Success: the raw counter only ever climbs, while the rate rises with the load and falls back towards 0 when it stops.

⚠️Common mistakes

Graphing or alerting on raw counters. Sign: lines that only go up. Fix: rate() over a window.
A fixed target for many copies. Sign: one target in Prometheus while three copies run. Fix: service discovery.
High-cardinality labels. Sign: Prometheus memory climbing every day. Fix: no IDs or free text in labels.

🛠️For your project

Project 3 test 3 runs 3 copies and expects 3 healthy targets, which needs dns_sd_configs (or similar) instead of one fixed address.

✅Quick check

Why use rate() on a counter?

Show the answer

A counter only rises; rate() turns it into a per-second speed and handles restarts.

What does histogram_quantile(0.95, ...) give you?

Show the answer

An estimate of the time 95% of requests finished within.