CD9: Monitoring, alerts and scaling
Scaling with load
🎯You'll be able to
Scale a service horizontally on a signal that matches its work, with bounds and a stabilisation window, and prove with a load test that scaling fails no requests.
🏠The everyday picture
A toll plaza opens more booths as queues grow, and closes them only after traffic has stayed light for a while, not the moment one car is left. There's a fixed number of booths it can't go beyond, and it never closes them all.
📘What it really is
-
Vertical scaling gives one copy more CPU or memory (simple, limited, often needs a restart). Horizontal scaling adds copies (needs stateless copies and a load balancer, CD2), which is how most web services scale.
-
The formula used by Kubernetes' HPA and RouteRani's
tools/autoscale.js: desired copies = ceil(current load ÷ target load per copy), kept between a minimum and a maximum. At 520 requests per second with a target of 150 per copy: ceil(520 ÷ 150) = 4. -
Choosing the signal: CPU works for CPU-bound services; requests per second fits RouteRani; queue length fits workers. The signal should rise before users feel pain.
-
Target per copy: load-test one copy to find where p95 starts to climb, then choose a target comfortably below it (a safety margin).
-
Up fast, down slowly: add copies straight away when load rises; remove them only after the recommendation has stayed lower for a full stabilisation window (RouteRani's default: 60 seconds; Kubernetes' default: 5 minutes). Otherwise copies flap up and down.
-
Bounds: a minimum (at least 2, for availability) and a maximum (to cap cost and protect the database).
-
When the signal is missing, do nothing. An autoscaler that treats "Prometheus is down" as "zero traffic" would scale to the minimum during an outage of the monitoring system.
-
Scaling has limits: the database, a third-party API or the computer itself can become the bottleneck. On the 4-core test machine, nginx and RouteRani together reached about 1,100 to 1,160 requests per second however many copies ran, because the machine's CPUs were the limit.
📝A worked example
Input: in the finished kit, terraform apply -var monitoring=true, start node tools/autoscale.js --min=2 --max=6 --target=150 --down-delay=60,
then run node tools/load.js http://localhost:8080/api/routes 90 10, and watch docker ps --filter name=routerani-dev-app.
Expected output: the autoscaler logs lines such as {"rps":612.3,"copies":2,"desired":5,"next":5} and scaling from 2 to 5 copies within a
few intervals; after the load stops, it waits for the window and logs scaling from 6 to 2 copies (or from wherever it peaked). In a run on
1 November 2026 the copies went 2 → 4 → 6 → 2 with no failed requests. If you stop Prometheus, the log shows no change this round and the
copy count stays put. These are Project 3 tests 6, 7 and 8.
🌍Where it's used
- Sale days, match days, result days: autoscaling adds copies as traffic arrives.
- Cost control: scaling down at night saves money in a cloud.
- Queue workers scale on backlog so jobs don't pile up.
📝Case scenario
A retailer's autoscaler scaled down the moment traffic dipped and up again a minute later, all afternoon. Each scale-up took two minutes, so users hit slow pages repeatedly. Adding a 10-minute stabilisation window for scale-down, and scaling on requests per second rather than CPU (which lagged), kept the copy count steady through the dips.
🖱️Try it
Read the starter's tools/autoscale.js and list its problems: rounding with Math.round, no minimum or maximum, no stabilisation window, and
treating a Prometheus error as 0 requests. For each, describe what would happen during a real traffic spike or monitoring outage. Success: four
problems, each with a consequence.
⚠️Common mistakes
- No minimum. Sign: the service scales to 0 or 1 copy at night and fails its first morning requests. Fix: a minimum of at least 2.
- No stabilisation window. Sign: copies flapping up and down. Fix: scale down only after a calm window.
- Acting on missing data. Sign: scaling to the minimum when monitoring breaks. Fix: change nothing when the signal can't be read.
🛠️For your project
Project 3 tests 6 to 8 check that the autoscaler reaches its maximum under load without failing requests, comes back to its minimum (never below) after a wait, and changes nothing while Prometheus is unreachable.
✅Quick check
At 700 requests per second with a target of 150 per copy and a maximum of 4, how many copies?
Show the answer
ceil(700 ÷ 150) = 5, capped at the maximum: 4.
Why scale down slowly but up quickly?
Show the answer
Up quickly protects users from load; down slowly avoids removing copies that are needed again a moment later.