Module 72: Production Operations
Alerts
🏠The everyday picture
A smoke alarm should ring for smoke, not for toast. One that rings every morning gets its battery removed, and then it's no use on the day of a real fire.
📘What it really is
- An alert tells a person something needs attention now. Good alerts are:
- about symptoms users feel: the site is down, errors are high, pages are slow; not every small internal wobble;
- actionable: someone can do something about it;
-
rare: too many alerts cause alert fatigue, and real ones get ignored.
-
Useful starting alerts for your CRM:
- uptime: an outside service checks
/healthevery minute and alerts if it fails twice in a row (UptimeRobot, Better Stack, or your provider); - error rate: more than, say, 2% of requests failing for 5 minutes;
- latency: the 95th percentile above 2 seconds for 10 minutes;
- disk over 80% full; certificate expiring within 14 days;
-
backup failed or didn't run (Lesson 4).
-
Levels: urgent alerts go to your phone (a call, SMS or app notification); less urgent ones go to email or a chat channel for working hours.
-
Each alert should link to a short runbook: what it means and the first steps to check.
- Test your alerts: stop the app on staging and confirm the alert arrives, and how fast.
🖱️Try it
Set up an outside uptime check on your staging /health, with alerts to your email or phone. Stop the API container and time
how long until you're told. Write a one-paragraph runbook for "site down": what to check first (container status, logs,
disk, database), and how to roll back.
✅Quick check
What's alert fatigue?
Show the answer
So many alerts that people start ignoring them, including the important ones.
Why check uptime from outside your server?
Show the answer
If the server itself is down, it can't report that it's down.