Module 72: Production Operations
Incident response
🏠The everyday picture
A fire drill: everyone knows their role, the exits and the meeting point, so on the real day they act calmly instead of panicking.
📘What it really is
- An incident is anything that harms users or data: an outage, very slow pages, wrong data, a security breach (Module 80).
- A simple process:
- Detect: an alert, a user report, or something you notice.
- Declare and assign: say it's an incident; one person leads (even if you're the only one).
- Mitigate first: restore service quickly, often by rolling back (Module 70), restarting, or turning off a feature flag. Find the root cause later.
- Communicate: update the status page and affected users in plain words: what's affected, what you're doing, when the next update comes.
- Resolve and verify: confirm metrics are back to normal.
-
Review: write a short post-incident review.
-
Keep a timeline as you go (times and actions), which makes the review easy and honest.
-
A blameless review asks "how did our system allow this?" rather than "who did it?": what happened, impact, timeline, root cause, what went well, and action items with owners and dates. People share the truth when they're not afraid.
-
Prepare in advance: an incident checklist, contact list, access to logs and dashboards, and your rollback and restore steps.
-
Severity levels help decide urgency: for example, SEV1 (site down or data at risk), SEV2 (major feature broken), SEV3 (minor issue).
🖱️Try it
Run an incident drill on staging: have someone (or yourself, with a timer) break the app (stop the database, or deploy a bad version). Follow your checklist: detect, mitigate, communicate (write the status update), resolve. Then write a blameless post-incident review with at least two action items.
✅Quick check
In an incident, what comes first: finding the root cause or restoring service?
Show the answer
Restoring service; investigate the root cause once users are no longer affected.
Why should post-incident reviews be blameless?
Show the answer
So people share what really happened, and the team fixes the system rather than blaming individuals.