BE9: Testing and running in production
Health, SLOs and error budgets
🎯You'll be able to
Set a reliability target users would notice, measure it, and use the error budget to decide between new features and reliability work.
🏠The everyday picture
A bus company promises that 95% of buses arrive within five minutes of the timetable. It doesn't promise every bus is on time, which would be impossible to keep. If it's at 97% this month, there's room to try a new route; if it's at 93%, the next priority is punctuality.
📘What it really is
- An SLI (service level indicator) measures something users feel: "the share of menu requests that answer successfully within 300 ms".
- An SLO (service level objective) is the target for an SLI over a window: "99.5% over 28 days".
- An SLA is a contract with customers, often with refunds; it's set looser than the SLO.
-
The error budget is what the SLO allows to go wrong: 0.5% of requests. If the menu gets 2,000,000 requests in 28 days, the budget is 10,000 slow or failed requests.
-
Use the budget:
- plenty left: release features, run experiments;
-
nearly used: slow releases, fix reliability first.
-
Alert on symptoms, not causes: page someone when the budget is burning fast ("at this rate it's gone in 2 days"), not for every high CPU reading. Each page should need a human to act.
-
Health endpoints (BE8 Lesson 5) feed load balancers; SLOs measure what users experience. Both are needed.
📝A worked example
Input: TiffinBox's menu route. Over 28 days: 2,000,000 requests; 6,500 took over 300 ms; 1,200 returned 5xx.
Calculation: bad requests = 6,500 + 1,200 = 7,700. Good share = (2,000,000 − 7,700) ÷ 2,000,000 = 99.615%. SLO 99.5% is met. Budget 10,000; used 7,700, so 77% used and 2,300 remaining.
Output: the team can release, but carefully; most budget went on slow responses, so caching work (BE4) comes before new menu features. (Illustrative figures.)
🌍Where it's used
- Large platforms, which use SLOs and error budgets to agree priorities between product and engineering.
- Cloud providers, whose SLAs are commitments backed by service credits.
- Internal teams, setting SLOs for services other teams depend on.
📝Case scenario
A payments team was paged 40 times a week for CPU and disk alerts, most of which needed no action, and they started ignoring pages. They replaced them with two alerts on the checkout SLO's burn rate. Pages dropped to about two a week, and each one was a real problem affecting customers.
🖱️Try it
Write an SLO for TiffinBox's POST /v1/orders: the SLI, the target, the window and the error budget for 500,000 orders a month. Then write
one burn-rate alert in plain words. Success: the SLI is something a customer would feel, and the budget is a number of orders.
⚠️Common mistakes
- Aiming for 100%. Sign: no change is ever safe enough to release. Fix: a target users accept, leaving a budget for change.
- SLIs users don't feel. Sign: "CPU under 70%" as an objective. Fix: measure success and latency as users see them.
- Alerting on everything. Sign: pages nobody acts on. Fix: alert on budget burn; send other signals to dashboards.
🛠️For your project
Your capstone write-up includes one SLO for the menu and one for orders, with the error budget worked out.
✅Quick check
What is an error budget?
Show the answer
The amount of failure an SLO allows over its window, such as 0.5% of requests.
Why alert on symptoms rather than causes?
Show the answer
Users feel symptoms; many causes need no action, and too many alerts get ignored.