Timeouts, retries and circuit breakers
🎯You'll be able to
Keep a slow or failing service from dragging others down, using timeouts, careful retries, circuit breakers and fallbacks.
🏠The everyday picture
You phone a shop. If nobody answers after a minute, you hang up and try later instead of holding forever. If the line has been dead all morning, you stop calling for a while and send a message instead. Waiting forever, or calling every second, helps nobody.
📘What it really is
-
Timeouts: every network call needs one. Without it, a slow service holds your threads or connections until you run out and fail too. Set it from what the caller can afford: if the whole request must answer in 1 second, a dependency gets a few hundred milliseconds.
-
Retries: retry only temporary failures (timeouts,
503) and only idempotent operations (or with an idempotency key, BE1). - wait longer each time (exponential backoff) with some randomness (jitter) so callers don't retry in step;
-
limit them (2 or 3 tries) and keep a retry budget, because retries multiply load on a service that's already struggling.
-
Circuit breaker: after many failures, stop calling for a while (open), answer from a fallback, then let a test call through (half-open); if it works, go back to normal (closed).
-
Fallbacks: answer something useful without the dependency: menu without ratings, cached data, or "try again soon".
- Bulkheads: separate connection pools per dependency, so one slow dependency can't use up every connection.
📝A worked example
Input: the finished TiffinBox API with Redis stopped, then GET /v1/kitchens/3/menu.
The kit's Redis client uses connectTimeout: 500, commandTimeout: 500 and enableOfflineQueue: false, so commands fail within about
half a second instead of waiting. The menu code catches that and loads from the database.
Expected output: 200 with the correct menu and X-Cache: BYPASS, within 1.5 seconds. When Redis is back, X-Cache: MISS then HIT.
This is Project 2 test 9. The rate limiter also fails open on Redis errors (BE6 Lesson 3).
🌍Where it's used
- Service meshes and libraries (Envoy, resilience4j, Polly) provide timeouts, retries and breakers.
- Payment calls: retried only with idempotency keys.
- Recommendation or rating panels: dropped quietly when their service is slow, so the main page still loads.
📝Case scenario
An airline's booking page called a seat-map service with no timeout. When the seat-map service hung, booking servers filled up with waiting requests and the whole site went down for two hours, although bookings didn't need seat maps. A 300 ms timeout, a breaker and a "choose seats later" fallback would have kept bookings working.
🖱️Try it
In the finished kit, start the API, stop Redis and request the menu three times with curl -i. Note X-Cache and the time taken
(curl -w '%{time_total}'). Start Redis again and repeat. Success: answers stay correct and fast enough while Redis is down, and caching
resumes.
⚠️Common mistakes
- No timeout, or a default of minutes. Sign: requests pile up when one dependency slows. Fix: an explicit timeout on every call.
- Retrying everything. Sign: retries of
POSTwithout an idempotency key make doubles. Fix: retry only safe or keyed operations. - Retry storms. Sign: every layer retries 3 times, so one failure becomes 27 calls. Fix: retry at one layer, with backoff, jitter and a budget.
🛠️For your project
Project 2 test 9 checks the menu answers within 1.5 s with Redis down; Project 3 test 5 checks orders are still accepted.
✅Quick check
When is it safe to retry a request?
Show the answer
When the failure is temporary and the operation is idempotent or carries an idempotency key.
What does an open circuit breaker do?
Show the answer
It stops calling the failing service for a while and uses a fallback, then tests with a single call.