AR6: Reliability across services: deadlines, retries, idempotency, outboxes and contracts
Deadlines, retries and circuit breakers
🎯You'll be able to
Set a deadline on a call, retry with backoff and jitter where it's safe, and explain how a circuit breaker protects both sides.
🏠The everyday picture
If a shop's phone line is busy, you don't redial every second; you wait a little longer each time, and after a few tries you go in person or come back tomorrow.
📘What it really is
-
A deadline (timeout) limits how long a caller waits:
AbortSignal.timeout(500)withfetchin Node.js. Choose it from what the caller can afford, not from how slow the other side might be. -
Retries help with brief failures, but only for requests that are safe to repeat (Lesson 3), and only a few times.
-
Exponential backoff with jitter: wait longer between tries (for example 200 ms, 400 ms, 800 ms) plus a random amount, so thousands of callers don't retry at the same instant.
-
Retry storms: if every caller retries three times, a struggling service gets four times the load. Retries need limits.
- A circuit breaker stops calling a service that keeps failing (open), tries one call after a pause (half-open), and returns to normal when it works (closed). Callers fail fast or use a fallback meanwhile.
📝A worked example
Input: RasoiRoute's remote sender calling the notify service.
Steps:
- Deadline: 500 ms. An order can afford that; it can't afford 10 seconds.
- On timeout or a 5xx answer: don't retry inside the order request; put the message in the outbox (Lesson 4).
- On a 4xx answer (bad request, unknown template): don't retry at all; record it as failed, because repeating won't help.
- The outbox is drained later, with backoff between attempts.
Expected output: an order request that never waits more than about half a second for messaging, and no retry storm.
🌍Where it's used
- Cloud SDKs retry with exponential backoff and jitter by default for many calls.
- Service meshes and gateways apply deadlines and circuit breakers between services.
- Payment gateways publish which errors are safe to retry.
📝Case scenario
A ticketing platform's services each retried failed calls five times with no backoff. When the seat service slowed during a big sale, the retries multiplied its load six-fold and it collapsed entirely. Fewer retries with backoff, and a circuit breaker, would have let it recover.
🖱️Try it
Write the waits for four retries with exponential backoff starting at 200 ms, then add a random jitter of up to 100 ms to each. Success: the waits grow, and two callers would almost never retry at the same moment.
⚠️Common mistakes
- No deadline. Sign: requests that hang for minutes. Fix: a deadline on every network call.
- Retrying everything. Sign: retries on "400 bad request". Fix: retry only brief failures, and only safe requests.
- Retrying inside the user's request. Sign: an order that waits for three retries. Fix: hand off to an outbox or background job.
🛠️For your project
Project 2 sets timeoutMs (500 in the reference). Test 7 checks the order answers within 1.5 seconds when the service never responds.
✅Quick check
Why add jitter to retry waits?
Show the answer
So many callers don't retry at the same instant and overload the service again.
What are a circuit breaker's three states?
Show the answer
Closed (calls go through), open (calls stop and fail fast), half-open (one test call after a pause).