AR6: Reliability across services: deadlines, retries, idempotency, outboxes and contracts
Networks fail in partial ways
🎯You'll be able to
Describe how calls between services fail, and why "slow" and "unknown" are harder to handle than "down".
🏠The everyday picture
You post a cheque and hear nothing. Did it get lost, is it still on the way, or was it paid and the receipt lost? Each needs a different response, and you can't tell which happened.
📘What it really is
-
The fallacies of distributed computing (a list from engineers at Sun Microsystems) include: the network is reliable, latency is zero, bandwidth is infinite, and the topology doesn't change. Designs that assume them break.
-
Three outcomes of a call: success, a clear failure, or no answer in time. With no answer, the request may or may not have happened.
-
Slow is worse than down. A service that refuses at once frees the caller; one that hangs ties up the caller's memory and connections, and the slowness spreads (AR1's Little's law).
-
Partial failure: some calls succeed and some don't, at the same time.
- Design question for every call: what happens if it's slow, if it fails, and if we don't know?
📝A worked example
Input: the starter's order flow, where orders waits for the SMS.
Steps:
- Provider down and refusing: the order fails fast with an error (bad, but quick).
- Provider slow (10 seconds): each order waits 10 seconds; at 3 a second, 30 orders are held at once.
- No answer at all: the order waits until something gives up, possibly never.
- Unknown: if the call times out, the SMS may still have been sent.
Expected output: a table of the four cases and what the customer sees in each, showing that slow and unknown need design.
🌍Where it's used
- Payment flows handle "unknown" results by checking the status later rather than charging again.
- Booking systems hold a seat briefly while they wait, and release it if no confirmation arrives.
- Mobile apps expect patchy networks and retry quietly.
📝Case scenario
An e-commerce site called a recommendations service on every product page with no deadline. One afternoon the service slowed to 20 seconds; every web server filled up with waiting requests and the whole site went down, though recommendations were optional. A 300 ms deadline would have shown pages without recommendations.
🖱️Try it
For one call in a system you know (an app calling a payment gateway), write what happens if it's slow, if it fails, and if you don't know whether it worked. Success: each case has a clear behaviour.
⚠️Common mistakes
- Designing for up or down only. Sign: no plan for slowness. Fix: a deadline on every call.
- Treating a timeout as "didn't happen". Sign: repeating a payment after a timeout. Fix: idempotency (Lesson 3).
- Optional parts on the critical path. Sign: the page waits for a nice-to-have. Fix: make it optional with a deadline.
🛠️For your project
Project 2's test 7 runs the monolith against a service that is down and one that never answers.
✅Quick check
Why is a slow service often worse than one that is down?
Show the answer
Callers wait and tie up resources, and the slowness spreads to them.
What does a timeout tell you about whether a request happened?
Show the answer
Nothing for certain: it may or may not have been carried out.