Module 94: Distributed Systems
Failures
🏠The everyday picture
A phone call drops mid-sentence. Did the other person hear your last words? You can't tell. Ringing back and repeating yourself is safe only if repeating doesn't cause harm.
📘What it really is
-
In distributed systems, partial failure is normal: one service is slow, a network drops, a message is duplicated, a server restarts mid-task.
-
The fundamental uncertainty: when a call times out, you don't know whether it failed or succeeded but the reply was lost. So:
- idempotency keys: the caller sends a unique key with a request (
Idempotency-Key: booking-812-pay); the receiver remembers results by key and returns the same result for repeats. Payment gateways support this; your own APIs should for important operations (Module 46); - retries with exponential backoff and jitter (random variation, so many clients don't retry at the same instant), and a limit;
- timeouts on every network call: never wait forever;
- circuit breakers: after repeated failures to a service, stop calling it for a while and fail fast (or use a fallback), so a struggling service can recover and your app doesn't pile up waiting requests;
- bulkheads: separate resources (connection pools, workers) per dependency, so one slow service can't use up everything;
-
graceful degradation: if recommendations are down, still show search results; if SMS fails, send email (Module 84).
-
Common fallacies of distributed computing: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn't change. Each is false and has caused outages.
-
Clocks disagree between machines; don't rely on comparing timestamps from different servers for ordering. Use database sequences or version numbers.
-
Test failures deliberately: stop a service, add delays, drop messages in staging (chaos testing, Module 96).
🖱️Try it
Add idempotency keys to your POST /v1/bookings and payment-creation calls: store the key and result, and return the stored result
on repeats. Add timeouts and backoff-with-jitter retries to your payment gateway adapter. Wrap the email adapter in a simple circuit
breaker (for example with the opossum library). In staging, make the payment gateway (or a fake) slow and failing, and check the app
stays responsive and nothing is double-charged.
✅Quick check
After a timeout, why is it unsafe to simply retry a payment?
Show the answer
The first request may have succeeded with the reply lost; without an idempotency key, the retry could charge twice.
What does a circuit breaker do?
Show the answer
After repeated failures, it stops calling a service for a while and fails fast, protecting both sides.