AR6: Reliability across services: deadlines, retries, idempotency, outboxes and contracts
The outbox
🎯You'll be able to
Use an outbox so work that can't be done now is kept safely and sent later, with at-least-once delivery made safe by idempotency.
🏠The everyday picture
If the post office is closed, you put the letter in your outbox tray and post it tomorrow. You don't stand at the door, and you don't throw the letter away.
📘What it really is
-
The outbox pattern: instead of calling a service directly in the middle of an important action, record the message durably, finish the action, and send the message later.
-
Durable means it survives a restart: a database table in the same transaction as the order, or (in the kit) a file of JSON lines.
-
A drain reads waiting messages, sends them, and removes the ones that succeeded. It can run on a schedule or when the service is back.
-
At-least-once: a message might be sent twice (for example if the drain crashes after sending but before removing it). Idempotency keys make that harmless.
-
Watch it: an outbox that keeps growing means something is wrong. Alert on its size and age.
📝A worked example
Input: the notify service is down; three orders arrive.
Steps:
- Each order is paid and saved; the remote call fails within 500 ms.
- Each message is appended to the outbox with its key; each order shows
notification: queued. - The service comes back;
drainOutbox()sends three messages and empties the file. - A second drain sends nothing.
Expected output: no order lost or delayed by the outage, and each customer messaged once when the service returned.
🌍Where it's used
- E-commerce platforms write "order placed" events to an outbox table in the same transaction as the order.
- Banks queue notifications and statements for later delivery.
- Mobile apps keep an offline queue and sync when back online (the Mobile path covers this).
📝Case scenario
A courier company sent "out for delivery" messages directly from its delivery app. When the SMS provider was down for an hour, those messages were lost for good. Writing them to an outbox first would have sent them when the provider returned.
🖱️Try it
Sketch the outbox for an action in a system you know: what's recorded, where, and what removes it. Success: you can explain what happens if the drain crashes half-way.
⚠️Common mistakes
- An outbox in memory. Sign: messages lost on restart. Fix: store it durably.
- Removing before sending. Sign: messages lost when a send fails. Fix: remove only after success.
- Nobody watching. Sign: 10,000 waiting messages found a week later. Fix: alerts on size and age.
🛠️For your project
Project 2's tests 7 and 8 check that orders go through with the message queued, and that draining sends each message once.
✅Quick check
Why must the outbox be durable?
Show the answer
So waiting messages survive a restart instead of being lost.
Why can an outbox send a message twice, and why is that acceptable?
Show the answer
A crash after sending but before removing it causes a resend; idempotency keys make the repeat harmless.