My progress
Menu

Software Architect / Tech Lead · Focus area 2: Technical strategy

AR6: Reliability across services: deadlines, retries, idempotency, outboxes and contracts

Goal: design calls between services that survive slow and failing networks: deadlines, sensible retries, circuit breakers, idempotency keys, an outbox for work that can't be done now, and contracts that callers can rely on.

You'll need: AR2, AR5, Module 24 (Asynchronous Programming), Module 47 (REST API Design), Module 94 (Distributed Systems), Module 96 (Reliability Engineering), and the RasoiRoute practice kit (services/notify, contracts/notify-v1.json).

Where this is used: payments, messaging, bookings, and anywhere one system calls another over a network. Most outages between services aren't caused by a service being down, but by it being slow, or by a retry doing something twice.

How it differs: inside one process, a function call either returns or throws. Across a network there's a third outcome: you don't know. This module is about designing for that.

Time: reading about 3 hours · practice (the Try it tasks) about 3 hours · assignment about 4 hours.

Lessons

  1. 1Networks fail in partial waysReading 15–25 minutes, plus practice✓
  2. 2Deadlines, retries and circuit breakersReading 15–25 minutes, plus practice✓
  3. 3Idempotency keysReading 15–25 minutes, plus practice✓
  4. 4The outboxReading 15–25 minutes, plus practice✓
  5. 5Contracts and versionsReading 15–25 minutes, plus practice✓

🛠️Assignment: Make notifications survive the network

  1. Build the notify service to contracts/notify-v1.json, with health checks, validation and idempotency.
  2. Add the remote client with a 500 ms deadline, treating 5xx and timeouts as "try later" and 4xx as "failed".
  3. Add a durable outbox and drainOutbox(), removing messages only after success.
  4. Run npm run test:extraction and fix until 10 of 10 pass.
  5. Write two paragraphs: what happens to an order when the service is slow, and why a duplicate in the outbox doesn't reach the customer.

Self-check before you finish:

✅Module check

Explain in a few sentences, in your own words:

How do you make one service's calls to another safe when the network is unreliable?

Show a good answer

A good answer mentions: three outcomes (success, failure, unknown) and slowness spreading; a deadline on every call chosen from what the caller can afford; retries only for brief failures and safe requests, with exponential backoff, jitter and limits; circuit breakers; idempotency keys derived from the action, stored long enough, with repeats answered with the stored result; a durable outbox drained later with at-least-once delivery, removed only after success, and watched; and a written contract with clear errors, contract tests, health checks and versions for breaking changes.

📚Official references

Official documentation and guidance for this module's topics, chosen on 2 November 2026. Tools, laws and standards change, so check the current version before you rely on a detail. If a link has moved, search the same site.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. A product page calls a recommendations service with no deadline. It slows to 20 seconds and the whole site goes down. What's the

Why each option is right or wrong

  • “Remove recommendations from the site so the call stops happening” (Incorrect): They're useful; the call just needs limits.
  • “Add more web servers so there's room for the waiting requests” (Incorrect): More servers fill up too, just a little later.
  • “Retry the recommendations call three times on every page view” (Incorrect): Retries make a slow service slower.
  • “A short deadline, and show the page without recommendations if it passes” is correct.

    The optional part can't hold up the page any more.

    • Name: graceful degradation.

📖 Revise: Lesson 1: Networks fail in partial ways

Question ID sp-architect-06-q1 · Report a mistake in this question
2. A call to a payment gateway times out. What can you conclude?

Why each option is right or wrong

  • “The payment failed, so it's safe to charge the card again” (Incorrect): It may have gone through; charging again risks a double charge.
  • “The payment succeeded, as gateways only time out after charging” (Incorrect): A timeout tells you nothing for certain.
  • “The gateway is down, so all payments should stop for the day” (Incorrect): One timeout doesn't show the gateway is down.
  • “It's unknown; check the status or repeat with the same key” is correct.

    Treat a timeout as "unknown" and resolve it safely.

    • Tool: an idempotency key.

📖 Revise: Lesson 1: Networks fail in partial ways

Question ID sp-architect-06-q2 · Report a mistake in this question
3. Every service in a ticketing platform retries failed calls five times with no wait. During a sale, the seat service collapses. Why?

Why each option is right or wrong

  • “Five retries are too few to get through during a busy sale” (Incorrect): More retries would add more load.
  • “The seat service's code is too slow for any ticket sale” (Incorrect): It was overloaded by the retries, not by the sale alone.
  • “Ticket sales can't be handled by services at all” (Incorrect): They can, with limits on retries.
  • “The retries multiplied its load, so it couldn't recover” is correct.

    Retry storms turn a slowdown into a collapse.

    • Fix: fewer retries, backoff, jitter, a circuit breaker.

📖 Revise: Lesson 2: Deadlines, retries and circuit breakers

Question ID sp-architect-06-q3 · Report a mistake in this question
4. The notify service answers 422 "unknown template". What should the monolith do?

Why each option is right or wrong

  • “Put it in the outbox and keep retrying until it's accepted” (Incorrect): It will be refused every time.
  • “Retry at once, three times, with the same Idempotency-Key” (Incorrect): Same request, same refusal.
  • “Stop taking orders until the template has been added” (Incorrect): One missing template shouldn't stop ordering.
  • “Record it as failed and not retry, as repeating can't help” is correct.

    A 4xx means the request itself is wrong.

    • Kit: status failed, not queued.

📖 Revise: Lesson 2: Deadlines, retries and circuit breakers

Question ID sp-architect-06-q4 · Report a mistake in this question
5. A school's fee portal retried timed-out payments with a new request each time, and some parents were charged twice. What's the fix?

Why each option is right or wrong

  • “Turn off retries so that a timed-out payment is simply lost” (Incorrect): Then parents' payments go missing instead.
  • “Ask parents to check their bank statements after paying” (Incorrect): That moves the problem to parents.
  • “Wait a full day before retrying any timed-out payment” (Incorrect): Waiting doesn't stop the double charge.
  • “One idempotency key per fee payment, sent with every retry” is correct.

    The gateway recognises retries and charges once.

    • Key: from the payment, not the attempt.

📖 Revise: Lesson 3: Idempotency keys

Question ID sp-architect-06-q5 · Report a mistake in this question
6. Which Idempotency-Key is best for RasoiRoute's order confirmation message?

Why each option is right or wrong

  • “A new random number generated for each send attempt” (Incorrect): Each retry would look like a new message.
  • “The current time in milliseconds when the message is sent” (Incorrect): That changes on every retry.
  • “The customer's phone number on its own” (Incorrect): Two orders from one customer would share it.
  • “<orderId>:order-placed, the same on every retry” is correct.

    It identifies the real action, so repeats are recognised.

    • Kit: RR1001:order-placed.

📖 Revise: Lesson 3: Idempotency keys

Question ID sp-architect-06-q6 · Report a mistake in this question
7. A courier company's "out for delivery" SMS were lost while the provider was down for an hour. What would have saved them?

Why each option is right or wrong

  • “Sending each message ten times in a row in case one gets through” (Incorrect): Repeats don't help when the provider is down, and annoy people when it's up.
  • “Keeping the messages in memory until the provider comes back” (Incorrect): Memory is lost on restart.
  • “Asking riders to phone each customer during an outage” (Incorrect): That doesn't scale and wastes riders' time.
  • “Writing each message to a durable outbox first, then sending it later” is correct.

    Messages wait safely and go out when the provider is back.

    • Plus: keys so repeats are harmless.

📖 Revise: Lesson 4: The outbox

Question ID sp-architect-06-q7 · Report a mistake in this question
8. An outbox drain sends a message, then crashes before removing it. What happens, and why is it acceptable?

Why each option is right or wrong

  • “The message is lost, which is acceptable for SMS messages” (Incorrect): It isn't lost; and losing messages isn't the goal.
  • “The outbox file is corrupted and has to be deleted by hand” (Incorrect): Appending lines and removing after success keeps it readable.
  • “The service refuses all further messages from the monolith” (Incorrect): A repeat key gets a 200 with the same ID, not a block.
  • “It's sent again next time, and the key makes the repeat harmless” is correct.

    At-least-once delivery plus idempotency gives one message.

    • Kit: test 8.

📖 Revise: Lesson 4: The outbox

Question ID sp-architect-06-q8 · Report a mistake in this question
9. A delivery platform renamed eta to estimatedArrival in place, and restaurant apps showed blank times. What should it have done?

Why each option is right or wrong

  • “Told restaurants to update their apps on the same day” (Incorrect): Callers can't all change at once.
  • “Kept both names forever, with no plan to remove either” (Incorrect): Adding is right, but removal needs a plan.
  • “Renamed it back the next day and left it unchanged after that” (Incorrect): That avoids the change rather than making it safely.
  • “Add the new field beside eta, and retire eta later” is correct.

    Additive changes don't break callers; removal comes with notice.

    • Also: contract tests.

📖 Revise: Lesson 5: Contracts and versions

Question ID sp-architect-06-q9 · Report a mistake in this question
10. Which change to contracts/notify-v1.json can be made without a new version?

Why each option is right or wrong

  • “Renaming to.phone to to.mobile for every request” (Incorrect): Every existing caller would break.
  • “Changing the 202 reply so that it no longer includes id” (Incorrect): Callers that read id would break.
  • “Making the Idempotency-Key header optional and ignored” (Incorrect): Callers rely on it to make repeats safe.
  • “Adding an optional to.email field beside to.phone” is correct.

    Existing requests still work as before.

    • Then: contract tests for both.

📖 Revise: Lesson 5: Contracts and versions

Question ID sp-architect-06-q10 · Report a mistake in this question

ProLearners Assist