My progress
Menu

Level 8 · Advanced Software Architecture and Technology Management · Course 8.2 Performance and Reliability

Module 96: Reliability Engineering

Goal: design systems that keep working when parts fail, see clearly inside them when something goes wrong, recover from disasters, plan capacity ahead of demand, and build a culture where reliability improves steadily.

You'll need: the booking platform with its distributed workflow and scaling work (Modules 94 and 95), and your Level 6 operations guide.

Lessons

  1. 1Fault tolerance15–25 minutes✓
  2. 2Observability15–25 minutes✓
  3. 3Disaster recovery15–25 minutes✓
  4. 4Capacity planning15–25 minutes✓
  5. 5A reliability culture15–25 minutes✓

🛠️Module practice: Project C, part 2, a reliability plan

  1. An FMEA table, a reduced-service plan with feature flags, and three chaos experiments recorded.
  2. OpenTelemetry traces through API, queue and worker, a business metric and a booking-flow dashboard.
  3. A DR plan with RPO and RTO, a chosen strategy, protected backups, infrastructure as code and a tabletop drill.
  4. A 12-month capacity model with peaks, headroom, lead times, costs and provider limits.
  5. A production readiness checklist, scored, and an error-budget policy.

With Module 95, this completes your Reliability and Performance project (Project C).

✅Module check

Explain in a few sentences, in your own words:

How do you engineer a system to be reliable?

Show a good answer

A good answer mentions: fault tolerance by removing single points of failure, using timeouts, retries, circuit breakers, bulkheads and graceful degradation with a planned reduced service and load shedding, tested through FMEA and chaos experiments; observability through logs, metrics and traces (OpenTelemetry) correlated by trace IDs, with business metrics and SLO-based alerts; disaster recovery with RPO and RTO, a strategy from backup-and-restore to active-active, infrastructure as code, protected backups and drills; capacity planning from usage, measured limits, forecasts, peaks, headroom and lead times; and an SRE culture of error budgets, blameless reviews, less toil, runbooks, safe changes, sustainable on-call and readiness reviews.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. What is fault tolerance?

Why each option is right or wrong

  • “The system is tested so thoroughly that it has no bugs” (Incorrect): Testing reduces bugs but doesn't remove them; fault tolerance plans for failures anyway.
  • “Every part runs on one powerful server to keep things simple” (Incorrect): One server is a single point of failure, the opposite of fault tolerance.
  • “Errors are caught and hidden from logs so users aren't worried” (Incorrect): Hiding errors makes problems harder to find; tolerance means carrying on safely.
  • “The system keeps working, perhaps degraded, when a part fails” is correct.

    Parts fail all the time: a server crashes, a disk dies, a third-party API slows down. A fault-tolerant system is designed so one failure doesn't take everything down. It may run with fewer features or a little slower, but users can still do the important things.

    • How: redundancy, timeouts, retries and fallbacks.
    • Accept: failures will happen; plan for them.

📖 Revise: Lesson 1: Fault tolerance

Question ID module-96-q1 · Report a mistake in this question
2. The recommendations service is down. What's graceful degradation?

Why each option is right or wrong

  • “Show an error page for the whole site until the service is back” (Incorrect): That turns one small failure into a full outage.
  • “Restart the main database so that every service reconnects” (Incorrect): The database isn't the problem, and restarting it causes more disruption.
  • “Keep the page waiting until the recommendations service replies” (Incorrect): Waiting ties up the page; degrading means carrying on without it.
  • “Show the page without recommendations instead of failing the whole page” is correct.

    Recommendations are nice to have; the product page is essential. If the recommendations call fails or times out, the page still loads with the main content, perhaps with a simple fallback like "popular items". Users barely notice.

    • Decide in advance: which features are essential and which are optional.
    • Use timeouts: so a slow service doesn't hold the page.

📖 Revise: Lesson 1: Fault tolerance

Question ID module-96-q2 · Report a mistake in this question
3. What are the three pillars of observability?

Why each option is right or wrong

  • “CPU, memory and disk usage” (Incorrect): Those are useful metrics, but only one part of observability.
  • “Alerts, dashboards and status pages” (Incorrect): These are built on top of logs, metrics and traces.
  • “Uptime, response time and error rate” (Incorrect): These are key metrics, but logs and traces are needed too.
  • “Logs, metrics and traces” is correct.

    Logs are detailed records of events, metrics are numbers over time like error rate and latency, and traces follow one request across services. Together they let you see what's happening inside a system and work out why something went wrong.

    • Metrics: tell you something is wrong.
    • Logs and traces: help you find out why.

📖 Revise: Lesson 2: Observability

Question ID module-96-q3 · Report a mistake in this question
4. What does a distributed trace show?

Why each option is right or wrong

  • “The full source code of each service that handled the request” (Incorrect): Traces show timing and steps, not source code.
  • “The total number of requests each server handled in a day” (Incorrect): That's a metric; a trace follows one request.
  • “Which data centre and country each server is located in” (Incorrect): Location may appear as a tag, but a trace is about the request's path.
  • “One request's path and timing across all the services it touched” is correct.

    When a request passes through the frontend, API, payment service and database, a trace records each step with its timing. You can see exactly where it slowed down or failed, which is hard to work out from separate logs.

    • How: a trace ID passed along with each call.
    • Tools: OpenTelemetry, with viewers like Jaeger.

📖 Revise: Lesson 2: Observability

Question ID module-96-q4 · Report a mistake in this question
5. What's the difference between RTO and RPO?

Why each option is right or wrong

  • “RTO is the target for restoring backups; RPO is the target for restarting servers” (Incorrect): Both apply to the whole recovery; one is about time, the other about data.
  • “RPO measures yearly uptime; RTO measures how much a recovery is allowed to cost” (Incorrect): Neither measures cost; RTO is time to recover, RPO is data loss.
  • “They're two names for the same recovery target, used by different cloud providers” (Incorrect): They measure different things: downtime and data loss.
  • “RTO is how long recovery may take; RPO is how much data you may lose” is correct.

    Recovery time objective (RTO): how long the system can be down, say 4 hours. Recovery point objective (RPO): how far back in time you might lose data, say 15 minutes, which decides how often you back up or replicate.

    • Lower targets: cost more.
    • Agree them: with the business.

📖 Revise: Lesson 3: Disaster recovery

Question ID module-96-q5 · Report a mistake in this question
6. Why practise disaster recovery with drills?

Why each option is right or wrong

  • “Because cloud providers require proof of drills before hosting” (Incorrect): Most providers don't require it; drills are for your own confidence.
  • “To check that the servers run faster after a restore” (Incorrect): Speed isn't the aim; finding gaps in the plan is.
  • “To use up the time set aside in the yearly plan” (Incorrect): Drills have a real purpose: proving recovery works.
  • “Untested plans usually fail when they're needed” is correct.

    Backups turn out to be incomplete, passwords are missing, and steps are out of date. A drill, such as restoring last night's backup to a fresh server, finds these problems while there's time to fix them, and shows whether you can meet your RTO and RPO.

    • Schedule: at least a few times a year.
    • Record: how long it took and what went wrong.

📖 Revise: Lesson 3: Disaster recovery

Question ID module-96-q6 · Report a mistake in this question
7. What is capacity planning?

Why each option is right or wrong

  • “Buying the biggest server available now so you don't have to plan” (Incorrect): That wastes money and may still not match future needs.
  • “Writing a roadmap of the features the team plans to build and launch next year” (Incorrect): That's product planning; capacity is about load and resources.
  • “Deleting inactive users regularly to free up space and keep the database fast” (Incorrect): Cleaning data can help, but it isn't capacity planning.
  • “Estimating future load and making sure resources will be ready in time” is correct.

    Using growth trends, launches and seasonal peaks, such as exam results or festival sales, you estimate what load is coming and check you'll have enough servers, database capacity and budget before you need them.

    • Look at: current usage, growth rate and known events.
    • Leave headroom: for unexpected spikes.

📖 Revise: Lesson 4: Capacity planning

Question ID module-96-q7 · Report a mistake in this question
8. What is an error budget?

Why each option is right or wrong

  • “The amount of money set aside each year for fixing bugs and paying for support” (Incorrect): It's measured in allowed downtime or errors, not money.
  • “A list of known errors that the team has decided to accept rather than fix this year” (Incorrect): That's a known-issues list; an error budget is an allowance of failure.
  • “The number of failed deployments each developer is allowed before reviews are required” (Incorrect): It applies to the service's reliability, not individuals.
  • “The downtime your uptime target allows; when it's used up, reliability work comes first” is correct.

    A 99.9% uptime target allows about 43 minutes of downtime a month. That allowance is the error budget. While there's budget left, the team ships features; when it's spent, they focus on reliability until things are stable.

    • Balances: speed of change and stability.
    • Agreed: between product and engineering.

📖 Revise: Lesson 5: A reliability culture

Question ID module-96-q8 · Report a mistake in this question
9. What does a blameless postmortem focus on?

Why each option is right or wrong

  • “Finding the person responsible so they can be retrained” (Incorrect): Blame makes people hide information; the focus is the system.
  • “Rewriting the affected part of the system from scratch” (Incorrect): Fixes should match the causes found, not be a rewrite by default.
  • “Keeping the incident private so customers don't hear of it” (Incorrect): Openness, inside and often outside the team, is part of learning.
  • “How the system and process allowed the failure, and what to change” is correct.

    People make mistakes; good systems make them hard to make and easy to catch. A blameless postmortem asks what made the mistake possible, such as missing checks, confusing tools or unclear steps, so people speak openly and the real causes get fixed.

    • Output: a timeline, causes and action items with owners.
    • Share it: so others learn too.

📖 Revise: Lesson 5: A reliability culture

Question ID module-96-q9 · Report a mistake in this question
10. What's a health check endpoint used for?

Why each option is right or wrong

  • “Running the full test suite against production on each request” (Incorrect): Health checks are quick status checks, not full tests.
  • “Showing users a status page about which of the site's features are working today” (Incorrect): It's for machines like load balancers, not a user page.
  • “Measuring how many users are signed in at any moment, to decide when to scale up” (Incorrect): That's a usage metric, not a health check.
  • “Letting load balancers and monitors know if an instance can serve traffic” is correct.

    A health check is a small URL, such as /health, that returns OK when the app and its essential dependencies are working. Load balancers call it regularly and stop sending traffic to instances that fail; monitors alert you.

    • Keep it quick: and cheap to call.
    • Don't expose: sensitive details in its response.

📖 Revise: Lesson 2: Observability

Question ID module-96-q10 · Report a mistake in this question

ProLearners Assist