DT3: Building reliable pipelines
When things go wrong
🎯You'll be able to
Make failures loud, log enough to diagnose them, alert the right person, and write a short runbook for the common failures.
🏠The everyday picture
A smoke alarm is useful because it's loud and specific: it goes off in the room with the smoke. An alarm that beeps quietly in the basement, or every day for no reason, is soon ignored.
📘What it really is
-
Fail loudly: a missing or broken file must end the run with a non-zero exit code, so the scheduler marks it failed. Printing an error and exiting with 0 looks like success.
-
Log each run (DT2): start and finish time, file, status, counts and the error message. A failed run gets a log row too.
- Alert on what matters:
- the run failed;
- the data is stale (freshness, DT7);
- the quarantine is unusually large (for example more than 1% of rows);
-
the row count is far from normal (for example under half of the usual).
-
Send alerts where someone will see them (email, chat, an on-call tool), with what failed, for which day, and a link to the log.
-
A runbook lists common failures and what to do: "file missing → check the source's upload; re-run with
--datewhen it arrives". -
Don't publish bad data: if checks fail, keep yesterday's dashboard with a note, rather than publishing wrong numbers.
📝A worked example
Input: python -m pipeline.run --date 2026-10-31 with the solution (there's no file for 31 October), then echo $?.
Expected output: ERROR: no raw file for 2026-10-31 ... and exit code 1. ops.load_log has a failed row for that date with the
message missing file orders/2026-10-31.csv, and the earlier days' rows are unchanged. The starter stops with a Python error and logs
nothing.
🌍Where it's used
- Data platforms: failed runs page the data engineer on call.
- Finance: a missing bank file blocks the morning reconciliation and alerts the operations team.
- Analytics: a "data delayed" banner on dashboards when freshness fails.
📝Case scenario
An airline's daily bookings file failed to arrive for three days. The loader printed a warning and exited normally, so the scheduler showed green. The revenue team made pricing decisions on three-day-old data. A non-zero exit code and a freshness alert would have flagged it on the first morning.
🖱️Try it
Write a five-line runbook for SabziSathi with three failures: a missing file, a quarantine over 1% of rows, and a dbt build test
failure. For each, give the sign, the first check and the fix. Success: someone else could follow it at 6 a.m.
⚠️Common mistakes
- Exit code 0 on failure. Sign: the scheduler shows success for a broken run. Fix: exit non-zero.
- Alerting on everything. Sign: people mute the channel. Fix: alert on failures and real problems only.
- Publishing after failed checks. Sign: wrong numbers in the morning meeting. Fix: keep the last good version and say so.
🛠️For your project
Project 1 test 9 checks a missing file ends with a non-zero exit code and a failed row in the load log.
✅Quick check
Why does the exit code matter?
Show the answer
The scheduler uses it to decide whether the run failed.
What should happen to the dashboard when checks fail?
Show the answer
Keep the last good version, with a note that new data is delayed.