DT2: Data quality you can prove
Validation rules and quarantine
🎯You'll be able to
Turn expectations into validation rules, apply them to each row, and keep rows that fail in a quarantine table with a reason.
🏠The everyday picture
At an airport, bags that fail the scan aren't thrown away or let through: they go to a separate belt, with a note saying why, and someone checks them.
📘What it really is
-
A row rule checks one row: the quantity is a positive whole number; the product code exists; the date parses and falls on the file's day.
-
A table rule checks the whole load: no repeated keys; row count within an expected range; at least one order per store.
- For each failure, choose one of three actions and write it down:
- quarantine the row, with the file, row number, reason and the original values;
- leave it out on purpose, and count it (staff test orders, exact repeats of a resent row);
-
stop the load (no rows at all, or the file is for the wrong day).
-
A quarantine table makes problems visible and fixable:
| source_file | row_number | reason | raw_row |
|---|---|---|---|
| orders/2026-10-05.csv | 118 | qty must be a positive whole number | {"order_id": "SS-100562", "qty": "-1", ...} |
- Check rules in order and record the first that fails, so each row has one clear reason.
- Someone owns the quarantine: they look at it, fix the source or the rule, and reload.
📝A worked example
Input: the solution's loader on 5 October (python -m pipeline.run --date 2026-10-05), then
select reason, count(*) from clean.quarantine group by reason.
Expected output: 2 rows with "qty must be a positive whole number" and 1 with "unknown sku". Across all 30 days: 60 and 30. Nothing else fails, because the date, amount and status rules from DT1 read every other value.
🌍Where it's used
- Payments: transactions that fail checks are held for review instead of being posted.
- Healthcare: lab results with impossible values are flagged before they reach a patient's record.
- Data platforms: tools such as Great Expectations, Soda and dbt tests run rules on every load.
📝Case scenario
An insurer's claims pipeline dropped rows that failed checks. A hospital's claims had a new date format and vanished for a month; the hospital called to ask why nothing was paid. With a quarantine table, the first day's 400 quarantined claims would have shown up on the morning report.
🖱️Try it
In the starter's pipeline/run.py, find every place a row is dropped without being counted. Success: you can point to the line that
drops rows silently and say what the solution does instead.
⚠️Common mistakes
- Dropping failed rows silently. Sign: totals fall and nobody knows why. Fix: quarantine with a reason.
- Fixing data by guessing. Sign: a missing quantity becomes 1. Fix: quarantine, then ask the source.
- Nobody owns the quarantine. Sign: it grows for months. Fix: report its size daily and name an owner.
🛠️For your project
Project 1 test 6 checks that 90 rows are quarantined, each with a reason, across the month.
✅Quick check
What three things can a rule do with a failing row?
Show the answer
Quarantine it, leave it out on purpose and count it, or stop the load.
What goes in a quarantine row?
Show the answer
The file, the row number, the reason and the original values.