DT2: Data quality you can prove
Counts that add up
🎯You'll be able to
Reconcile a load so every row read is accounted for, compare totals with an independent source, and log the counts of every run.
🏠The everyday picture
At the end of the day a shop counts the cash and checks it against the sales slips. If they don't agree, the shop doesn't close until it knows why.
📘What it really is
- Row reconciliation: for every load,
rows read = rows loaded + rows quarantined + exact repeats + rows left out on purpose.
If the two sides differ, rows were lost or invented somewhere, and the load should fail.
-
Total reconciliation: compare a total (revenue, count of orders) with an independent figure: the source system's own report, a payment settlement or yesterday's total for the same day.
-
A load log records each run: when it ran, which file, its status, and each of the counts. It answers "did yesterday's load run, and what did it do?" without opening any code.
-
Independent checks are worth more than repeated ones. Recomputing a total with the same code finds nothing; recomputing it from the raw file another way (a separate script, a spreadsheet, the source's report) can.
📝A worked example
Input: after loading all 30 days, run
select sum(rows_in), sum(rows_loaded), sum(rows_quarantined), sum(rows_duplicate), sum(rows_test) from ops.load_log where status = 'ok'.
Expected output: 10660, 10181, 90, 329, 60, and 10,181 + 90 + 329 + 60 = 10,660. A separate script that reads the raw files with
Python's csv module, without the pipeline's code, also finds 10,181 lines and ₹19,65,764 of delivered sales. Both were checked on
1 November 2026.
🌍Where it's used
- Finance: month-end reconciliations of ledgers and bank statements.
- Data migrations: counts and totals per table compared before and after a move.
- Regulated reporting: evidence that a figure was checked against a source.
📝Case scenario
A school board moved exam results to a new system. Row counts matched, but a reconciliation of total marks per school showed one school was 9,000 marks short: a subject's marks had been loaded into the wrong column for 300 students. Counts alone would have passed.
🖱️Try it
Load 5 October with the solution's pipeline and check the reconciliation for that day by hand from the log. Success: 317 = 302 + 3 + 10 + 2.
⚠️Common mistakes
- Checking counts but not totals. Sign: the right number of rows with wrong values. Fix: reconcile at least one total too.
- Rechecking with the same code. Sign: checks pass while the code is wrong. Fix: compare with an independent calculation.
- Logging only failures. Sign: you can't tell whether yesterday's load ran at all. Fix: log every run with its counts.
🛠️For your project
Project 1 test 6 checks the reconciliation for every load, and test 9 checks a failed run is logged.
✅Quick check
Write the row reconciliation equation.
Show the answer
Rows read = loaded + quarantined + repeats + rows left out on purpose.
Why is an independent check better than re-running the same code?
Show the answer
The same code repeats the same mistakes; a different method can catch them.