DT10: Capstone: three data projects for SabziSathi

Lesson 2 of 5 · reading 15–25 minutes; practice and project time is on the module page

Project 1: a pipeline you can trust

🎯You'll be able to

Bring the starter's loader up to a reliable standard that passes all 10 pipeline tests.

🏠The everyday picture

A bank's cash counting machine sorts notes, rejects torn or fake ones into a separate tray, and prints a slip that says how many notes went in and where each one ended up. The slip has to add up.

📘What it really is

Project 1 asks you to fix these in the starter:

cleaning rules (DT1): parse_time reads one of three formats, parse_amount uses floats and misses commas and "Rs", and statuses are kept as written. Fix: three explicit date formats, whole paise from a strict parser, and normalised statuses checked against a list;
what's kept (DT2): rows that fail are dropped silently, exact repeats and staff test orders are kept, and customer_id is text. Fix: remove repeats and test orders and count them, quarantine failures with a reason in clean.quarantine, and store customer_id as a number;
counts (DT2 Lesson 4): add ops.load_log with the counts for each run, and make a load fail if they don't add up;
re-runs and failures (DT3): each run appends. Replace the file's rows inside a transaction with a primary key, and make a missing file end with a non-zero exit code and a failed log row;
the schedule (DT3 Lesson 4): refresh.yml runs on every push with write-all, no concurrency and the dashboard before the tests. Fix: a daily cron in UTC, read-only permissions, a concurrency group, a timeout, and load → dbt build → dashboard.

📝A worked example

Input: after your changes, run pytest tests/test_pipeline.py -v.

Expected output, in part:

tests/test_pipeline.py::test_01_one_day_loads_the_right_rows PASSED
tests/test_pipeline.py::test_02_running_a_day_twice_gives_the_same_result PASSED
...
tests/test_pipeline.py::test_10_the_refresh_is_scheduled_safely PASSED
10 passed

On the test machine on 1 November 2026, the reference solution passed all 10 in about 5 seconds. Across the month it loaded 10,181 lines, quarantined 90, and counted 329 repeats and 60 test-order lines, adding up to the 10,660 rows read.

🌍Where it's used

Taking over loaders that work on good days only.
Audits that ask how a reported figure was checked.
Data operations: a load log and quarantine reviewed every morning.

📝Case scenario

A learner's test 1 keeps failing with 301 rows for 5 October instead of 302. Their quarantine has an extra row with the reason "order date outside the file date": an order at 05-Oct-2026 12:05 AM was parsed as 5 October 00:05 correctly, but their check compared the date with the file name using a different format. Printing the one quarantined row's original values found it in minutes.

🖱️Try it

Fix the cleaning functions first and run only tests 3, 4 and 8 (pytest tests/test_pipeline.py -k "03 or 04 or 08"), then move on to the loader. Success: each fix turns at least one more test green, and you know which change did it.

⚠️Common mistakes

Fixing totals without fixing counts. Sign: revenue matches but test 6 fails. Fix: account for every row in the load log.
Quarantining repeats. Sign: 329 extra quarantined rows. Fix: count exact repeats separately; quarantine is for rows that break rules.
A schedule in local time. Sign: test 10 passes but the job would run at 11:30 in Pune. Fix: write the cron in UTC with a comment.

🛠️For your project

Save the full pytest tests/test_pipeline.py -v output and the month's load log totals on the Project 1 page.

✅Quick check

What must the load log's counts add up to?

Show the answer

Rows read = loaded + quarantined + repeats + test orders.

What does test 2 prove?

Show the answer

Running a day twice gives the same result, so re-runs and retries are safe.