DT10: Capstone: three data projects for SabziSathi
Planning the capstone
🎯You'll be able to
Plan the capstone: scope, order of work, evidence and time.
🏠The everyday picture
A water supply engineer fixes the filters at the treatment plant first, then the pipes, then the taps. Polishing the taps while dirty water flows through them means doing it twice.
📘What it really is
- The three projects, all in the SabziSathi kit:
- A pipeline you can trust: every date and money format read, money in paise, duplicates and test orders removed, bad rows
quarantined with reasons, counts that add up, safe re-runs, a loud failure for a missing file and a safe daily schedule (10 tests in
pytest tests/test_pipeline.py); - A warehouse with personal data protected: a dbt build that passes, one row per order line, no personal data in the marts, salted
keys, area history, net revenue leaving out cancellations and refunds, sources with freshness, safe incremental rebuilds and full
documentation (10 tests in
pytest tests/test_warehouse.py); - A dashboard people can trust: built from the marts only, with the right headline numbers, plain definitions, honest bars, small
counts hidden, no personal data, a freshness line and text alternatives (8 tests in
pytest tests/test_dashboard.py). -
Build in this order: Project 1 first (the warehouse's numbers depend on clean data), then Project 2 (the dashboard reads its marts), then Project 3.
-
Each project has acceptance tests in the kit and a marking guide on its project page. Add your own criteria too, such as "the quarantine is under 1% of rows on every day".
-
Use this overview to judge your own work (each project page has the detailed marking guide):
| Area | What "good" looks like |
|---|---|
| Cleaning | Every format read on purpose, money in paise, nothing guessed (DT1) |
| Quality | Measurable rules, quarantine with reasons, counts that add up, tests with independent numbers (DT2) |
| Pipeline | Idempotent, backfillable, scheduled safely, loud on failure (DT3) |
| Scale judgement | A reasoned view on streaming, Spark and file formats (DT4) |
| Warehouse | Layers, cost awareness, least privilege and personal data minimised (DT5) |
| Model | Declared grain, star schema, correct measures, type 2 history, salted keys (DT6) |
| Tested code | dbt tests, docs, incremental models and freshness (DT7) |
| Analysis | Readable SQL, window functions, cohorts and funnels without traps (DT8) |
| Dashboard | Decisions first, defined metrics, honest charts, privacy and freshness (DT9) |
📝A worked example
A learner's capstone plan (illustrative):
| Week | Work | Done when |
|---|---|---|
| 1 | Project 1: raw notes, profile, cleaning functions and unit tests | Pipeline tests 3, 4 and 8 pass |
| 2 | Project 1: quarantine, load log, idempotent loads, missing file, schedule | All 10 pipeline tests pass twice in a row |
| 3 | Project 2: staging without personal data, salted keys, area history | Warehouse tests 1 to 6 pass |
| 4 | Project 2: net revenue, sources and freshness, incremental fact, docs | All 10 warehouse tests pass |
| 5 | Project 3: brief, marts-only build, definitions, honest bars, suppression | All 8 dashboard tests pass |
| 6 | Capstone checks, demo, self-review, design notes and case studies | pytest tests shows 28 passed; every project page saved as self-checked complete |
Doing Project 1 first meant the warehouse's revenue matched the expected ₹19,65,764 as soon as the model was right, instead of chasing a number that was wrong for reasons upstream.
🌍Where it's used
- Data teams taking over a fragile reporting set-up.
- Analytics engineering projects that move scheduled queries into dbt.
- Portfolios for data analyst, analytics engineer and data engineer applications.
📝Case scenario
A learner starts with the dashboard because it's the part people see. They get its layout right, but the headline revenue is wrong, and they spend two days adjusting the dashboard's SQL. The real cause is upstream: the starter pipeline loses rows with commas in their amounts and keeps cancelled orders. Fixing Project 1 first would have made the dashboard's numbers right without touching its SQL.
🖱️Try it
Download the kit, follow sabzisathi/README.md, and run pytest tests once (about 20 seconds; most tests fail at this stage). Then write
your plan with a "done when" check for each week. Success: your plan lists all 28 tests across the three projects.
⚠️Common mistakes
- Starting without running the tests. Sign: no idea how far you are. Fix: run
pytest testsfirst and record what fails. - Starting with the dashboard. Sign: chasing wrong numbers in the last step. Fix: pipeline, then warehouse, then dashboard.
- Leaving evidence to the end. Sign: no record of what changed. Fix: save test output and numbers as each project passes.
🛠️For your project
Run the kit's tests once before you plan, so you know what fails now: on the starter, 2 of the 28 pass: Project 2 test 1 (the starter's dbt project builds, it just builds the wrong things) and test 5 (its date dimension already covers every order date).
✅Quick check
Why build Project 1 before Project 2?
Show the answer
The warehouse's numbers depend on the pipeline's cleaning; wrong input makes every later number wrong.
What should you record as you go?
Show the answer
Test output and key numbers for each project, saved as each one passes.