My progress
Menu

Data Analyst / Engineer · Focus area 1: Data pipelines

DT2: Data quality you can prove

Goal: describe data quality in measurable terms, profile a new dataset in minutes, turn cleaning rules into checks that run every time, keep a quarantine with reasons, and prove with counts that no row went missing.

You'll need: DT1, Module 26 (Debugging and Testing), Module 55 (SQL Fundamentals), and the SabziSathi practice kit.

Where this is used: a hospital's bed-occupancy report, a bank's daily risk figures and an online shop's sales dashboard all carry an unwritten promise that the numbers are right. Data quality checks are how that promise is kept and shown.

How it differs: in app testing you check that code does what it should. In data work the code can be right and the data still wrong, because it arrives from outside. So you test the data itself, every time it arrives.

Time: reading about 2 hours · practice (the Try it tasks) about 2½ hours · assignment about 3 hours.

Lessons

  1. 1What "good data" meansReading 15–25 minutes, plus practice✓
  2. 2Profiling a new datasetReading 15–25 minutes, plus practice✓
  3. 3Validation rules and quarantineReading 15–25 minutes, plus practice✓
  4. 4Counts that add upReading 15–25 minutes, plus practice✓
  5. 5Testing data codeReading 15–25 minutes, plus practice✓

🛠️Assignment: Checks for SabziSathi's orders

This is the second part of Project 1.

  1. Profile all 30 raw order files and the customer file, and add the profile to docs/raw-notes.md.
  2. Write the row rules and table rules from your notes, each with the action it takes (quarantine, leave out and count, or stop).
  3. Add the clean.quarantine table and fill it from your rules.
  4. Add the ops.load_log table with the counts of every run, and make a load fail if its counts don't add up.
  5. Write tests/test_clean.py with unit tests for your three cleaning functions, using awkward values from the profile.

Self-check before you finish:

✅Module check

Explain in a few sentences, in your own words:

How do you show someone that a dataset can be trusted?

Show a good answer

A good answer mentions: measurable checks for completeness, validity, uniqueness, consistency, timeliness and accuracy; a profile of the raw data; row and table rules with a clear action for each failure; a quarantine table with reasons and an owner; a reconciliation that accounts for every row; a total compared with an independent source; a load log; and automated tests with independently computed expected numbers.

📚Official references

Official documentation and guidance for this module's topics, chosen on 2 November 2026. Tools, laws and standards change, so check the current version before you rely on a detail. If a link has moved, search the same site.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. A council's complaints dashboard shows complaints about streetlights fell 40% last month. A completeness check finds one ward sent

Why each option is right or wrong

  • “Publish the figure, since 40% is a real fall in the data received” (Incorrect): It's a fall in data received, not in complaints. Publishing it would mislead.
  • “Copy the ward's numbers from the month before to fill the gap” (Incorrect): That invents data and hides the problem.
  • “Remove that ward from all past months so the trend stays fair” (Incorrect): That changes history to hide a gap instead of fixing it.
  • “Mark the month incomplete and chase the ward's missing data first” is correct.

    A missing source can explain the fall. Readers need to know the figure is incomplete until the data arrives.

    • Then: add a rule such as "every ward reports at least once a day".

📖 Revise: Lesson 1: What "good data" means

Question ID sp-data-02-q1 · Report a mistake in this question
2. A profile of a new hospital admissions file shows a minimum age of -1 and a maximum of 999. What do these most likely mean?

Why each option is right or wrong

  • “Patients aged −1 and 999 years were admitted” (Incorrect): Neither is possible; they're codes or mistakes.
  • “The file was read with the wrong text encoding” (Incorrect): Encoding problems change characters, not numbers like these.
  • “Ages should be converted to months before profiling” (Incorrect): Converting units doesn't explain values that can't be real.
  • “Placeholder codes for "unknown" that need handling, not averaging” is correct.

    Systems often use values that can't be real to mean unknown. Averaging them distorts results.

    • Next: ask the source what each code means.
    • Then: treat them as unknown, and count them.

📖 Revise: Lesson 2: Profiling a new dataset

Question ID sp-data-02-q2 · Report a mistake in this question
3. A loader drops any row whose product code isn't in the product list. A new product line launches, and for two weeks its sales don't

Why each option is right or wrong

  • “A faster server, so the loader finishes before the reports run” (Incorrect): Speed has nothing to do with dropped rows.
  • “Sorting the file by product code before loading” (Incorrect): Sorting doesn't change which rows are dropped.
  • “A rule that drops even more rows that look unusual” (Incorrect): Dropping more makes the problem bigger.
  • “Quarantining those rows and reporting its size daily” is correct.

    A visible quarantine of hundreds of new-product rows would show up the next morning.

    • Then: update the product list or the rule, and reload.

📖 Revise: Lesson 3: Validation rules and quarantine

Question ID sp-data-02-q3 · Report a mistake in this question
4. A load reads 5,000 rows. The log shows 4,870 loaded, 60 quarantined and 50 repeats removed. There are no test orders. What should

Why each option is right or wrong

  • “Nothing: 4,870 is over 97% of the file, which is good enough” (Incorrect): A percentage hides the question of where 20 rows went.
  • “Load the 20 rows again on the next run” (Incorrect): You don't yet know which rows they are or why they went missing.
  • “Count the 20 rows as repeats so the numbers add up” (Incorrect): That makes the log lie.
  • “The load should fail: 20 rows aren't accounted for” is correct.

    4,870 + 60 + 50 = 4,980, not 5,000. Rows lost without a record are exactly what reconciliation is meant to catch.

    • Next: compare ids before and after each step.

📖 Revise: Lesson 4: Counts that add up

Question ID sp-data-02-q4 · Report a mistake in this question
5. A test for a revenue function has an expected value that the developer copied from the function's own output. What's wrong with

Why each option is right or wrong

  • “Nothing, as long as the test passes every time it runs” (Incorrect): It passes because it repeats the function's answer, right or wrong.
  • “Expected values should be rounded to the nearest hundred rupees” (Incorrect): Rounding doesn't fix where the value came from.
  • “Tests shouldn't contain specific numbers at all” (Incorrect): Specific, independently checked numbers are what make a test strong.
  • “It copies the function's mistakes, so it can't catch them” is correct.

    A test is useful only if its expected value comes from somewhere independent: a hand calculation, another method or a trusted report.

    • Better: work the expected value out by hand or from the raw file another way.
    • Then: a wrong function fails its test.

📖 Revise: Lesson 5: Testing data code

Question ID sp-data-02-q5 · Report a mistake in this question
6. Which check measures the uniqueness of an order-lines table?

Why each option is right or wrong

  • “Every product code exists in the product list” (Incorrect): That's consistency between two tables.
  • “The newest order is less than a day old” (Incorrect): That's timeliness.
  • “Quantities are positive whole numbers” (Incorrect): That's validity.
  • “No two rows share an order id and line number” is correct.

    Uniqueness asks whether each thing is recorded once; the order id and line number identify a line.

    • Test: count rows against distinct pairs of order id and line number.
    • In dbt: a unique test on the grain (DT7).

📖 Revise: Lesson 1: What "good data" means

Question ID sp-data-02-q6 · Report a mistake in this question
7. An upstream team changes its export without telling anyone. Which habit is most likely to show the change within a day?

Why each option is right or wrong

  • “Reading the export's documentation once a year” (Incorrect): Documentation often lags behind changes.
  • “Running the same report twice to see if it matches” (Incorrect): Two runs on the same changed data agree with each other.
  • “Asking analysts to look at the dashboard more often” (Incorrect): People notice large problems late, and small ones rarely.
  • “Comparing each day's profile with the last one” is correct.

    New distinct values, new formats or shifted minimums show up straight away in a profile comparison.

    • Then: turn the surprise into a rule.

📖 Revise: Lesson 2: Profiling a new dataset

Question ID sp-data-02-q7 · Report a mistake in this question
8. A pipeline finds a row with a missing quantity. Which action is safest?

Why each option is right or wrong

  • “Set the quantity to 1, since most orders are for a single unit” (Incorrect): That invents a value that may be wrong.
  • “Set the quantity to 0 so the row adds nothing to totals” (Incorrect): That quietly changes revenue and order counts.
  • “Delete the row so it doesn't confuse anyone” (Incorrect): Silent deletes hide problems and lose real orders.
  • “Quarantine the row with the reason and its original values” is correct.

    It keeps the problem visible and lets someone fix it at the source.

    • Then: reload once it's fixed.

📖 Revise: Lesson 3: Validation rules and quarantine

Question ID sp-data-02-q8 · Report a mistake in this question
9. Row counts in a migration match exactly, but the total of all invoice amounts is ₹2.4 lakh lower in the new system. What does this

Why each option is right or wrong

  • “The migration is correct, because every row arrived” (Incorrect): Rows arrived, but some values didn't arrive correctly.
  • “The old system's total was wrong all along” (Incorrect): Possible, but the difference appeared with the move, so check the move first.
  • “Totals naturally drift a little during migrations” (Incorrect): A correct migration keeps totals exactly.
  • “Some values changed in the move; counts alone can't show that” is correct.

    A total reconciliation catches changed values that a row count misses.

    • Next: compare totals per customer or month to find where.

📖 Revise: Lesson 4: Counts that add up

Question ID sp-data-02-q9 · Report a mistake in this question
10. Why do the SabziSathi kit's tests build a warehouse in a temporary folder?

Why each option is right or wrong

  • “Temporary folders make DuckDB queries faster” (Incorrect): Speed isn't the reason.
  • “pytest can only read files in temporary folders” (Incorrect): pytest can read files anywhere.
  • “To hide the test results from other users” (Incorrect): The results are printed; nothing is hidden.
  • “So tests don't change your own warehouse” is correct.

    Tests load, reload and break things on purpose. Doing that in a temporary database keeps your own data safe.

    • Kit: each test session builds its own warehouse with SABZI_DB set to a temporary file.
    • Your data: data/sabzisathi.duckdb is left alone.

📖 Revise: Lesson 5: Testing data code

Question ID sp-data-02-q10 · Report a mistake in this question

ProLearners Assist