DT1: Python and pandas for messy data
Raw data and the question behind it
🎯You'll be able to
Look at a raw file before cleaning it, say what one row means (its grain), and list the problems you can see.
🏠The everyday picture
Before a tailor cuts cloth, they measure twice and check the cloth for flaws. Cutting first and checking later wastes the cloth. Data is the same: look before you change anything.
📘What it really is
- Raw data is data as it arrived: files, exports, API responses. Keep it unchanged, so you can always go back and clean it again.
-
The grain is what one row stands for. In SabziSathi's order files, one row is one line of one order (an order of three vegetables has three rows). Getting the grain wrong is the most common reason a total is wrong: count rows when you meant orders and you count three times too many.
-
A first look answers: how many rows and columns, what each column holds, which values are missing, and which look odd.
- Questions to ask the people who send the file:
- Is a file ever sent twice, or corrected later?
- Are there test orders, staff orders or cancelled ones?
-
Which time zone are the times in, and which currency are the amounts in?
-
Write down what you find before you fix anything. Those notes become your cleaning rules (Lesson 3) and your tests (DT2).
📝A worked example
Input: in the SabziSathi kit, open raw/orders/2026-10-05.csv in a text editor, then run:
import pandas as pd
df = pd.read_csv('raw/orders/2026-10-05.csv', dtype=str, keep_default_na=False)
print(len(df), df.duplicated().sum(), (df['customer_id'] == 'TEST').sum())
print(df['status'].value_counts())
Expected output: 317 10 2: 317 rows, 10 of them exact repeats of another row, and 2 staff test orders. The status counts show ten
different spellings, such as Delivered, DELIVERED and delivered with spaces, for what are really three statuses. Measured on
1 November 2026 with pandas 3.1.0.
🌍Where it's used
- Retail and delivery: daily order exports from shop systems and delivery apps.
- Healthcare: appointment and lab-result files sent between systems overnight.
- Public services: beneficiary lists shared as spreadsheets between departments.
📝Case scenario
A district office reported that a scheme reached 1.2 lakh families. An analyst noticed the list had one row per family member, not per family. The real number of families was about 31,000. The report was corrected before it went to the state government.
🖱️Try it
Open raw/customers.csv and raw/products.json. For each, write one sentence saying what one row (or one object) stands for, and list
two things that look inconsistent. Success: you've found the mixed date formats in signup_date and the mixed upper- and lower-case
emails.
⚠️Common mistakes
- Cleaning before looking. Sign: you can't say how many rows changed. Fix: count and note problems first, then clean.
- Guessing the grain. Sign: an "orders" total is three times higher than the shop expects. Fix: check whether an id repeats, and why.
- Editing the raw file by hand. Sign: re-running your work gives different results. Fix: leave raw files unchanged and clean in code.
🛠️For your project
Project 1 starts here: the notes you write about the raw order files become the pipeline's cleaning rules and its tests.
✅Quick check
What is the grain of a table?
Show the answer
What one row stands for, such as one order line, one order or one customer.
Why keep raw files unchanged?
Show the answer
So you can clean them again when a rule turns out to be wrong.