DT7: dbt: transformations as tested code
dbt in a team
🎯You'll be able to
Use separate development and production environments, run dbt in CI on every pull request, check source freshness, and review changes to models safely.
🏠The everyday picture
A newspaper's reporters write drafts in their own files; editors check them; only approved pages go to print. Nobody edits the printed paper directly.
📘What it really is
-
Environments: each developer builds into their own schema or database (for example
dev_meera); the scheduled job builds production. Targets inprofiles.ymlselect which one. -
Pull requests: every model change goes through Git review. Reviewers check the SQL, the grain, the tests and the descriptions.
-
CI: on each pull request, build the changed models and everything after them, with tests, against a copy of the data (
dbt build --select state:modified+with the production manifest). A failing test blocks the merge. -
Source freshness:
dbt source freshnesscompares the newestloaded_atwith the limits insources.yml:
yaml
loaded_at_field: loaded_at
freshness:
warn_after: {count: 30, period: hour}
error_after: {count: 54, period: hour}
Run it in the scheduled job (DT3 Lesson 4) so stale data alerts someone.
-
Contracts and versions: for models many teams depend on, dbt can enforce column names and types (model contracts) and keep old versions while users move.
-
Secrets: connection details and salts come from environment variables, never from committed files.
📝A worked example
Input: in the solution, after loading the data, run dbt source freshness --profiles-dir ..
Expected output: PASS freshness of clean.order_lines, because loaded_at is from the load you just ran. If the pipeline stopped for more
than 30 hours, the same command would warn, and after 54 hours it would fail. Project 2 test 8 checks the freshness settings and runs this
command.
🌍Where it's used
- Analytics engineering teams: CI on every pull request to the dbt project.
- Data platforms: freshness checks feeding alerts and "data delayed" banners.
- Large organisations: contracts on shared models used by several teams.
📝Case scenario
An analyst at a quick-commerce company edited a production model directly to fix an urgent number. The fix broke two other marts, and nobody knew until morning. With development schemas and CI on pull requests, the breakage would have shown as a failed check before anything changed in production.
🖱️Try it
Write the sources.yml freshness settings you'd use for SabziSathi if files arrived every hour instead of once a day, and explain your
choice. Success: your warning and error times fit the new arrival rate with some slack.
⚠️Common mistakes
- Editing production directly. Sign: fixes break other models overnight. Fix: changes through pull requests and CI.
- Freshness configured but never run. Sign: stale data with no alert. Fix: run
dbt source freshnessin the scheduled job. - Committed credentials. Sign: a password in
profiles.ymlin Git. Fix:env_var()and a secret store.
🛠️For your project
Project 2 test 8 checks the freshness settings on order_lines; your refresh.yml from Project 1 runs freshness after dbt build.
✅Quick check
What does CI on a dbt project do?
Show the answer
Builds and tests the changed models (and what depends on them) before a change is merged.
What does dbt source freshness compare?
Show the answer
The newest loaded_at time with the warning and error limits.