DT3: Building reliable pipelines
Scheduling and orchestration
🎯You'll be able to
Schedule a pipeline with cron or GitHub Actions, explain what an orchestrator such as Airflow or Prefect adds, and set retries, timeouts and limits on running at the same time.
🏠The everyday picture
A railway timetable says when each train leaves; the station master also makes sure two trains don't use one platform at once, and holds a train if the one before it is late.
📘What it really is
-
cron runs a command on a schedule.
30 0 * * *means 00:30 every day. Schedulers in the cloud usually use UTC: 00:30 UTC is 06:00 in Pune. -
GitHub Actions can run on a schedule (
on: schedule: - cron:), which SabziSathi'srefresh.ymluses. GitHub notes scheduled runs can start late at busy times, so check freshness instead of assuming the time. -
An orchestrator runs many steps with dependencies:
- Airflow: a DAG (graph of tasks) written in Python; retries, back-fills ("catchup"), alerts and a web interface; widely used.
- Prefect, Dagster: similar ideas with lighter set-up; Dagster centres on the data assets each step produces.
-
Cloud versions: Cloud Composer (Airflow on Google Cloud), Amazon MWAA.
-
Settings every scheduled job needs:
- order: load → build and test the warehouse → publish the dashboard, each only if the one before passed;
- retries with a wait, for brief network errors, but not for bad data;
- a timeout, so a stuck run doesn't block the next one;
- concurrency limits, so two runs don't load into the same warehouse at once (
concurrency: group:in GitHub Actions); -
least privilege:
permissions: contents: readand secrets from the secret store, not printed. -
Orchestrators were not run in the kit (not verified here); the kit uses a workflow file that the tests read.
📝A worked example
Input: the solution's .github/workflows/refresh.yml.
Expected output: a daily schedule (30 0 * * *), concurrency: group: refresh with cancel-in-progress: false, read-only permissions, a
20-minute timeout, and steps in the order load → dbt build → freshness → dashboard. Project 1 test 10 checks these rules; it doesn't
run the workflow on GitHub (not verified).
🌍Where it's used
- Small teams: cron or GitHub Actions for a few daily jobs.
- Data platforms: Airflow or Dagster for hundreds of jobs with dependencies.
- Cloud warehouses: built-in schedulers such as BigQuery scheduled queries and Snowflake tasks.
📝Case scenario
An e-commerce company's dashboard refresh and its nightly load were scheduled independently. On a slow night, the dashboard refreshed before the load finished and showed half a day's sales to the morning meeting. Making the dashboard a step after the load, which runs only if the load and tests pass, fixed it.
🖱️Try it
Write the cron expression for "07:15 every Monday in Pune" in UTC. Then read the starter's refresh.yml and list four things wrong with
it. Success: 45 1 * * 1, and your list includes the trigger, the permissions, concurrency and the step order.
⚠️Common mistakes
- Forgetting the scheduler's time zone. Sign: the job runs five and a half hours off. Fix: convert to UTC and comment it.
- Retrying bad data. Sign: the same failure three times, then an alert. Fix: retry network errors; stop on data errors.
- Independent schedules for dependent steps. Sign: reports built from half-loaded data. Fix: one job with steps in order.
🛠️For your project
Project 1 test 10 reads your refresh.yml and checks the schedule, concurrency, permissions and step order.
✅Quick check
What does cancel-in-progress: false protect?
Show the answer
A running load isn't killed half-way by the next one; the next one waits.
When should a pipeline retry?
Show the answer
On brief errors such as a network timeout, not on bad data.