Task DAGs & Orchestration
Snowflake · Views, Pipelines & Automation
Theory
A task graph, often called a DAG (directed acyclic graph), links tasks so that finishing one starts the next. Only the root task has a SCHEDULE; every child task declares AFTER one or more predecessor tasks and runs when all of them finish successfully. For example, dag_extract runs at 2 AM, then dag_transform runs AFTER dag_extract, and dag_load runs AFTER dag_transform.
Each task in the graph must be resumed for it to run. SYSTEM$TASK_DEPENDENTS_ENABLE('root_task') resumes the root and all its children in one call. To change a child task you suspend the root first. A finalizer task can be added to run after the whole graph, for example for cleanup or notifications.
Why we use it
Real pipelines have several steps that must run in order. A task graph expresses those dependencies inside Snowflake, so a step never runs on half-finished data from the step before.
When to use it
Use task graphs for multi-step SQL pipelines inside Snowflake. Use an external orchestrator when steps span other systems or need complex retries and alerts across tools, and dynamic tables when dependencies can be declared as queries.
Building a DAG
Only the root task carries a SCHEDULE; every child task instead declares AFTER <parent_task>, so it runs the moment its parent finishes successfully rather than on its own clock.
The Extract → Transform → Load → Validate Flow
A common pattern chains four steps end to end: task_extract (SCHEDULE) │ AFTER ▼ task_transform │ AFTER ▼ task_validate Each step only starts once the previous one has actually completed.
Resuming a DAG
Tasks are created SUSPENDED. Resume leaf tasks first and work back up to the root — resuming the root before its children exist as active tasks means they'd never get triggered; suspending the root task pauses the entire DAG downstream of it.
How it works
- Root task schedule fires
- Root task runs (extract)
- Did all predecessors of the next task succeed? Yes: Run the child task. No: Stop this branch; run is marked failed.
- Children run in dependency order (transform, load, validate)
- Finalizer task, if defined, runs at the end
Example
CREATE OR REPLACE TASK dag_extract
WAREHOUSE = etl_wh SCHEDULE = 'USING CRON 0 2 * * * UTC'
AS INSERT INTO staging_customers (raw_data, loaded_at) SELECT event_data, CURRENT_TIMESTAMP() FROM raw_events;
CREATE OR REPLACE TASK dag_transform
WAREHOUSE = etl_wh AFTER dag_extract
AS INSERT INTO customers (first_name, email)
SELECT raw_data:first_name::STRING, raw_data:email::STRING FROM staging_customers;
CREATE OR REPLACE TASK dag_validate
WAREHOUSE = etl_wh AFTER dag_transform
AS INSERT INTO temp_analysis (metric, value) SELECT 'new_customers', COUNT(*) FROM customers;
ALTER TASK dag_validate RESUME;
ALTER TASK dag_transform RESUME;
ALTER TASK dag_extract RESUME;Key points
- Only the root task has a SCHEDULE; children use AFTER
- A child can have several predecessors and waits for all of them
- SYSTEM$TASK_DEPENDENTS_ENABLE resumes the root and all children
- Suspend the root before changing tasks in the graph
- Suspending the root pauses everything downstream of it