DAG scheduler, task scheduler & scheduling pools

Apache Spark · Spark Architecture

Theory

When an action runs, two schedulers inside the driver turn it into work. The DAG scheduler takes the job's lineage, cuts it into stages at every shuffle boundary, skips stages whose shuffle output already exists, and submits each ready stage as a set of tasks, one per partition. The task scheduler then places those tasks on free executor cores, preferring data locality: a task first waits briefly (spark.locality.wait, 3s by default) for a slot on the node or process that already holds its data before falling back to any slot. Failed tasks are retried up to spark.task.maxFailures (4) times.

Between jobs, Spark uses FIFO scheduling by default: the first job gets all the resources it wants. With spark.scheduler.mode=FAIR, jobs submitted from different threads share executors in round-robin fashion, and you can group them into named pools with sc.setLocalProperty('spark.scheduler.pool', 'reports'). For example, one reduceByKey job on a 2-partition RDD into 3 partitions becomes 2 stages: 2 map tasks, then 3 reduce tasks.

Why we use it

Reading stages and tasks in the Spark UI, and explaining why a stage was skipped or a task ran 'ANY' instead of 'PROCESS_LOCAL', requires knowing what the two schedulers do. FAIR pools matter when one application (a notebook server or Thrift server) serves many users.

When to use it

Use this knowledge when reading the Spark UI and tuning shuffles. Switch to FAIR mode only when several jobs run concurrently inside one application, such as a shared notebook cluster or a service. Batch jobs that run one action at a time are fine with FIFO.

How it works

  1. An action (collect, count, write) submits a job
  2. DAG scheduler splits the lineage into stages at shuffles
  3. Is a parent stage's shuffle output already there? Yes: Skip that stage. No: Run the parent stage first.
  4. Each ready stage becomes a task set: one task per partition
  5. Task scheduler places tasks on free cores, trying data-local slots first
  6. Did a task fail? Yes: Retry it, up to spark.task.maxFailures. No: Mark the stage done.
  7. Last stage returns the result to the driver

Syntax

spark.conf / SparkConf: spark.scheduler.mode = FIFO (default) | FAIR
sc.setLocalProperty("spark.scheduler.pool", "reports")
sc.setJobGroup("daily", "daily sales report")
sc.statusTracker().getJobIdsForGroup("daily")
spark.locality.wait = 3s ; spark.task.maxFailures = 4
spark.scheduler.allocation.file = fairscheduler.xml

Example

from pyspark.sql import SparkSession

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .config("spark.scheduler.mode", "FAIR")      # default is FIFO
         .getOrCreate())
sc = spark.sparkContext
print("scheduler mode:", sc.getConf().get("spark.scheduler.mode"))

# Jobs submitted from this thread go to the "reports" pool and the "daily" job group
sc.setLocalProperty("spark.scheduler.pool", "reports")
sc.setJobGroup("daily", "daily sales report")
print("pool for this thread:", sc.getLocalProperty("spark.scheduler.pool"))

orders = sc.parallelize([("North", 55000.0), ("West", 30000.0), ("North", 1500.0),
                         ("South", 52000.0), ("West", 25000.0), ("East", 15000.0)], 2)

# One action = one job. reduceByKey adds a shuffle, so the DAG scheduler cuts 2 stages.
totals = orders.reduceByKey(lambda a, b: a + b, numPartitions=3).collect()
print(sorted(totals))

tracker = sc.statusTracker()
for job_id in sorted(tracker.getJobIdsForGroup("daily")):
    job = tracker.getJobInfo(job_id)
    print("job", job_id, job.status, "| stages:", sorted(job.stageIds))
    for sid in sorted(job.stageIds):
        st = tracker.getStageInfo(sid)
        print("  stage", sid, "tasks:", st.numTasks)   # task scheduler runs one task per partition

print("locality wait:", sc.getConf().get("spark.locality.wait", "3s"))
sc.setLocalProperty("spark.scheduler.pool", None)        # back to the default pool

# Output:
# scheduler mode: FAIR
# pool for this thread: reports
# [('East', 15000.0), ('North', 56500.0), ('South', 52000.0), ('West', 55000.0)]
# job 0 SUCCEEDED | stages: [0, 1]
#   stage 0 tasks: 2
#   stage 1 tasks: 3
# locality wait: 3s

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic