DAG scheduler, task scheduler & scheduling pools
Apache Spark · Spark Architecture
Theory
When an action runs, two schedulers inside the driver turn it into work. The DAG scheduler takes the job's lineage, cuts it into stages at every shuffle boundary, skips stages whose shuffle output already exists, and submits each ready stage as a set of tasks, one per partition. The task scheduler then places those tasks on free executor cores, preferring data locality: a task first waits briefly (spark.locality.wait, 3s by default) for a slot on the node or process that already holds its data before falling back to any slot. Failed tasks are retried up to spark.task.maxFailures (4) times.
Between jobs, Spark uses FIFO scheduling by default: the first job gets all the resources it wants. With spark.scheduler.mode=FAIR, jobs submitted from different threads share executors in round-robin fashion, and you can group them into named pools with sc.setLocalProperty('spark.scheduler.pool', 'reports'). For example, one reduceByKey job on a 2-partition RDD into 3 partitions becomes 2 stages: 2 map tasks, then 3 reduce tasks.
Why we use it
Reading stages and tasks in the Spark UI, and explaining why a stage was skipped or a task ran 'ANY' instead of 'PROCESS_LOCAL', requires knowing what the two schedulers do. FAIR pools matter when one application (a notebook server or Thrift server) serves many users.
When to use it
Use this knowledge when reading the Spark UI and tuning shuffles. Switch to FAIR mode only when several jobs run concurrently inside one application, such as a shared notebook cluster or a service. Batch jobs that run one action at a time are fine with FIFO.
How it works
- An action (collect, count, write) submits a job
- DAG scheduler splits the lineage into stages at shuffles
- Is a parent stage's shuffle output already there? Yes: Skip that stage. No: Run the parent stage first.
- Each ready stage becomes a task set: one task per partition
- Task scheduler places tasks on free cores, trying data-local slots first
- Did a task fail? Yes: Retry it, up to spark.task.maxFailures. No: Mark the stage done.
- Last stage returns the result to the driver
Syntax
spark.conf / SparkConf: spark.scheduler.mode = FIFO (default) | FAIR
sc.setLocalProperty("spark.scheduler.pool", "reports")
sc.setJobGroup("daily", "daily sales report")
sc.statusTracker().getJobIdsForGroup("daily")
spark.locality.wait = 3s ; spark.task.maxFailures = 4
spark.scheduler.allocation.file = fairscheduler.xmlExample
from pyspark.sql import SparkSession
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.config("spark.scheduler.mode", "FAIR") # default is FIFO
.getOrCreate())
sc = spark.sparkContext
print("scheduler mode:", sc.getConf().get("spark.scheduler.mode"))
# Jobs submitted from this thread go to the "reports" pool and the "daily" job group
sc.setLocalProperty("spark.scheduler.pool", "reports")
sc.setJobGroup("daily", "daily sales report")
print("pool for this thread:", sc.getLocalProperty("spark.scheduler.pool"))
orders = sc.parallelize([("North", 55000.0), ("West", 30000.0), ("North", 1500.0),
("South", 52000.0), ("West", 25000.0), ("East", 15000.0)], 2)
# One action = one job. reduceByKey adds a shuffle, so the DAG scheduler cuts 2 stages.
totals = orders.reduceByKey(lambda a, b: a + b, numPartitions=3).collect()
print(sorted(totals))
tracker = sc.statusTracker()
for job_id in sorted(tracker.getJobIdsForGroup("daily")):
job = tracker.getJobInfo(job_id)
print("job", job_id, job.status, "| stages:", sorted(job.stageIds))
for sid in sorted(job.stageIds):
st = tracker.getStageInfo(sid)
print(" stage", sid, "tasks:", st.numTasks) # task scheduler runs one task per partition
print("locality wait:", sc.getConf().get("spark.locality.wait", "3s"))
sc.setLocalProperty("spark.scheduler.pool", None) # back to the default pool
# Output:
# scheduler mode: FAIR
# pool for this thread: reports
# [('East', 15000.0), ('North', 56500.0), ('South', 52000.0), ('West', 55000.0)]
# job 0 SUCCEEDED | stages: [0, 1]
# stage 0 tasks: 2
# stage 1 tasks: 3
# locality wait: 3s
Key points
- DAG scheduler: job -> stages (cut at shuffles), handles stage retries
- Task scheduler: stage -> tasks on executor cores, with locality preference
- Locality levels: PROCESS_LOCAL > NODE_LOCAL > RACK_LOCAL > ANY
- Stages whose shuffle files already exist are skipped on reuse
- FIFO by default; FAIR shares resources between concurrent jobs
- Pools are set per thread with setLocalProperty('spark.scheduler.pool', ...)