Dynamic allocation & speculative execution
Apache Spark · Performance Tuning
Theory
With static allocation an application keeps a fixed number of executors for its whole life. Dynamic resource allocation (spark.dynamicAllocation.enabled=true) lets Spark request more executors when tasks are pending and release executors that have been idle for spark.dynamicAllocation.executorIdleTimeout (60s by default), within minExecutors and maxExecutors. Removed executors may hold shuffle files that later stages need, so it needs either an external shuffle service or shuffle tracking (spark.dynamicAllocation.shuffleTracking.enabled), which keeps executors with live shuffle data. It does nothing in local mode.
Speculative execution (spark.speculation=true, off by default) fights stragglers: once a fraction of a stage's tasks have finished (spark.speculation.quantile, 0.9), any task running more than spark.speculation.multiplier (3) times slower than the median gets a duplicate copy on another executor, and whichever finishes first wins. For example, if nine tasks took about 10 seconds, a task still running at 45 seconds gets a backup copy.
Why we use it
Shared clusters run many jobs with changing needs, so dynamic allocation saves cost and frees resources for others. Speculation keeps one slow or unhealthy node from holding up a whole stage. Both are common interview and cluster-tuning topics.
When to use it
Enable dynamic allocation on shared or autoscaling clusters and for jobs whose stages vary a lot in size; keep static allocation for steady streaming jobs or strict SLAs. Enable speculation when slowdowns come from bad nodes, not from data skew, and only if tasks are idempotent (safe to run twice).
How it works
- Stage starts with its pending tasks
- Tasks pending past the backlog timeout? Yes: Request more executors. No: Keep current ones.
- Tasks run; finished ones report their durations
- Task over 3x median after 90% done? Yes: Launch a speculative copy. No: Let it finish.
- First copy to finish wins; the other is killed
- Executor idle past executorIdleTimeout? Yes: Release it. No: Keep it.
- Stage completes with right-sized resources
Syntax
--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.shuffleTracking.enabled=true # or spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.minExecutors=2 --conf spark.dynamicAllocation.maxExecutors=20
--conf spark.dynamicAllocation.executorIdleTimeout=60s
--conf spark.speculation=true
--conf spark.speculation.multiplier=3 --conf spark.speculation.quantile=0.9Example
from pyspark.sql import SparkSession
# These settings are what you pass on YARN / Kubernetes / standalone clusters.
# Dynamic allocation does nothing in local mode, so here we only build and read the config.
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
# --- dynamic resource allocation ---
.config("spark.dynamicAllocation.enabled", "true")
.config("spark.dynamicAllocation.shuffleTracking.enabled", "true") # or the external shuffle service
.config("spark.dynamicAllocation.minExecutors", "2")
.config("spark.dynamicAllocation.maxExecutors", "20")
.config("spark.dynamicAllocation.executorIdleTimeout", "60s")
# --- speculative execution ---
.config("spark.speculation", "true")
.config("spark.speculation.multiplier", "3") # 3x slower than the median task
.config("spark.speculation.quantile", "0.9") # after 90% of tasks have finished
.getOrCreate())
conf = spark.sparkContext.getConf()
for k in ["spark.dynamicAllocation.enabled", "spark.dynamicAllocation.shuffleTracking.enabled",
"spark.dynamicAllocation.minExecutors", "spark.dynamicAllocation.maxExecutors",
"spark.dynamicAllocation.executorIdleTimeout", "spark.speculation",
"spark.speculation.multiplier", "spark.speculation.quantile"]:
print(f"{k} = {conf.get(k)}")
# A tiny simulation of the speculation rule on one stage's task durations (seconds)
durations = {0: 10, 1: 11, 2: 9, 3: 12, 4: 10, 5: 10, 6: 11, 7: 9, 8: 10, 9: 45}
finished = sorted(durations.values())[:9] # 90% of tasks are done
median = finished[len(finished) // 2]
slow = [t for t, d in durations.items() if d > 3 * median]
print("median:", median, "s | speculate tasks:", slow)
# The job itself runs exactly the same; the settings only change how the cluster behaves
print(spark.range(1000).selectExpr("sum(id) AS total").first().total)
# Output:
# spark.dynamicAllocation.enabled = true
# spark.dynamicAllocation.shuffleTracking.enabled = true
# spark.dynamicAllocation.minExecutors = 2
# spark.dynamicAllocation.maxExecutors = 20
# spark.dynamicAllocation.executorIdleTimeout = 60s
# spark.speculation = true
# spark.speculation.multiplier = 3
# spark.speculation.quantile = 0.9
# median: 10 s | speculate tasks: [9]
# 499500
Key points
- Dynamic allocation adds executors for pending tasks, removes idle ones
- Needs an external shuffle service or shuffle tracking to keep shuffle files
- Bounded by minExecutors / maxExecutors; not active in local mode
- Speculation re-launches stragglers; first copy to finish wins
- Speculation does not fix data skew: the copy has the same big partition
- Speculated tasks must be safe to run twice (careful with external writes)