Dynamic allocation & speculative execution

Apache Spark · Performance Tuning

Theory

With static allocation an application keeps a fixed number of executors for its whole life. Dynamic resource allocation (spark.dynamicAllocation.enabled=true) lets Spark request more executors when tasks are pending and release executors that have been idle for spark.dynamicAllocation.executorIdleTimeout (60s by default), within minExecutors and maxExecutors. Removed executors may hold shuffle files that later stages need, so it needs either an external shuffle service or shuffle tracking (spark.dynamicAllocation.shuffleTracking.enabled), which keeps executors with live shuffle data. It does nothing in local mode.

Speculative execution (spark.speculation=true, off by default) fights stragglers: once a fraction of a stage's tasks have finished (spark.speculation.quantile, 0.9), any task running more than spark.speculation.multiplier (3) times slower than the median gets a duplicate copy on another executor, and whichever finishes first wins. For example, if nine tasks took about 10 seconds, a task still running at 45 seconds gets a backup copy.

Why we use it

Shared clusters run many jobs with changing needs, so dynamic allocation saves cost and frees resources for others. Speculation keeps one slow or unhealthy node from holding up a whole stage. Both are common interview and cluster-tuning topics.

When to use it

Enable dynamic allocation on shared or autoscaling clusters and for jobs whose stages vary a lot in size; keep static allocation for steady streaming jobs or strict SLAs. Enable speculation when slowdowns come from bad nodes, not from data skew, and only if tasks are idempotent (safe to run twice).

How it works

  1. Stage starts with its pending tasks
  2. Tasks pending past the backlog timeout? Yes: Request more executors. No: Keep current ones.
  3. Tasks run; finished ones report their durations
  4. Task over 3x median after 90% done? Yes: Launch a speculative copy. No: Let it finish.
  5. First copy to finish wins; the other is killed
  6. Executor idle past executorIdleTimeout? Yes: Release it. No: Keep it.
  7. Stage completes with right-sized resources

Syntax

--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.shuffleTracking.enabled=true   # or spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.minExecutors=2 --conf spark.dynamicAllocation.maxExecutors=20
--conf spark.dynamicAllocation.executorIdleTimeout=60s
--conf spark.speculation=true
--conf spark.speculation.multiplier=3 --conf spark.speculation.quantile=0.9

Example

from pyspark.sql import SparkSession

# These settings are what you pass on YARN / Kubernetes / standalone clusters.
# Dynamic allocation does nothing in local mode, so here we only build and read the config.
spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         # --- dynamic resource allocation ---
         .config("spark.dynamicAllocation.enabled", "true")
         .config("spark.dynamicAllocation.shuffleTracking.enabled", "true")   # or the external shuffle service
         .config("spark.dynamicAllocation.minExecutors", "2")
         .config("spark.dynamicAllocation.maxExecutors", "20")
         .config("spark.dynamicAllocation.executorIdleTimeout", "60s")
         # --- speculative execution ---
         .config("spark.speculation", "true")
         .config("spark.speculation.multiplier", "3")     # 3x slower than the median task
         .config("spark.speculation.quantile", "0.9")     # after 90% of tasks have finished
         .getOrCreate())
conf = spark.sparkContext.getConf()

for k in ["spark.dynamicAllocation.enabled", "spark.dynamicAllocation.shuffleTracking.enabled",
          "spark.dynamicAllocation.minExecutors", "spark.dynamicAllocation.maxExecutors",
          "spark.dynamicAllocation.executorIdleTimeout", "spark.speculation",
          "spark.speculation.multiplier", "spark.speculation.quantile"]:
    print(f"{k} = {conf.get(k)}")

# A tiny simulation of the speculation rule on one stage's task durations (seconds)
durations = {0: 10, 1: 11, 2: 9, 3: 12, 4: 10, 5: 10, 6: 11, 7: 9, 8: 10, 9: 45}
finished = sorted(durations.values())[:9]                  # 90% of tasks are done
median = finished[len(finished) // 2]
slow = [t for t, d in durations.items() if d > 3 * median]
print("median:", median, "s | speculate tasks:", slow)

# The job itself runs exactly the same; the settings only change how the cluster behaves
print(spark.range(1000).selectExpr("sum(id) AS total").first().total)

# Output:
# spark.dynamicAllocation.enabled = true
# spark.dynamicAllocation.shuffleTracking.enabled = true
# spark.dynamicAllocation.minExecutors = 2
# spark.dynamicAllocation.maxExecutors = 20
# spark.dynamicAllocation.executorIdleTimeout = 60s
# spark.speculation = true
# spark.speculation.multiplier = 3
# spark.speculation.quantile = 0.9
# median: 10 s | speculate tasks: [9]
# 499500

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic