Reading the Spark UI
Apache Spark · Performance Tuning
Theory
The Spark UI is the web page every Spark application serves (on the driver at port 4040 by default; Databricks, EMR and Dataproc link to it). Each action creates a job, each job is split into stages at shuffle boundaries, and each stage runs one task per partition. The Stages tab is where most tuning happens: its summary metrics show min, median and max task duration, input, shuffle read/write and spill per task.
The SQL / DataFrame tab shows each query's plan as a graph with real metrics such as rows output per operator, so you can see which join or scan is expensive and whether AQE changed the plan. The Storage tab lists cached DataFrames and how much is in memory or on disk. The Executors tab shows memory, GC time, failed tasks and shuffle totals per executor. The Environment tab shows the effective Spark configuration. Streaming queries get a Structured Streaming tab with input and processing rates.
Why we use it
Performance problems are invisible in code but obvious in the UI: a max task time 50 times the median means skew, large spill means too little memory per task, thousands of tiny tasks mean too many partitions. Reading it turns guesswork into targeted fixes.
When to use it
Open it whenever a job is slower than expected, fails with out-of-memory, or before and after a tuning change. Start from the slowest job, open its slowest stage, compare max vs median task metrics, then check the SQL tab plan for the operator responsible. After an application ends, use the History Server.
How it works
- Job is slow or failing
- Jobs tab: find the slowest job and open it
- Open the slowest stage and read summary metrics
- Is max task time far above the median? Yes: Skew: check keys, AQE skew join, salting. No: Check spill, task count and GC time.
- SQL / DataFrame tab: find the expensive operator in the plan
- Apply one fix and compare in the UI again
Syntax
spark.sparkContext.uiWebUrl # URL of this app's UI
spark.sparkContext.setJobDescription("daily sales") # label jobs
df.explain("formatted") # same plan as the SQL tab
# Tabs: Jobs | Stages | Storage | Environment | Executors | SQL / DataFrame | Structured StreamingExample
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
customers = spark.createDataFrame([
(101, "Asha", "Jaipur", "Retail"),
(102, "Ravi", "Pune", "Corporate"),
(103, "Meera", "Delhi", "Retail"),
(104, "Karan", "Jaipur", "SMB"),
(105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")
# Where is the UI? (None if spark.ui.enabled=false)
print("Spark UI:", spark.sparkContext.uiWebUrl)
# Give jobs readable names so they are easy to find in the Jobs tab
spark.sparkContext.setJobDescription("TechTown: sales by city")
sales = (orders.join(customers, "customer_id")
.groupBy("city").agg(F.sum("amount").alias("sales"))
.orderBy(F.desc("sales")))
sales.show()
# The formatted plan matches the graph in the SQL / DataFrame tab:
# Exchange = shuffle = stage boundary, BroadcastHashJoin vs SortMergeJoin, AQE nodes
sales.explain("formatted")
spark.sparkContext.setJobDescription(None)
# Output:
# Spark UI: http://192.0.2.2:4040
# +------+-------+
# | city| sales|
# +------+-------+
# |Jaipur|74500.0|
# |Mumbai|60000.0|
# | Pune|55000.0|
# | Delhi|53000.0|
# +------+-------+
#
# == Physical Plan ==
# AdaptiveSparkPlan (18)
# +- Sort (17)
# +- Exchange (16)
# +- HashAggregate (15)
# +- Exchange (14)
# +- HashAggregate (13)
# +- Project (12)
# +- SortMergeJoin Inner (11)
# :- Sort (5)
# : +- Exchange (4)
# : +- Project (3)
# : +- Filter (2)
# : +- Scan ExistingRDD (1)
# +- Sort (10)
# +- Exchange (9)
# +- Project (8)
# +- Filter (7)
# +- Scan ExistingRDD (6)
#
#
# (1) Scan ExistingRDD
# Output [7]: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6]
# Arguments: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6], MapPartitionsRDD[4] at applySchemaToPythonRDD at NativeMethodAccessorImpl.java:0, ExistingRDD, UnknownPartitioning(0)
#
# (2) Filter
# Input [7]: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6]
# Condition : isnotnull(customer_id#1)
#
# (3) Project
# Output [2]: [customer_id#1, amount#5]
# Input [7]: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6]
#
# (4) Exchange
# Input [2]: [customer_id#1, amount#5]
# Arguments: hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=273]
#
# (5) Sort
# Input [2]: [customer_id#1, amount#5]
# Arguments: [customer_id#1 ASC NULLS FIRST], false, 0
#
# (6) Scan ExistingRDD
# Output [4]: [customer_id#8, name#9, city#10, segment#11]
# Arguments: [customer_id#8, name#9, city#10, segment#11], MapPartitionsRDD[9] at applySchemaToPythonRDD at NativeMethodAccessorImpl.java:0, ExistingRDD, UnknownPartitioning(0)
#
# (7) Filter
# Input [4]: [customer_id#8, name#9, city#10, segment#11]
# Condition : isnotnull(customer_id#8)
#
# (8) Project
# Output [2]: [customer_id#8, city#10]
# Input [4]: [customer_id#8, name#9, city#10, segment#11]
#
# (9) Exchange
# Input [2]: [customer_id#8, city#10]
# Arguments: hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=274]
#
# (10) Sort
# Input [2]: [customer_id#8, city#10]
# Arguments: [customer_id#8 ASC NULLS FIRST], false, 0
#
# (11) SortMergeJoin
# Left keys [1]: [customer_id#1]
# Right keys [1]: [customer_id#8]
# Join type: Inner
# Join condition: None
#
# (12) Project
# Output [2]: [amount#5, city#10]
# Input [4]: [customer_id#1, amount#5, customer_id#8, city#10]
#
# (13) HashAggregate
# Input [2]: [amount#5, city#10]
# Keys [1]: [city#10]
# Functions [1]: [partial_sum(amount#5)]
# Aggregate Attributes [1]: [sum#26]
# Results [2]: [city#10, sum#27]
#
# (14) Exchange
# Input [2]: [city#10, sum#27]
# Arguments: hashpartitioning(city#10, 200), ENSURE_REQUIREMENTS, [plan_id=281]
#
# (15) HashAggregate
# Input [2]: [city#10, sum#27]
# Keys [1]: [city#10]
# Functions [1]: [sum(amount#5)]
# Aggregate Attributes [1]: [sum(amount#5)#23]
# Results [2]: [city#10, sum(amount#5)#23 AS sales#12]
#
# (16) Exchange
# Input [2]: [city#10, sales#12]
# Arguments: rangepartitioning(sales#12 DESC NULLS LAST, 200), ENSURE_REQUIREMENTS, [plan_id=284]
#
# (17) Sort
# Input [2]: [city#10, sales#12]
# Arguments: [sales#12 DESC NULLS LAST], true, 0
#
# (18) AdaptiveSparkPlan
# Output [2]: [city#10, sales#12]
# Arguments: isFinalPlan=false
Key points
- Action -> job; shuffle boundaries split a job into stages; one task per partition
- Stages tab: compare max vs median task time and input to spot skew
- Spill (memory/disk) in stage metrics means tasks lack memory: more partitions or memory
- SQL / DataFrame tab shows the physical plan with real row counts per operator
- Storage tab confirms caching; Environment tab shows effective configs
- Use setJobDescription to label jobs; use the History Server after the app ends