Reading the Spark UI

Apache Spark · Performance Tuning

Theory

The Spark UI is the web page every Spark application serves (on the driver at port 4040 by default; Databricks, EMR and Dataproc link to it). Each action creates a job, each job is split into stages at shuffle boundaries, and each stage runs one task per partition. The Stages tab is where most tuning happens: its summary metrics show min, median and max task duration, input, shuffle read/write and spill per task.

The SQL / DataFrame tab shows each query's plan as a graph with real metrics such as rows output per operator, so you can see which join or scan is expensive and whether AQE changed the plan. The Storage tab lists cached DataFrames and how much is in memory or on disk. The Executors tab shows memory, GC time, failed tasks and shuffle totals per executor. The Environment tab shows the effective Spark configuration. Streaming queries get a Structured Streaming tab with input and processing rates.

Why we use it

Performance problems are invisible in code but obvious in the UI: a max task time 50 times the median means skew, large spill means too little memory per task, thousands of tiny tasks mean too many partitions. Reading it turns guesswork into targeted fixes.

When to use it

Open it whenever a job is slower than expected, fails with out-of-memory, or before and after a tuning change. Start from the slowest job, open its slowest stage, compare max vs median task metrics, then check the SQL tab plan for the operator responsible. After an application ends, use the History Server.

How it works

  1. Job is slow or failing
  2. Jobs tab: find the slowest job and open it
  3. Open the slowest stage and read summary metrics
  4. Is max task time far above the median? Yes: Skew: check keys, AQE skew join, salting. No: Check spill, task count and GC time.
  5. SQL / DataFrame tab: find the expensive operator in the plan
  6. Apply one fix and compare in the UI again

Syntax

spark.sparkContext.uiWebUrl            # URL of this app's UI
spark.sparkContext.setJobDescription("daily sales")   # label jobs
df.explain("formatted")                # same plan as the SQL tab
# Tabs: Jobs | Stages | Storage | Environment | Executors | SQL / DataFrame | Structured Streaming

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

customers = spark.createDataFrame([
    (101, "Asha",  "Jaipur", "Retail"),
    (102, "Ravi",  "Pune",   "Corporate"),
    (103, "Meera", "Delhi",  "Retail"),
    (104, "Karan", "Jaipur", "SMB"),
    (105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")

# Where is the UI? (None if spark.ui.enabled=false)
print("Spark UI:", spark.sparkContext.uiWebUrl)

# Give jobs readable names so they are easy to find in the Jobs tab
spark.sparkContext.setJobDescription("TechTown: sales by city")
sales = (orders.join(customers, "customer_id")
               .groupBy("city").agg(F.sum("amount").alias("sales"))
               .orderBy(F.desc("sales")))
sales.show()

# The formatted plan matches the graph in the SQL / DataFrame tab:
# Exchange = shuffle = stage boundary, BroadcastHashJoin vs SortMergeJoin, AQE nodes
sales.explain("formatted")
spark.sparkContext.setJobDescription(None)

# Output:
# Spark UI: http://192.0.2.2:4040
# +------+-------+
# |  city|  sales|
# +------+-------+
# |Jaipur|74500.0|
# |Mumbai|60000.0|
# |  Pune|55000.0|
# | Delhi|53000.0|
# +------+-------+
#
# == Physical Plan ==
# AdaptiveSparkPlan (18)
# +- Sort (17)
#    +- Exchange (16)
#       +- HashAggregate (15)
#          +- Exchange (14)
#             +- HashAggregate (13)
#                +- Project (12)
#                   +- SortMergeJoin Inner (11)
#                      :- Sort (5)
#                      :  +- Exchange (4)
#                      :     +- Project (3)
#                      :        +- Filter (2)
#                      :           +- Scan ExistingRDD (1)
#                      +- Sort (10)
#                         +- Exchange (9)
#                            +- Project (8)
#                               +- Filter (7)
#                                  +- Scan ExistingRDD (6)
#
#
# (1) Scan ExistingRDD
# Output [7]: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6]
# Arguments: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6], MapPartitionsRDD[4] at applySchemaToPythonRDD at NativeMethodAccessorImpl.java:0, ExistingRDD, UnknownPartitioning(0)
#
# (2) Filter
# Input [7]: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6]
# Condition : isnotnull(customer_id#1)
#
# (3) Project
# Output [2]: [customer_id#1, amount#5]
# Input [7]: [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, order_date#6]
#
# (4) Exchange
# Input [2]: [customer_id#1, amount#5]
# Arguments: hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=273]
#
# (5) Sort
# Input [2]: [customer_id#1, amount#5]
# Arguments: [customer_id#1 ASC NULLS FIRST], false, 0
#
# (6) Scan ExistingRDD
# Output [4]: [customer_id#8, name#9, city#10, segment#11]
# Arguments: [customer_id#8, name#9, city#10, segment#11], MapPartitionsRDD[9] at applySchemaToPythonRDD at NativeMethodAccessorImpl.java:0, ExistingRDD, UnknownPartitioning(0)
#
# (7) Filter
# Input [4]: [customer_id#8, name#9, city#10, segment#11]
# Condition : isnotnull(customer_id#8)
#
# (8) Project
# Output [2]: [customer_id#8, city#10]
# Input [4]: [customer_id#8, name#9, city#10, segment#11]
#
# (9) Exchange
# Input [2]: [customer_id#8, city#10]
# Arguments: hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=274]
#
# (10) Sort
# Input [2]: [customer_id#8, city#10]
# Arguments: [customer_id#8 ASC NULLS FIRST], false, 0
#
# (11) SortMergeJoin
# Left keys [1]: [customer_id#1]
# Right keys [1]: [customer_id#8]
# Join type: Inner
# Join condition: None
#
# (12) Project
# Output [2]: [amount#5, city#10]
# Input [4]: [customer_id#1, amount#5, customer_id#8, city#10]
#
# (13) HashAggregate
# Input [2]: [amount#5, city#10]
# Keys [1]: [city#10]
# Functions [1]: [partial_sum(amount#5)]
# Aggregate Attributes [1]: [sum#26]
# Results [2]: [city#10, sum#27]
#
# (14) Exchange
# Input [2]: [city#10, sum#27]
# Arguments: hashpartitioning(city#10, 200), ENSURE_REQUIREMENTS, [plan_id=281]
#
# (15) HashAggregate
# Input [2]: [city#10, sum#27]
# Keys [1]: [city#10]
# Functions [1]: [sum(amount#5)]
# Aggregate Attributes [1]: [sum(amount#5)#23]
# Results [2]: [city#10, sum(amount#5)#23 AS sales#12]
#
# (16) Exchange
# Input [2]: [city#10, sales#12]
# Arguments: rangepartitioning(sales#12 DESC NULLS LAST, 200), ENSURE_REQUIREMENTS, [plan_id=284]
#
# (17) Sort
# Input [2]: [city#10, sales#12]
# Arguments: [sales#12 DESC NULLS LAST], true, 0
#
# (18) AdaptiveSparkPlan
# Output [2]: [city#10, sales#12]
# Arguments: isFinalPlan=false

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic