Top PySpark Interview Questions
Apache Spark · Interview Prep
Theory
Most PySpark interviews for data and analytics engineers come back to a core set of concepts. Lazy evaluation: transformations (select, filter, join) only build a plan; actions (show, count, collect, write) run it. Narrow vs wide transformations: filter and withColumn work within a partition, while groupBy, join, distinct and repartition need a shuffle and start a new stage. Jobs, stages and tasks: one action creates one or more jobs, shuffles split them into stages, and each stage runs one task per partition.
Other favourites: repartition vs coalesce, cache vs persist, broadcast join vs sort-merge join, what AQE does, how to handle skew, groupByKey vs reduceByKey (map-side combine), why Python UDFs are slow, the difference between rank and dense_rank, how a watermark works, and what the Catalyst optimizer and Tungsten do. Good answers are short, give the mechanism and end with when you would use it.
Why we use it
Interviewers use these questions to check that you understand how Spark executes code, not just the API. Clear, mechanism-based answers show you can debug and tune real jobs, which is what senior data engineering roles need.
When to use it
Revise these before any data engineer, analytics engineer or Databricks interview. Practise saying each answer in two or three sentences, then back it with a small demo such as explain() output. For system-design rounds, combine them with the scenario questions in the next topic.
How it works
- Interviewer asks a concept question
- Define it in one sentence
- Explain the mechanism (shuffle, plan, memory, state)
- Can you show it? Yes: Point to explain() output or a Spark UI metric. No: Give a concrete data example.
- Say when you would and would not use it
- Link it to a real project
Syntax
df.explain(True) # parsed, analysed, optimised and physical plans
df.explain("formatted") # readable physical plan
# Exchange in a plan = shuffle = wide transformation = new stage
spark.sparkContext.defaultParallelism; df.rdd.getNumPartitions()Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
customers = spark.createDataFrame([
(101, "Asha", "Jaipur", "Retail"),
(102, "Ravi", "Pune", "Corporate"),
(103, "Meera", "Delhi", "Retail"),
(104, "Karan", "Jaipur", "SMB"),
(105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")
import time
# Q1. Lazy evaluation: building the plan is instant, the action does the work
start = time.time()
plan = orders.filter(F.col("amount") > 20000).withColumn("gst", F.col("amount") * 0.18)
print(f"transformations took {time.time() - start:.4f}s (nothing ran yet)")
print("action result, rows:", plan.count())
# Q2. Narrow vs wide: only the wide transformation adds an Exchange (shuffle) to the plan
plan.explain() # Filter + Project, no Exchange
wide = orders.groupBy("region").agg(F.sum("amount").alias("sales"))
wide.explain()
# Q3. Broadcast join vs sort-merge join
orders.join(F.broadcast(customers), "customer_id").explain()
# Q4. rank vs dense_rank with ties
from pyspark.sql.window import Window
w = Window.orderBy(F.desc("qty"))
orders.select("order_id", "qty", F.rank().over(w).alias("rank"),
F.dense_rank().over(w).alias("dense_rank")).show()
# Output:
# transformations took 0.2201s (nothing ran yet)
# action result, rows: 5
# == Physical Plan ==
# *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7, (amount#5 * 0.18) AS gst#12]
# +- *(1) Filter (isnotnull(amount#5) AND (amount#5 > 20000.0))
# +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[region#2], functions=[sum(amount#5)])
# +- Exchange hashpartitioning(region#2, 200), ENSURE_REQUIREMENTS, [plan_id=67]
# +- HashAggregate(keys=[region#2], functions=[partial_sum(amount#5)])
# +- Project [region#2, amount#5]
# +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- Project [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11]
# +- BroadcastHashJoin [customer_id#1], [customer_id#8], Inner, BuildRight, false
# :- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# : +- Filter isnotnull(customer_id#1)
# : +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=97]
# +- Filter isnotnull(customer_id#8)
# +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# +--------+---+----+----------+
# |order_id|qty|rank|dense_rank|
# +--------+---+----+----------+
# | 3| 3| 1| 1|
# | 2| 2| 2| 2|
# | 7| 2| 2| 2|
# | 1| 1| 4| 3|
# | 4| 1| 4| 3|
# | 5| 1| 4| 3|
# | 6| 1| 4| 3|
# | 8| 1| 4| 3|
# +--------+---+----+----------+
Key points
- Transformations are lazy; actions trigger jobs
- Wide transformations (groupBy, join, distinct) shuffle and start a new stage
- Action -> job -> stages (split at shuffles) -> tasks (one per partition)
- Broadcast join ships the small table to every executor and avoids a shuffle
- Catalyst optimises the logical plan; Tungsten handles memory and code generation
- Answer with the mechanism plus when you would use it