Scenario & Optimisation Questions
Apache Spark · Interview Prep
Theory
Scenario questions describe a symptom and ask what you would do: "a daily job that took 20 minutes now takes 2 hours", "one task out of 200 never finishes", "executors die with out-of-memory", "the job writes 50,000 tiny files". A strong answer goes: reproduce and look at the Spark UI (slowest job, slowest stage, max vs median task, spill, GC), find the cause, apply a targeted fix, and verify with the same metrics.
Typical cause-fix pairs: skew (one huge task) -> AQE skew join, broadcast the small side, salting; too much data read -> filter early, select only needed columns, partition pruning, Parquet or Delta; big shuffle on a join with a small table -> broadcast join; spill or OOM -> more, smaller partitions, avoid collect() and huge collect_list groups; tiny files -> coalesce or repartition before writing; slow Python -> replace UDFs with built-ins or pandas UDFs; repeated computation -> cache a reused DataFrame.
Why we use it
Senior interviews focus on troubleshooting because real jobs fail in production in exactly these ways. Showing a structured diagnose-fix-verify approach proves you can own pipelines, control cloud cost and avoid guessing with random configuration changes.
When to use it
Use this approach for any 'job is slow / failing / expensive' question, and in system design rounds when asked how you would scale a pipeline. Always mention measuring first (Spark UI, explain()) and changing one thing at a time.
How it works
- Symptom: slow, failing or expensive job
- Open the Spark UI: slowest job, then slowest stage
- Is one task far slower than the median? Yes: Skew: AQE skew join, broadcast, salting. No: Check spill, GC, input size and task count.
- Check the plan: shuffles, join strategy, scanned columns and files
- Apply one targeted fix
- Re-run and compare the same metrics
Syntax
orders.join(F.broadcast(customers), "customer_id") # avoid shuffling the big side
df.filter(...).select(needed_cols) # read less, early
df.repartition(n, "key") / df.coalesce(n) # fix partition sizes
spark.conf.set("spark.sql.shuffle.partitions", "400")
df.explain("formatted") # verify the plan changedExample
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
customers = spark.createDataFrame([
(101, "Asha", "Jaipur", "Retail"),
(102, "Ravi", "Pune", "Corporate"),
(103, "Meera", "Delhi", "Retail"),
(104, "Karan", "Jaipur", "SMB"),
(105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")
# Scenario: "join of a big orders table with a small customers table is slow"
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1") # pretend stats are missing
slow = orders.join(customers, "customer_id").groupBy("city").agg(F.sum("amount").alias("sales"))
print("Before:"); slow.explain() # SortMergeJoin + 2 Exchanges
fast = (orders
.filter(F.col("order_date") >= "2026-01-06") # 1. filter early
.select("customer_id", "amount") # 2. keep only needed columns
.join(F.broadcast(customers.select("customer_id", "city")), "customer_id") # 3. broadcast
.groupBy("city").agg(F.sum("amount").alias("sales")))
print("After:"); fast.explain() # BroadcastHashJoin, no join shuffle
fast.orderBy("city").show()
spark.conf.unset("spark.sql.autoBroadcastJoinThreshold")
# Scenario: "the job writes thousands of tiny files"
many = orders.repartition(8)
print("partitions before write:", many.rdd.getNumPartitions(), "-> after coalesce:", many.coalesce(1).rdd.getNumPartitions())
# Output:
# Before:
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[city#10], functions=[sum(amount#5)])
# +- Exchange hashpartitioning(city#10, 200), ENSURE_REQUIREMENTS, [plan_id=48]
# +- HashAggregate(keys=[city#10], functions=[partial_sum(amount#5)])
# +- Project [amount#5, city#10]
# +- SortMergeJoin [customer_id#1], [customer_id#8], Inner
# :- Sort [customer_id#1 ASC NULLS FIRST], false, 0
# : +- Exchange hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=40]
# : +- Project [customer_id#1, amount#5]
# : +- Filter isnotnull(customer_id#1)
# : +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- Sort [customer_id#8 ASC NULLS FIRST], false, 0
# +- Exchange hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=41]
# +- Project [customer_id#8, city#10]
# +- Filter isnotnull(customer_id#8)
# +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# After:
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[city#10], functions=[sum(amount#5)])
# +- Exchange hashpartitioning(city#10, 200), ENSURE_REQUIREMENTS, [plan_id=94]
# +- HashAggregate(keys=[city#10], functions=[partial_sum(amount#5)])
# +- Project [amount#5, city#10]
# +- BroadcastHashJoin [customer_id#1], [customer_id#8], Inner, BuildRight, false
# :- Project [customer_id#1, amount#5]
# : +- Filter ((isnotnull(order_date#6) AND (cast(order_date#6 as date) >= 2026-01-06)) AND isnotnull(customer_id#1))
# : +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=89]
# +- Project [customer_id#8, city#10]
# +- Filter isnotnull(customer_id#8)
# +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# +------+-------+
# | city| sales|
# +------+-------+
# | Delhi|53000.0|
# |Jaipur|19500.0|
# |Mumbai|60000.0|
# | Pune|55000.0|
# +------+-------+
#
# partitions before write: 8 -> after coalesce: 1
Key points
- Start with evidence: Spark UI stages, task metrics and explain()
- One giant task = skew; many tiny tasks = too many partitions
- Spill and OOM: smaller partitions, no collect(), avoid huge groups
- Broadcast small tables; filter and select early to read less
- Tiny files: coalesce/repartition before write or compact the table
- Change one thing at a time and re-measure