Scenario & Optimisation Questions

Apache Spark · Interview Prep

Theory

Scenario questions describe a symptom and ask what you would do: "a daily job that took 20 minutes now takes 2 hours", "one task out of 200 never finishes", "executors die with out-of-memory", "the job writes 50,000 tiny files". A strong answer goes: reproduce and look at the Spark UI (slowest job, slowest stage, max vs median task, spill, GC), find the cause, apply a targeted fix, and verify with the same metrics.

Typical cause-fix pairs: skew (one huge task) -> AQE skew join, broadcast the small side, salting; too much data read -> filter early, select only needed columns, partition pruning, Parquet or Delta; big shuffle on a join with a small table -> broadcast join; spill or OOM -> more, smaller partitions, avoid collect() and huge collect_list groups; tiny files -> coalesce or repartition before writing; slow Python -> replace UDFs with built-ins or pandas UDFs; repeated computation -> cache a reused DataFrame.

Why we use it

Senior interviews focus on troubleshooting because real jobs fail in production in exactly these ways. Showing a structured diagnose-fix-verify approach proves you can own pipelines, control cloud cost and avoid guessing with random configuration changes.

When to use it

Use this approach for any 'job is slow / failing / expensive' question, and in system design rounds when asked how you would scale a pipeline. Always mention measuring first (Spark UI, explain()) and changing one thing at a time.

How it works

  1. Symptom: slow, failing or expensive job
  2. Open the Spark UI: slowest job, then slowest stage
  3. Is one task far slower than the median? Yes: Skew: AQE skew join, broadcast, salting. No: Check spill, GC, input size and task count.
  4. Check the plan: shuffles, join strategy, scanned columns and files
  5. Apply one targeted fix
  6. Re-run and compare the same metrics

Syntax

orders.join(F.broadcast(customers), "customer_id")      # avoid shuffling the big side
df.filter(...).select(needed_cols)                        # read less, early
df.repartition(n, "key") / df.coalesce(n)                 # fix partition sizes
spark.conf.set("spark.sql.shuffle.partitions", "400")
df.explain("formatted")                                   # verify the plan changed

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

customers = spark.createDataFrame([
    (101, "Asha",  "Jaipur", "Retail"),
    (102, "Ravi",  "Pune",   "Corporate"),
    (103, "Meera", "Delhi",  "Retail"),
    (104, "Karan", "Jaipur", "SMB"),
    (105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")

# Scenario: "join of a big orders table with a small customers table is slow"
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")      # pretend stats are missing
slow = orders.join(customers, "customer_id").groupBy("city").agg(F.sum("amount").alias("sales"))
print("Before:"); slow.explain()                                   # SortMergeJoin + 2 Exchanges

fast = (orders
        .filter(F.col("order_date") >= "2026-01-06")               # 1. filter early
        .select("customer_id", "amount")                           # 2. keep only needed columns
        .join(F.broadcast(customers.select("customer_id", "city")), "customer_id")  # 3. broadcast
        .groupBy("city").agg(F.sum("amount").alias("sales")))
print("After:"); fast.explain()                                    # BroadcastHashJoin, no join shuffle
fast.orderBy("city").show()

spark.conf.unset("spark.sql.autoBroadcastJoinThreshold")

# Scenario: "the job writes thousands of tiny files"
many = orders.repartition(8)
print("partitions before write:", many.rdd.getNumPartitions(), "-> after coalesce:", many.coalesce(1).rdd.getNumPartitions())

# Output:
# Before:
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[city#10], functions=[sum(amount#5)])
#    +- Exchange hashpartitioning(city#10, 200), ENSURE_REQUIREMENTS, [plan_id=48]
#       +- HashAggregate(keys=[city#10], functions=[partial_sum(amount#5)])
#          +- Project [amount#5, city#10]
#             +- SortMergeJoin [customer_id#1], [customer_id#8], Inner
#                :- Sort [customer_id#1 ASC NULLS FIRST], false, 0
#                :  +- Exchange hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=40]
#                :     +- Project [customer_id#1, amount#5]
#                :        +- Filter isnotnull(customer_id#1)
#                :           +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#                +- Sort [customer_id#8 ASC NULLS FIRST], false, 0
#                   +- Exchange hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=41]
#                      +- Project [customer_id#8, city#10]
#                         +- Filter isnotnull(customer_id#8)
#                            +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# After:
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[city#10], functions=[sum(amount#5)])
#    +- Exchange hashpartitioning(city#10, 200), ENSURE_REQUIREMENTS, [plan_id=94]
#       +- HashAggregate(keys=[city#10], functions=[partial_sum(amount#5)])
#          +- Project [amount#5, city#10]
#             +- BroadcastHashJoin [customer_id#1], [customer_id#8], Inner, BuildRight, false
#                :- Project [customer_id#1, amount#5]
#                :  +- Filter ((isnotnull(order_date#6) AND (cast(order_date#6 as date) >= 2026-01-06)) AND isnotnull(customer_id#1))
#                :     +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#                +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=89]
#                   +- Project [customer_id#8, city#10]
#                      +- Filter isnotnull(customer_id#8)
#                         +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# +------+-------+
# |  city|  sales|
# +------+-------+
# | Delhi|53000.0|
# |Jaipur|19500.0|
# |Mumbai|60000.0|
# |  Pune|55000.0|
# +------+-------+
#
# partitions before write: 8 -> after coalesce: 1

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic