Top PySpark Interview Questions

Apache Spark · Interview Prep

Theory

Most PySpark interviews for data and analytics engineers come back to a core set of concepts. Lazy evaluation: transformations (select, filter, join) only build a plan; actions (show, count, collect, write) run it. Narrow vs wide transformations: filter and withColumn work within a partition, while groupBy, join, distinct and repartition need a shuffle and start a new stage. Jobs, stages and tasks: one action creates one or more jobs, shuffles split them into stages, and each stage runs one task per partition.

Other favourites: repartition vs coalesce, cache vs persist, broadcast join vs sort-merge join, what AQE does, how to handle skew, groupByKey vs reduceByKey (map-side combine), why Python UDFs are slow, the difference between rank and dense_rank, how a watermark works, and what the Catalyst optimizer and Tungsten do. Good answers are short, give the mechanism and end with when you would use it.

Why we use it

Interviewers use these questions to check that you understand how Spark executes code, not just the API. Clear, mechanism-based answers show you can debug and tune real jobs, which is what senior data engineering roles need.

When to use it

Revise these before any data engineer, analytics engineer or Databricks interview. Practise saying each answer in two or three sentences, then back it with a small demo such as explain() output. For system-design rounds, combine them with the scenario questions in the next topic.

How it works

  1. Interviewer asks a concept question
  2. Define it in one sentence
  3. Explain the mechanism (shuffle, plan, memory, state)
  4. Can you show it? Yes: Point to explain() output or a Spark UI metric. No: Give a concrete data example.
  5. Say when you would and would not use it
  6. Link it to a real project

Syntax

df.explain(True)        # parsed, analysed, optimised and physical plans
df.explain("formatted") # readable physical plan
# Exchange in a plan = shuffle = wide transformation = new stage
spark.sparkContext.defaultParallelism; df.rdd.getNumPartitions()

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

customers = spark.createDataFrame([
    (101, "Asha",  "Jaipur", "Retail"),
    (102, "Ravi",  "Pune",   "Corporate"),
    (103, "Meera", "Delhi",  "Retail"),
    (104, "Karan", "Jaipur", "SMB"),
    (105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")

import time

# Q1. Lazy evaluation: building the plan is instant, the action does the work
start = time.time()
plan = orders.filter(F.col("amount") > 20000).withColumn("gst", F.col("amount") * 0.18)
print(f"transformations took {time.time() - start:.4f}s (nothing ran yet)")
print("action result, rows:", plan.count())

# Q2. Narrow vs wide: only the wide transformation adds an Exchange (shuffle) to the plan
plan.explain()                                   # Filter + Project, no Exchange
wide = orders.groupBy("region").agg(F.sum("amount").alias("sales"))
wide.explain()

# Q3. Broadcast join vs sort-merge join
orders.join(F.broadcast(customers), "customer_id").explain()

# Q4. rank vs dense_rank with ties
from pyspark.sql.window import Window
w = Window.orderBy(F.desc("qty"))
orders.select("order_id", "qty", F.rank().over(w).alias("rank"),
              F.dense_rank().over(w).alias("dense_rank")).show()

# Output:
# transformations took 0.2201s (nothing ran yet)
# action result, rows: 5
# == Physical Plan ==
# *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7, (amount#5 * 0.18) AS gst#12]
# +- *(1) Filter (isnotnull(amount#5) AND (amount#5 > 20000.0))
#    +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[region#2], functions=[sum(amount#5)])
#    +- Exchange hashpartitioning(region#2, 200), ENSURE_REQUIREMENTS, [plan_id=67]
#       +- HashAggregate(keys=[region#2], functions=[partial_sum(amount#5)])
#          +- Project [region#2, amount#5]
#             +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- Project [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11]
#    +- BroadcastHashJoin [customer_id#1], [customer_id#8], Inner, BuildRight, false
#       :- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#       :  +- Filter isnotnull(customer_id#1)
#       :     +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#       +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=97]
#          +- Filter isnotnull(customer_id#8)
#             +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# +--------+---+----+----------+
# |order_id|qty|rank|dense_rank|
# +--------+---+----+----------+
# |       3|  3|   1|         1|
# |       2|  2|   2|         2|
# |       7|  2|   2|         2|
# |       1|  1|   4|         3|
# |       4|  1|   4|         3|
# |       5|  1|   4|         3|
# |       6|  1|   4|         3|
# |       8|  1|   4|         3|
# +--------+---+----+----------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic