Data Skew & Salting
Apache Spark · Performance Tuning
Theory
Data skew means a few keys hold far more rows than the rest, for example one region or one big customer making up most orders. Because a shuffle sends all rows of a key to the same task, that task runs for minutes while the others finish in seconds, and it may spill or run out of memory. In the Stages tab of the Spark UI one task shows far more input and time than the median.
Fixes, in order of effort: let AQE's skew join split the oversized partitions (on by default for sort-merge joins); broadcast the small side so no shuffle is needed; filter or handle hot keys such as null separately; or salt the key. Salting adds a random number (0 to N-1) to the hot side's key and replicates the other side N times with every salt value, so one hot key is spread over N tasks. For aggregations, a two-step salt works: aggregate by (key, salt) first, then by key. Salting adds work, so use it only when simpler fixes are not enough.
Why we use it
A single skewed key can make a job many times slower than it should be, or make it fail, no matter how many executors you add. Detecting and fixing skew is one of the most asked Spark optimisation topics in interviews.
When to use it
Suspect skew when one or a few tasks in a stage run far longer than the median. First try AQE skew handling and broadcast joins, and drop or isolate null keys. Use salting for heavily skewed aggregations or joins where the other side is too large to broadcast.
How it works
- One task in a stage is much slower than the rest
- Check key counts: groupBy(key).count() ordered desc
- Is the other join side small enough to broadcast? Yes: Broadcast it: no shuffle, no skew. No: Check AQE skew join is enabled.
- Still skewed after AQE? Yes: Salt the hot key and replicate the other side N times. No: Done.
- Work is spread evenly across tasks
Syntax
salted = big.withColumn("salt", (F.rand() * N).cast("int"))
small_x = small.crossJoin(spark.range(N).withColumnRenamed("id", "salt"))
salted.join(small_x, ["key", "salt"])
# two-step aggregation
df.groupBy("key", "salt").agg(F.sum("v").alias("p")).groupBy("key").agg(F.sum("p"))Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
N = 4 # number of salt buckets
# Skewed orders: region "North" has most rows
orders = (spark.range(10_000)
.withColumn("region", F.when(F.col("id") % 10 < 8, "North")
.when(F.col("id") % 10 == 8, "West").otherwise("South"))
.withColumn("amount", (F.col("id") % 1000).cast("double")))
regions = spark.createDataFrame([("North", "Asha"), ("West", "Ravi"), ("South", "Meera")],
"region STRING, manager STRING")
orders.groupBy("region").count().orderBy(F.desc("count")).show() # see the skew
# 1. Salted join: spread North over N tasks
salted_orders = orders.withColumn("salt", (F.col("id") % N).cast("int")) # F.rand()*N in practice
salted_regions = regions.crossJoin(spark.range(N).withColumnRenamed("id", "salt")
.withColumn("salt", F.col("salt").cast("int")))
joined = salted_orders.join(salted_regions, ["region", "salt"])
joined.groupBy("salt").count().orderBy("salt").show() # work per salt bucket
# 2. Two-step salted aggregation gives the same totals as a plain groupBy
two_step = (salted_orders.groupBy("region", "salt").agg(F.sum("amount").alias("partial"))
.groupBy("region").agg(F.sum("partial").alias("total")))
plain = orders.groupBy("region").agg(F.sum("amount").alias("total"))
two_step.orderBy("region").show()
print("same result:", sorted(two_step.collect()) == sorted(plain.collect()))
# Output:
# +------+-----+
# |region|count|
# +------+-----+
# | North| 8000|
# | South| 1000|
# | West| 1000|
# +------+-----+
#
# +----+-----+
# |salt|count|
# +----+-----+
# | 0| 2500|
# | 1| 2500|
# | 2| 2500|
# | 3| 2500|
# +----+-----+
#
# +------+---------+
# |region| total|
# +------+---------+
# | North|3988000.0|
# | South| 504000.0|
# | West| 503000.0|
# +------+---------+
#
# same result: True
Key points
- Skew = a few keys with most rows; one task does most of the work
- Spot it in the Stages tab: max task time and input far above the median
- Try AQE skew join and broadcast joins before salting
- Salting: random salt on the big side, replicate the small side for every salt value
- Two-step aggregation: group by (key, salt), then by key
- Null join keys often cause skew: filter or handle them separately