Data Skew & Salting

Apache Spark · Performance Tuning

Theory

Data skew means a few keys hold far more rows than the rest, for example one region or one big customer making up most orders. Because a shuffle sends all rows of a key to the same task, that task runs for minutes while the others finish in seconds, and it may spill or run out of memory. In the Stages tab of the Spark UI one task shows far more input and time than the median.

Fixes, in order of effort: let AQE's skew join split the oversized partitions (on by default for sort-merge joins); broadcast the small side so no shuffle is needed; filter or handle hot keys such as null separately; or salt the key. Salting adds a random number (0 to N-1) to the hot side's key and replicates the other side N times with every salt value, so one hot key is spread over N tasks. For aggregations, a two-step salt works: aggregate by (key, salt) first, then by key. Salting adds work, so use it only when simpler fixes are not enough.

Why we use it

A single skewed key can make a job many times slower than it should be, or make it fail, no matter how many executors you add. Detecting and fixing skew is one of the most asked Spark optimisation topics in interviews.

When to use it

Suspect skew when one or a few tasks in a stage run far longer than the median. First try AQE skew handling and broadcast joins, and drop or isolate null keys. Use salting for heavily skewed aggregations or joins where the other side is too large to broadcast.

How it works

  1. One task in a stage is much slower than the rest
  2. Check key counts: groupBy(key).count() ordered desc
  3. Is the other join side small enough to broadcast? Yes: Broadcast it: no shuffle, no skew. No: Check AQE skew join is enabled.
  4. Still skewed after AQE? Yes: Salt the hot key and replicate the other side N times. No: Done.
  5. Work is spread evenly across tasks

Syntax

salted = big.withColumn("salt", (F.rand() * N).cast("int"))
small_x = small.crossJoin(spark.range(N).withColumnRenamed("id", "salt"))
salted.join(small_x, ["key", "salt"])
# two-step aggregation
df.groupBy("key", "salt").agg(F.sum("v").alias("p")).groupBy("key").agg(F.sum("p"))

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

N = 4   # number of salt buckets

# Skewed orders: region "North" has most rows
orders = (spark.range(10_000)
    .withColumn("region", F.when(F.col("id") % 10 < 8, "North")
                           .when(F.col("id") % 10 == 8, "West").otherwise("South"))
    .withColumn("amount", (F.col("id") % 1000).cast("double")))
regions = spark.createDataFrame([("North", "Asha"), ("West", "Ravi"), ("South", "Meera")],
                                "region STRING, manager STRING")

orders.groupBy("region").count().orderBy(F.desc("count")).show()   # see the skew

# 1. Salted join: spread North over N tasks
salted_orders = orders.withColumn("salt", (F.col("id") % N).cast("int"))   # F.rand()*N in practice
salted_regions = regions.crossJoin(spark.range(N).withColumnRenamed("id", "salt")
                                        .withColumn("salt", F.col("salt").cast("int")))
joined = salted_orders.join(salted_regions, ["region", "salt"])
joined.groupBy("salt").count().orderBy("salt").show()                      # work per salt bucket

# 2. Two-step salted aggregation gives the same totals as a plain groupBy
two_step = (salted_orders.groupBy("region", "salt").agg(F.sum("amount").alias("partial"))
                         .groupBy("region").agg(F.sum("partial").alias("total")))
plain = orders.groupBy("region").agg(F.sum("amount").alias("total"))
two_step.orderBy("region").show()
print("same result:", sorted(two_step.collect()) == sorted(plain.collect()))

# Output:
# +------+-----+
# |region|count|
# +------+-----+
# | North| 8000|
# | South| 1000|
# |  West| 1000|
# +------+-----+
#
# +----+-----+
# |salt|count|
# +----+-----+
# |   0| 2500|
# |   1| 2500|
# |   2| 2500|
# |   3| 2500|
# +----+-----+
#
# +------+---------+
# |region|    total|
# +------+---------+
# | North|3988000.0|
# | South| 504000.0|
# |  West| 503000.0|
# +------+---------+
#
# same result: True

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic