groupByKey vs reduceByKey vs aggregateByKey vs combineByKey
Apache Spark · RDDs
Theory
All four group a pair RDD by key, but they move very different amounts of data. groupByKey shuffles every single (key, value) record and builds a full list of values per key on the reducer, which is slow and can run out of memory for a hot key. reduceByKey(f) first combines values inside each partition (a map-side combine), so each partition sends at most one partial result per key; f must take and return the value type, such as a + b.
aggregateByKey(zero, seqOp, combOp) also combines map-side but lets the result type differ from the value type: start from a zero value such as (0.0, 0), add each value with seqOp, and merge partial results across partitions with combOp. combineByKey(createCombiner, mergeValue, mergeCombiners) is the most general form, where the first value seen for a key creates the accumulator. For example, to get the average order per region you keep (sum, count) per key: North (104500, 3) gives 34833.3.
Why we use it
Choosing groupByKey when a combining operator would work is a classic cause of slow jobs and executor out-of-memory errors, and the four-way comparison is a very common interview question. The DataFrame groupBy().agg() already does the efficient version for you.
When to use it
Use reduceByKey for sums, counts, min and max. Use aggregateByKey or combineByKey when the result has a different shape, such as (sum, count) for averages or sets of distinct values. Use groupByKey only when you truly need every value per key, such as sorting all events of a user. In DataFrame code use groupBy().agg().
How it works
- Need one result per key from a pair RDD
- Need every value of each key? Yes: groupByKey (shuffles all). No: Use a combining operator.
- Result same type as value (sum, max)? Yes: reduceByKey. No: aggregateByKey / combineByKey.
- Combine values inside each partition first
- Shuffle only the partial results per key
- Merge partials into one value per key
Syntax
pairs.groupByKey().mapValues(sum)
pairs.reduceByKey(lambda a, b: a + b)
pairs.aggregateByKey((0.0, 0),
lambda acc, v: (acc[0] + v, acc[1] + 1), # seqOp (in a partition)
lambda a, b: (a[0] + b[0], a[1] + b[1])) # combOp (across partitions)
pairs.combineByKey(lambda v: (v, 1), mergeValue, mergeCombiners)Example
from pyspark.sql import SparkSession
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.getOrCreate())
sc = spark.sparkContext
sales = sc.parallelize([("North", 55000.0), ("West", 30000.0), ("North", 1500.0),
("South", 52000.0), ("West", 25000.0), ("North", 48000.0)], 2)
# 1) groupByKey: ships EVERY value across the network, then you aggregate
g = sales.groupByKey().mapValues(lambda vals: sum(vals))
print("groupByKey :", sorted(g.collect()))
# 2) reduceByKey: combines inside each partition first (map-side combine)
r = sales.reduceByKey(lambda a, b: a + b)
print("reduceByKey :", sorted(r.collect()))
# 3) aggregateByKey: zero value + within-partition fn + across-partition fn.
# The result type (sum, count) differs from the value type (float).
sc_pair = sales.aggregateByKey((0.0, 0),
lambda acc, v: (acc[0] + v, acc[1] + 1), # seqOp
lambda a, b: (a[0] + b[0], a[1] + b[1])) # combOp
print("aggregateByKey:", sorted(sc_pair.collect()))
# 4) combineByKey: like aggregateByKey, but the first value creates the accumulator
avg = (sales.combineByKey(lambda v: (v, 1), # createCombiner
lambda acc, v: (acc[0] + v, acc[1] + 1), # mergeValue
lambda a, b: (a[0] + b[0], a[1] + b[1])) # mergeCombiners
.mapValues(lambda t: round(t[0] / t[1], 1)))
print("combineByKey avg:", sorted(avg.collect()))
# How many records each partition would send to the shuffle
print("records per partition, no combine:", sales.glom().map(len).collect())
print("keys per partition after combine :", sales.mapPartitions(lambda it: [len({k for k, _ in it})]).collect())
# Output:
# groupByKey : [('North', 104500.0), ('South', 52000.0), ('West', 55000.0)]
# reduceByKey : [('North', 104500.0), ('South', 52000.0), ('West', 55000.0)]
# aggregateByKey: [('North', (104500.0, 3)), ('South', (52000.0, 1)), ('West', (55000.0, 2))]
# combineByKey avg: [('North', 34833.3), ('South', 52000.0), ('West', 27500.0)]
# records per partition, no combine: [3, 3]
# keys per partition after combine : [2, 3]
Key points
- groupByKey shuffles every record; no map-side combine
- reduceByKey: same input and output type, combines before the shuffle
- aggregateByKey: zero value + seqOp + combOp; result type can differ
- combineByKey: most general; first value creates the accumulator
- Functions must be associative (and commutative) to give correct results
- DataFrame groupBy().agg() uses partial aggregation automatically