groupByKey vs reduceByKey vs aggregateByKey vs combineByKey

Apache Spark · RDDs

Theory

All four group a pair RDD by key, but they move very different amounts of data. groupByKey shuffles every single (key, value) record and builds a full list of values per key on the reducer, which is slow and can run out of memory for a hot key. reduceByKey(f) first combines values inside each partition (a map-side combine), so each partition sends at most one partial result per key; f must take and return the value type, such as a + b.

aggregateByKey(zero, seqOp, combOp) also combines map-side but lets the result type differ from the value type: start from a zero value such as (0.0, 0), add each value with seqOp, and merge partial results across partitions with combOp. combineByKey(createCombiner, mergeValue, mergeCombiners) is the most general form, where the first value seen for a key creates the accumulator. For example, to get the average order per region you keep (sum, count) per key: North (104500, 3) gives 34833.3.

Why we use it

Choosing groupByKey when a combining operator would work is a classic cause of slow jobs and executor out-of-memory errors, and the four-way comparison is a very common interview question. The DataFrame groupBy().agg() already does the efficient version for you.

When to use it

Use reduceByKey for sums, counts, min and max. Use aggregateByKey or combineByKey when the result has a different shape, such as (sum, count) for averages or sets of distinct values. Use groupByKey only when you truly need every value per key, such as sorting all events of a user. In DataFrame code use groupBy().agg().

How it works

  1. Need one result per key from a pair RDD
  2. Need every value of each key? Yes: groupByKey (shuffles all). No: Use a combining operator.
  3. Result same type as value (sum, max)? Yes: reduceByKey. No: aggregateByKey / combineByKey.
  4. Combine values inside each partition first
  5. Shuffle only the partial results per key
  6. Merge partials into one value per key

Syntax

pairs.groupByKey().mapValues(sum)
pairs.reduceByKey(lambda a, b: a + b)
pairs.aggregateByKey((0.0, 0),
    lambda acc, v: (acc[0] + v, acc[1] + 1),      # seqOp (in a partition)
    lambda a, b: (a[0] + b[0], a[1] + b[1]))      # combOp (across partitions)
pairs.combineByKey(lambda v: (v, 1), mergeValue, mergeCombiners)

Example

from pyspark.sql import SparkSession

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())
sc = spark.sparkContext

sales = sc.parallelize([("North", 55000.0), ("West", 30000.0), ("North", 1500.0),
                        ("South", 52000.0), ("West", 25000.0), ("North", 48000.0)], 2)

# 1) groupByKey: ships EVERY value across the network, then you aggregate
g = sales.groupByKey().mapValues(lambda vals: sum(vals))
print("groupByKey    :", sorted(g.collect()))

# 2) reduceByKey: combines inside each partition first (map-side combine)
r = sales.reduceByKey(lambda a, b: a + b)
print("reduceByKey   :", sorted(r.collect()))

# 3) aggregateByKey: zero value + within-partition fn + across-partition fn.
#    The result type (sum, count) differs from the value type (float).
sc_pair = sales.aggregateByKey((0.0, 0),
                               lambda acc, v: (acc[0] + v, acc[1] + 1),     # seqOp
                               lambda a, b: (a[0] + b[0], a[1] + b[1]))     # combOp
print("aggregateByKey:", sorted(sc_pair.collect()))

# 4) combineByKey: like aggregateByKey, but the first value creates the accumulator
avg = (sales.combineByKey(lambda v: (v, 1),                                 # createCombiner
                          lambda acc, v: (acc[0] + v, acc[1] + 1),          # mergeValue
                          lambda a, b: (a[0] + b[0], a[1] + b[1]))          # mergeCombiners
            .mapValues(lambda t: round(t[0] / t[1], 1)))
print("combineByKey avg:", sorted(avg.collect()))

# How many records each partition would send to the shuffle
print("records per partition, no combine:", sales.glom().map(len).collect())
print("keys per partition after combine :", sales.mapPartitions(lambda it: [len({k for k, _ in it})]).collect())

# Output:
# groupByKey    : [('North', 104500.0), ('South', 52000.0), ('West', 55000.0)]
# reduceByKey   : [('North', 104500.0), ('South', 52000.0), ('West', 55000.0)]
# aggregateByKey: [('North', (104500.0, 3)), ('South', (52000.0, 1)), ('West', (55000.0, 2))]
# combineByKey avg: [('North', 34833.3), ('South', 52000.0), ('West', 27500.0)]
# records per partition, no combine: [3, 3]
# keys per partition after combine : [2, 3]

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic