Pair RDDs & word count
Apache Spark · RDDs
Theory
A pair RDD is an RDD whose elements are two-item tuples (key, value). Spark adds key-based operations for them: reduceByKey, groupByKey, aggregateByKey, countByKey, sortByKey, mapValues, keys, values and joins such as join and leftOuterJoin. Keys decide which partition a record goes to during a shuffle, so all values for one key end up together.
Word count is the classic example and the 'hello world' of distributed computing. Read lines of text, flatMap each line into words, map each word to (word, 1), then reduceByKey(add) to sum the ones per word. For example, the lines 'spark is fast', 'spark is fun' and 'fast data' produce spark 2, is 2, fast 2, fun 1, data 1. The same pattern counts orders per city, clicks per page or errors per log level, and in DataFrame form it is explode(split(...)) followed by groupBy().count().
Why we use it
Pair RDDs are how you aggregate, group and join in the RDD API, and word count shows the full map, shuffle and reduce cycle in five lines. It is asked in interviews both in RDD and DataFrame form.
When to use it
Use pair RDD operations when you are already working with RDDs, for example on raw text. For structured data, the DataFrame version (groupBy, agg, join) is shorter and optimized. Prefer reduceByKey or aggregateByKey over groupByKey for aggregations.
How it works
- Read lines of text
- flatMap: split each line into words
- map: each word -> (word, 1)
- reduceByKey: add the 1s per word inside each partition
- Shuffle so each word is in one partition
- Final sum per word
- Sort or save the counts
Syntax
pairs = rdd.map(lambda x: (key, value))
pairs.reduceByKey(add) / groupByKey() / mapValues(f)
pairs.sortByKey(); pairs.join(other_pairs)
words.map(lambda w: (w, 1)).reduceByKey(add)Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
from operator import add
lines = spark.sparkContext.parallelize(["spark is fast", "spark is fun", "fast data"])
counts = (lines.flatMap(lambda line: line.split()) # words
.map(lambda w: (w, 1)) # (word, 1)
.reduceByKey(add)) # sum per word
print(sorted(counts.collect(), key=lambda kv: (-kv[1], kv[0])))
# Pair RDD join: total amount per customer name
names = spark.sparkContext.parallelize([(1, "Asha"), (2, "Ravi"), (3, "Meera")])
amounts = orders.rdd.map(lambda r: (r.customer_id, r.amount))
joined = names.join(amounts) # (id, (name, amount))
print(sorted(joined.values().reduceByKey(add).collect()))
# The same word count with DataFrames
df = spark.createDataFrame([("spark is fast",), ("spark is fun",), ("fast data",)], "line STRING")
(df.select(F.explode(F.split("line", " ")).alias("word"))
.groupBy("word").count().orderBy(F.desc("count"), "word").show())
# Output:
# [('fast', 2), ('is', 2), ('spark', 2), ('data', 1), ('fun', 1)]
# [('Asha', 56500.0), ('Meera', 54000.0), ('Ravi', 45000.0)]
# +-----+-----+
# | word|count|
# +-----+-----+
# | fast| 2|
# | is| 2|
# |spark| 2|
# | data| 1|
# | fun| 1|
# +-----+-----+
Key points
- Pair RDD = RDD of (key, value) tuples; unlocks reduceByKey, join, sortByKey
- Word count = flatMap -> map(word, 1) -> reduceByKey(add)
- groupByKey ships every value across the network; avoid it for sums and counts
- mapValues changes values without touching keys, keeping the partitioning
- DataFrame equivalent: explode(split(line, ' ')) then groupBy('word').count()