mapPartitions, sortByKey & joins on pair RDDs
Apache Spark · RDDs
Theory
map calls your function once per element; mapPartitions calls it once per partition and hands it an iterator over all the elements of that partition. That matters when each call needs costly setup, such as opening a database connection, loading a model or compiling a regex: you do the setup once and then loop over the rows. sortByKey(ascending) sorts a pair RDD by key; it samples the keys to build a range partitioner, so partition 0 holds the smallest keys, and then sorts inside each partition.
Pair RDDs also join by key. join keeps keys present on both sides, leftOuterJoin and rightOuterJoin keep one side and fill None, fullOuterJoin keeps all keys, and cogroup returns, for each key, the full list of values from both sides. For example, joining customers (1, Asha), (3, Meera) with spend (1, 55000), (1, 1500) gives (1, ('Asha', 55000)), (1, ('Asha', 1500)) for join, plus (3, ('Meera', None)) for leftOuterJoin.
Why we use it
mapPartitions is the standard way to make per-record work cheaper by sharing setup, and pair-RDD joins and sorts appear in older codebases and in interviews about how DataFrame joins work underneath.
When to use it
Use mapPartitions when each element needs a shared, expensive resource, and make it yield results instead of building a huge list. Use pair-RDD joins only when you are already on RDDs; DataFrame joins are optimised (broadcast, sort-merge, AQE) and are the normal choice. Use mapInPandas for the DataFrame equivalent of mapPartitions.
How it works
- Partition of rows arrives at a task
- Rows need costly setup (connection, model)? Yes: mapPartitions: set up once. No: map: f per row.
- Yield transformed rows from the iterator
- For pair RDDs, shuffle both sides by key to join
- Match keys and emit (key, (left, right)); outer joins fill None
- Downstream steps receive the joined pairs
Syntax
rdd.mapPartitions(lambda it: (f(x) for x in it))
rdd.mapPartitionsWithIndex(lambda idx, it: ...)
pairs.sortByKey(ascending=False, numPartitions=4)
a.join(b); a.leftOuterJoin(b); a.rightOuterJoin(b); a.fullOuterJoin(b)
a.cogroup(b) # key -> (values_from_a, values_from_b)Example
from pyspark.sql import SparkSession
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.getOrCreate())
sc = spark.sparkContext
orders = sc.parallelize([(101, 55000.0), (102, 30000.0), (103, 1500.0),
(104, 52000.0), (105, 25000.0), (106, 15000.0)], 2)
# 1) mapPartitions: the function gets an iterator over a whole partition,
# so expensive setup (DB connection, model load) happens once per partition
def with_gst(rows):
rate = 0.18 # imagine: load a config / open a connection here
for oid, amt in rows:
yield oid, round(amt * (1 + rate), 1)
print(orders.mapPartitions(with_gst).take(2))
# mapPartitionsWithIndex also passes the partition number
print(orders.mapPartitionsWithIndex(lambda i, it: [(i, sum(1 for _ in it))]).collect())
# 2) sortByKey: range-partitions by key, then sorts inside each partition
print(orders.map(lambda kv: (kv[1], kv[0])).sortByKey(ascending=False).values().take(3))
# 3) Joins on pair RDDs: both sides are (key, value); the key is the join column
cust = sc.parallelize([(1, "Asha"), (2, "Ravi"), (3, "Meera"), (4, "Karan")])
spend = sc.parallelize([(1, 55000.0), (2, 30000.0), (1, 1500.0), (5, 9000.0)])
print("join :", sorted(cust.join(spend).collect()))
print("leftOuter :", sorted(cust.leftOuterJoin(spend).collect()))
print("rightOuter:", sorted(cust.rightOuterJoin(spend).collect()))
print("fullOuter :", sorted(cust.fullOuterJoin(spend).collect(), key=lambda kv: kv[0]))
# 4) cogroup: for each key, all values from both sides
print("cogroup :", sorted((k, (list(a), list(b))) for k, (a, b) in cust.cogroup(spend).collect()))
# Output:
# [(101, 64900.0), (102, 35400.0)]
# [(0, 3), (1, 3)]
# [101, 104, 102]
# join : [(1, ('Asha', 1500.0)), (1, ('Asha', 55000.0)), (2, ('Ravi', 30000.0))]
# leftOuter : [(1, ('Asha', 1500.0)), (1, ('Asha', 55000.0)), (2, ('Ravi', 30000.0)), (3, ('Meera', None)), (4, ('Karan', None))]
# rightOuter: [(1, ('Asha', 1500.0)), (1, ('Asha', 55000.0)), (2, ('Ravi', 30000.0)), (5, (None, 9000.0))]
# fullOuter : [(1, ('Asha', 55000.0)), (1, ('Asha', 1500.0)), (2, ('Ravi', 30000.0)), (3, ('Meera', None)), (4, ('Karan', None)), (5, (None, 9000.0))]
# cogroup : [(1, (['Asha'], [55000.0, 1500.0])), (2, (['Ravi'], [30000.0])), (3, (['Meera'], [])), (4, (['Karan'], [])), (5, ([], [9000.0]))]
Key points
- mapPartitions: one call per partition, gets an iterator
- Do setup once per partition, then yield each output row
- sortByKey uses a range partitioner, so output is globally sorted
- join/leftOuterJoin/rightOuterJoin/fullOuterJoin return (key, (left, right))
- Missing side becomes None in outer joins
- cogroup gives all values from both sides per key