mapPartitions, sortByKey & joins on pair RDDs

Apache Spark · RDDs

Theory

map calls your function once per element; mapPartitions calls it once per partition and hands it an iterator over all the elements of that partition. That matters when each call needs costly setup, such as opening a database connection, loading a model or compiling a regex: you do the setup once and then loop over the rows. sortByKey(ascending) sorts a pair RDD by key; it samples the keys to build a range partitioner, so partition 0 holds the smallest keys, and then sorts inside each partition.

Pair RDDs also join by key. join keeps keys present on both sides, leftOuterJoin and rightOuterJoin keep one side and fill None, fullOuterJoin keeps all keys, and cogroup returns, for each key, the full list of values from both sides. For example, joining customers (1, Asha), (3, Meera) with spend (1, 55000), (1, 1500) gives (1, ('Asha', 55000)), (1, ('Asha', 1500)) for join, plus (3, ('Meera', None)) for leftOuterJoin.

Why we use it

mapPartitions is the standard way to make per-record work cheaper by sharing setup, and pair-RDD joins and sorts appear in older codebases and in interviews about how DataFrame joins work underneath.

When to use it

Use mapPartitions when each element needs a shared, expensive resource, and make it yield results instead of building a huge list. Use pair-RDD joins only when you are already on RDDs; DataFrame joins are optimised (broadcast, sort-merge, AQE) and are the normal choice. Use mapInPandas for the DataFrame equivalent of mapPartitions.

How it works

  1. Partition of rows arrives at a task
  2. Rows need costly setup (connection, model)? Yes: mapPartitions: set up once. No: map: f per row.
  3. Yield transformed rows from the iterator
  4. For pair RDDs, shuffle both sides by key to join
  5. Match keys and emit (key, (left, right)); outer joins fill None
  6. Downstream steps receive the joined pairs

Syntax

rdd.mapPartitions(lambda it: (f(x) for x in it))
rdd.mapPartitionsWithIndex(lambda idx, it: ...)
pairs.sortByKey(ascending=False, numPartitions=4)
a.join(b); a.leftOuterJoin(b); a.rightOuterJoin(b); a.fullOuterJoin(b)
a.cogroup(b)          # key -> (values_from_a, values_from_b)

Example

from pyspark.sql import SparkSession

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())
sc = spark.sparkContext

orders = sc.parallelize([(101, 55000.0), (102, 30000.0), (103, 1500.0),
                         (104, 52000.0), (105, 25000.0), (106, 15000.0)], 2)

# 1) mapPartitions: the function gets an iterator over a whole partition,
#    so expensive setup (DB connection, model load) happens once per partition
def with_gst(rows):
    rate = 0.18                          # imagine: load a config / open a connection here
    for oid, amt in rows:
        yield oid, round(amt * (1 + rate), 1)
print(orders.mapPartitions(with_gst).take(2))

# mapPartitionsWithIndex also passes the partition number
print(orders.mapPartitionsWithIndex(lambda i, it: [(i, sum(1 for _ in it))]).collect())

# 2) sortByKey: range-partitions by key, then sorts inside each partition
print(orders.map(lambda kv: (kv[1], kv[0])).sortByKey(ascending=False).values().take(3))

# 3) Joins on pair RDDs: both sides are (key, value); the key is the join column
cust = sc.parallelize([(1, "Asha"), (2, "Ravi"), (3, "Meera"), (4, "Karan")])
spend = sc.parallelize([(1, 55000.0), (2, 30000.0), (1, 1500.0), (5, 9000.0)])
print("join      :", sorted(cust.join(spend).collect()))
print("leftOuter :", sorted(cust.leftOuterJoin(spend).collect()))
print("rightOuter:", sorted(cust.rightOuterJoin(spend).collect()))
print("fullOuter :", sorted(cust.fullOuterJoin(spend).collect(), key=lambda kv: kv[0]))

# 4) cogroup: for each key, all values from both sides
print("cogroup   :", sorted((k, (list(a), list(b))) for k, (a, b) in cust.cogroup(spend).collect()))

# Output:
# [(101, 64900.0), (102, 35400.0)]
# [(0, 3), (1, 3)]
# [101, 104, 102]
# join      : [(1, ('Asha', 1500.0)), (1, ('Asha', 55000.0)), (2, ('Ravi', 30000.0))]
# leftOuter : [(1, ('Asha', 1500.0)), (1, ('Asha', 55000.0)), (2, ('Ravi', 30000.0)), (3, ('Meera', None)), (4, ('Karan', None))]
# rightOuter: [(1, ('Asha', 1500.0)), (1, ('Asha', 55000.0)), (2, ('Ravi', 30000.0)), (5, (None, 9000.0))]
# fullOuter : [(1, ('Asha', 55000.0)), (1, ('Asha', 1500.0)), (2, ('Ravi', 30000.0)), (3, ('Meera', None)), (4, ('Karan', None)), (5, (None, 9000.0))]
# cogroup   : [(1, (['Asha'], [55000.0, 1500.0])), (2, (['Ravi'], [30000.0])), (3, (['Meera'], [])), (4, (['Karan'], [])), (5, ([], [9000.0]))]

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic