map, filter, flatMap, reduceByKey
Apache Spark · RDDs
Theory
These four are the core RDD operations. map(f) applies a function to every element and returns exactly one output per input, for example turning a Row into a (region, amount) pair. filter(f) keeps only elements where the function returns True. flatMap(f) can return zero, one or many outputs per input and flattens them into one RDD, which is how you split lines into words.
reduceByKey(f) works on pair RDDs of (key, value): it combines all values with the same key using a function such as lambda a, b: a + b. It first combines values inside each partition (a map-side combine), then shuffles only those partial results, so it moves far less data than groupByKey. For example, from orders you can filter amount > 10000, map to (region, amount) and reduceByKey to get total sales per region: North 103000, West 55000, South 52000, East 15000.
Why we use it
Most RDD programs are a chain of these operations. Knowing the difference between map and flatMap, and why reduceByKey beats groupByKey, is a favourite interview topic and also teaches how DataFrame aggregations work under the hood.
When to use it
Use map for one-to-one changes, flatMap for one-to-many (splitting text, exploding lists), filter to drop data early, and reduceByKey (or aggregateByKey) for per-key totals. In DataFrame code, the equivalents are select/withColumn, explode, filter and groupBy().agg.
How it works
- Start with an RDD of rows
- filter: drop elements that fail the test
- map: turn each row into a (key, value) pair
- reduceByKey: combine values per key inside each partition
- Shuffle partial results so each key lands in one partition
- Combine partial results per key
- collect() the totals
Syntax
rdd.map(lambda x: ...) # 1 -> 1
rdd.filter(lambda x: cond) # keep where True
rdd.flatMap(lambda x: [...]) # 1 -> 0..n
pairs.reduceByKey(lambda a, b: a + b)Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
pairs = (orders.rdd
.filter(lambda r: r.amount > 10000) # keep big orders
.map(lambda r: (r.region, r.amount))) # Row -> (key, value)
totals = pairs.reduceByKey(lambda a, b: a + b) # combine per key
print(sorted(totals.collect(), key=lambda kv: -kv[1]))
products = orders.rdd.map(lambda r: r.product) # map: one output per row
print(products.take(3))
lines = spark.sparkContext.parallelize(["Laptop Mouse", "Phone", "Tablet Mouse"])
print(lines.map(lambda s: s.split()).collect()) # map keeps lists
print(lines.flatMap(lambda s: s.split()).collect()) # flatMap flattens
# Output:
# [('North', 103000.0), ('West', 55000.0), ('South', 52000.0), ('East', 15000.0)]
# ['Laptop', 'Phone', 'Mouse']
# [['Laptop', 'Mouse'], ['Phone'], ['Tablet', 'Mouse']]
# ['Laptop', 'Mouse', 'Phone', 'Tablet', 'Mouse']
Key points
- map: exactly one output per input; flatMap: zero or more, flattened
- filter early to shrink data before shuffles
- reduceByKey combines inside partitions first, so it shuffles less than groupByKey
- The reduce function must be associative and commutative (sum, max, min)
- Lambdas run in Python workers; DataFrame functions are usually faster