map, filter, flatMap, reduceByKey

Apache Spark · RDDs

Theory

These four are the core RDD operations. map(f) applies a function to every element and returns exactly one output per input, for example turning a Row into a (region, amount) pair. filter(f) keeps only elements where the function returns True. flatMap(f) can return zero, one or many outputs per input and flattens them into one RDD, which is how you split lines into words.

reduceByKey(f) works on pair RDDs of (key, value): it combines all values with the same key using a function such as lambda a, b: a + b. It first combines values inside each partition (a map-side combine), then shuffles only those partial results, so it moves far less data than groupByKey. For example, from orders you can filter amount > 10000, map to (region, amount) and reduceByKey to get total sales per region: North 103000, West 55000, South 52000, East 15000.

Why we use it

Most RDD programs are a chain of these operations. Knowing the difference between map and flatMap, and why reduceByKey beats groupByKey, is a favourite interview topic and also teaches how DataFrame aggregations work under the hood.

When to use it

Use map for one-to-one changes, flatMap for one-to-many (splitting text, exploding lists), filter to drop data early, and reduceByKey (or aggregateByKey) for per-key totals. In DataFrame code, the equivalents are select/withColumn, explode, filter and groupBy().agg.

How it works

  1. Start with an RDD of rows
  2. filter: drop elements that fail the test
  3. map: turn each row into a (key, value) pair
  4. reduceByKey: combine values per key inside each partition
  5. Shuffle partial results so each key lands in one partition
  6. Combine partial results per key
  7. collect() the totals

Syntax

rdd.map(lambda x: ...)            # 1 -> 1
rdd.filter(lambda x: cond)        # keep where True
rdd.flatMap(lambda x: [...])      # 1 -> 0..n
pairs.reduceByKey(lambda a, b: a + b)

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

pairs = (orders.rdd
         .filter(lambda r: r.amount > 10000)          # keep big orders
         .map(lambda r: (r.region, r.amount)))        # Row -> (key, value)

totals = pairs.reduceByKey(lambda a, b: a + b)        # combine per key
print(sorted(totals.collect(), key=lambda kv: -kv[1]))

products = orders.rdd.map(lambda r: r.product)        # map: one output per row
print(products.take(3))

lines = spark.sparkContext.parallelize(["Laptop Mouse", "Phone", "Tablet Mouse"])
print(lines.map(lambda s: s.split()).collect())       # map keeps lists
print(lines.flatMap(lambda s: s.split()).collect())   # flatMap flattens

# Output:
# [('North', 103000.0), ('West', 55000.0), ('South', 52000.0), ('East', 15000.0)]
# ['Laptop', 'Phone', 'Mouse']
# [['Laptop', 'Mouse'], ['Phone'], ['Tablet', 'Mouse']]
# ['Laptop', 'Mouse', 'Phone', 'Tablet', 'Mouse']

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic