Pair RDDs & word count

Apache Spark · RDDs

Theory

A pair RDD is an RDD whose elements are two-item tuples (key, value). Spark adds key-based operations for them: reduceByKey, groupByKey, aggregateByKey, countByKey, sortByKey, mapValues, keys, values and joins such as join and leftOuterJoin. Keys decide which partition a record goes to during a shuffle, so all values for one key end up together.

Word count is the classic example and the 'hello world' of distributed computing. Read lines of text, flatMap each line into words, map each word to (word, 1), then reduceByKey(add) to sum the ones per word. For example, the lines 'spark is fast', 'spark is fun' and 'fast data' produce spark 2, is 2, fast 2, fun 1, data 1. The same pattern counts orders per city, clicks per page or errors per log level, and in DataFrame form it is explode(split(...)) followed by groupBy().count().

Why we use it

Pair RDDs are how you aggregate, group and join in the RDD API, and word count shows the full map, shuffle and reduce cycle in five lines. It is asked in interviews both in RDD and DataFrame form.

When to use it

Use pair RDD operations when you are already working with RDDs, for example on raw text. For structured data, the DataFrame version (groupBy, agg, join) is shorter and optimized. Prefer reduceByKey or aggregateByKey over groupByKey for aggregations.

How it works

  1. Read lines of text
  2. flatMap: split each line into words
  3. map: each word -> (word, 1)
  4. reduceByKey: add the 1s per word inside each partition
  5. Shuffle so each word is in one partition
  6. Final sum per word
  7. Sort or save the counts

Syntax

pairs = rdd.map(lambda x: (key, value))
pairs.reduceByKey(add) / groupByKey() / mapValues(f)
pairs.sortByKey(); pairs.join(other_pairs)
words.map(lambda w: (w, 1)).reduceByKey(add)

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

from operator import add

lines = spark.sparkContext.parallelize(["spark is fast", "spark is fun", "fast data"])
counts = (lines.flatMap(lambda line: line.split())   # words
               .map(lambda w: (w, 1))                # (word, 1)
               .reduceByKey(add))                    # sum per word
print(sorted(counts.collect(), key=lambda kv: (-kv[1], kv[0])))

# Pair RDD join: total amount per customer name
names = spark.sparkContext.parallelize([(1, "Asha"), (2, "Ravi"), (3, "Meera")])
amounts = orders.rdd.map(lambda r: (r.customer_id, r.amount))
joined = names.join(amounts)                     # (id, (name, amount))
print(sorted(joined.values().reduceByKey(add).collect()))

# The same word count with DataFrames
df = spark.createDataFrame([("spark is fast",), ("spark is fun",), ("fast data",)], "line STRING")
(df.select(F.explode(F.split("line", " ")).alias("word"))
   .groupBy("word").count().orderBy(F.desc("count"), "word").show())

# Output:
# [('fast', 2), ('is', 2), ('spark', 2), ('data', 1), ('fun', 1)]
# [('Asha', 56500.0), ('Meera', 54000.0), ('Ravi', 45000.0)]
# +-----+-----+
# | word|count|
# +-----+-----+
# | fast|    2|
# |   is|    2|
# |spark|    2|
# | data|    1|
# |  fun|    1|
# +-----+-----+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic