RDD basics
Apache Spark · RDDs
Theory
An RDD (Resilient Distributed Dataset) is Spark's original, low-level data abstraction: an immutable collection of Python or JVM objects split into partitions that live on different executors. Distributed means each partition can be processed on a different machine; resilient means Spark remembers the lineage (the chain of transformations that produced the RDD), so a lost partition can be rebuilt by replaying those steps instead of keeping copies.
You create RDDs with sc.parallelize(list, numSlices) for local data, sc.textFile(path) for text files, or df.rdd from a DataFrame. Like DataFrames, RDDs have lazy transformations (map, filter, flatMap, reduceByKey) and actions (collect, count, take, reduce). Unlike DataFrames they have no schema and no Catalyst optimizer, so Spark cannot look inside your Python functions. For example, sc.parallelize(range(1, 11), 2) creates an RDD of ten numbers in two partitions, and glom().collect() shows [[1..5], [6..10]].
Why we use it
RDDs are the foundation underneath DataFrames, and understanding partitions, lineage and lazy transformations at this level makes the rest of Spark easier to reason about. Interviewers still ask about RDDs, and some legacy code uses them.
When to use it
Use DataFrames for almost all new work: they are faster and simpler. Drop down to RDDs only for unstructured data that does not fit rows and columns, fine-grained control of partitioning, or old code. RDD APIs are not available on Spark Connect.
How it works
- Create an RDD (parallelize, textFile, df.rdd)
- Spark splits it into partitions
- Apply lazy transformations (map, filter, flatMap)
- Action called? Yes: Tasks run per partition on executors. No: Lineage just grows.
- Partition lost? Yes: Rebuild it from lineage. No: Return results.
- Results to the driver or storage
Syntax
sc = spark.sparkContext
rdd = sc.parallelize(data, numSlices)
rdd = sc.textFile("path/*.txt")
rdd = df.rdd # RDD of Row objects
rdd.getNumPartitions(); rdd.glom().collect()Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
sc = spark.sparkContext
nums = sc.parallelize(range(1, 11), 2) # 10 numbers in 2 partitions
print("partitions:", nums.getNumPartitions())
print("per partition:", nums.glom().collect())
squares = nums.map(lambda x: x * x) # lazy transformation
print("sum of squares:", squares.sum()) # action
rows = orders.rdd # DataFrame -> RDD of Row objects
print(rows.map(lambda r: (r.order_id, r.region)).take(3))
# Output:
# partitions: 2
# per partition: [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
# sum of squares: 385
# [(101, 'North'), (102, 'West'), (103, 'North')]
Key points
- RDD = immutable, partitioned collection of objects with lineage for recovery
- Create with parallelize, textFile or df.rdd
- Same lazy transformation / action model as DataFrames
- No schema and no Catalyst optimization, so usually slower than DataFrames in Python
- Python RDD functions run in Python worker processes, adding serialization cost