RDD basics

Apache Spark · RDDs

Theory

An RDD (Resilient Distributed Dataset) is Spark's original, low-level data abstraction: an immutable collection of Python or JVM objects split into partitions that live on different executors. Distributed means each partition can be processed on a different machine; resilient means Spark remembers the lineage (the chain of transformations that produced the RDD), so a lost partition can be rebuilt by replaying those steps instead of keeping copies.

You create RDDs with sc.parallelize(list, numSlices) for local data, sc.textFile(path) for text files, or df.rdd from a DataFrame. Like DataFrames, RDDs have lazy transformations (map, filter, flatMap, reduceByKey) and actions (collect, count, take, reduce). Unlike DataFrames they have no schema and no Catalyst optimizer, so Spark cannot look inside your Python functions. For example, sc.parallelize(range(1, 11), 2) creates an RDD of ten numbers in two partitions, and glom().collect() shows [[1..5], [6..10]].

Why we use it

RDDs are the foundation underneath DataFrames, and understanding partitions, lineage and lazy transformations at this level makes the rest of Spark easier to reason about. Interviewers still ask about RDDs, and some legacy code uses them.

When to use it

Use DataFrames for almost all new work: they are faster and simpler. Drop down to RDDs only for unstructured data that does not fit rows and columns, fine-grained control of partitioning, or old code. RDD APIs are not available on Spark Connect.

How it works

  1. Create an RDD (parallelize, textFile, df.rdd)
  2. Spark splits it into partitions
  3. Apply lazy transformations (map, filter, flatMap)
  4. Action called? Yes: Tasks run per partition on executors. No: Lineage just grows.
  5. Partition lost? Yes: Rebuild it from lineage. No: Return results.
  6. Results to the driver or storage

Syntax

sc = spark.sparkContext
rdd = sc.parallelize(data, numSlices)
rdd = sc.textFile("path/*.txt")
rdd = df.rdd                # RDD of Row objects
rdd.getNumPartitions(); rdd.glom().collect()

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

sc = spark.sparkContext

nums = sc.parallelize(range(1, 11), 2)        # 10 numbers in 2 partitions
print("partitions:", nums.getNumPartitions())
print("per partition:", nums.glom().collect())

squares = nums.map(lambda x: x * x)            # lazy transformation
print("sum of squares:", squares.sum())        # action

rows = orders.rdd                              # DataFrame -> RDD of Row objects
print(rows.map(lambda r: (r.order_id, r.region)).take(3))

# Output:
# partitions: 2
# per partition: [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
# sum of squares: 385
# [(101, 'North'), (102, 'West'), (103, 'North')]

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic