Reading & writing files (CSV, JSON, Parquet, ORC, Delta/Iceberg note, save modes, partitionBy)
Apache Spark · DataFrames
Theory
spark.read loads files into a DataFrame and df.write saves one. You choose a format (csv, json, parquet, orc, text), options and a path: spark.read.option('header', True).schema(ddl).csv(path). CSV and JSON are text formats that need parsing and, without a schema, an extra pass to infer types. Parquet and ORC are columnar binary formats that store the schema, compress well and let Spark read only the needed columns and skip row groups using statistics, so they are the normal choice for analytics.
When writing, the save mode decides what happens if the path already exists: errorifexists (the default) fails, overwrite replaces, append adds files and ignore does nothing. partitionBy('region') writes one sub-folder per value (region=North/, region=West/), so queries filtering on region read only matching folders. Table formats such as Delta Lake and Apache Iceberg add ACID transactions, MERGE and time travel on top of Parquet; they need their own packages and are written with format('delta') or format('iceberg').
Why we use it
Almost every Spark job starts with a read and ends with a write. Choosing the right format, schema, save mode and partitioning decides how fast queries are, how much storage you use and whether a rerun duplicates or destroys data.
When to use it
Read CSV/JSON at the edge of the pipeline, then store curated data as Parquet, ORC or a table format (Delta, Iceberg). Partition by low-cardinality columns that queries filter on, such as date or region; never by IDs. Use overwrite for full refreshes and append for new batches.
How it works
- spark.read.format(...).option(...).schema(...)
- Text format (CSV/JSON)? Yes: Parse rows using the given schema. No: Read schema and only needed columns from Parquet/ORC.
- Transform the DataFrame
- df.write.mode(...).partitionBy(...)
- Does the target path exist? Yes: Apply mode: error, overwrite, append or ignore. No: Create it.
- Files are written, one or more per task
Syntax
spark.read.format("csv").option("header", True).schema(ddl).load(path)
spark.read.json(path); spark.read.parquet(path); spark.read.orc(path)
df.write.mode("overwrite").partitionBy("region").parquet(path)
modes: errorifexists (default) | overwrite | append | ignore
df.write.format("delta").save(path) # needs the Delta Lake packageExample
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
import os, tempfile
base = tempfile.mkdtemp()
# CSV with header; read back with an explicit schema
orders.write.mode("overwrite").option("header", True).csv(f"{base}/csv")
ddl = ("order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date DATE")
csv_df = spark.read.option("header", True).schema(ddl).csv(f"{base}/csv")
print("csv rows:", csv_df.count())
# Parquet partitioned by region: one folder per value
orders.write.mode("overwrite").partitionBy("region").parquet(f"{base}/pq")
print(sorted(d for d in os.listdir(f"{base}/pq") if d.startswith("region=")))
north = spark.read.parquet(f"{base}/pq").filter("region = 'North'")
print("north rows:", north.count())
# Save modes
orders.limit(2).write.mode("append").partitionBy("region").parquet(f"{base}/pq")
print("after append:", spark.read.parquet(f"{base}/pq").count())
try:
orders.write.parquet(f"{base}/pq") # default mode = errorifexists
except Exception as e:
print("default mode failed:", type(e).__name__)
# JSON and ORC work the same way
orders.write.mode("overwrite").json(f"{base}/json")
orders.write.mode("overwrite").orc(f"{base}/orc")
print(spark.read.json(f"{base}/json").count(), spark.read.orc(f"{base}/orc").count())
# Output:
# csv rows: 8
# ['region=East', 'region=North', 'region=South', 'region=West']
# north rows: 3
# after append: 10
# default mode failed: AnalysisException
# 8 8
Key points
- Pass a schema when reading CSV/JSON; inferSchema costs an extra pass and can guess wrong
- Parquet/ORC are columnar and compressed: column pruning and predicate pushdown
- Default save mode errorifexists fails if the path exists; overwrite replaces everything
- partitionBy creates col=value folders and enables partition pruning
- Too many partition values (e.g. by customer_id) creates millions of tiny files
- Delta Lake and Iceberg add ACID, MERGE and time travel; install their packages to use them