Reading & writing files (CSV, JSON, Parquet, ORC, Delta/Iceberg note, save modes, partitionBy)

Apache Spark · DataFrames

Theory

spark.read loads files into a DataFrame and df.write saves one. You choose a format (csv, json, parquet, orc, text), options and a path: spark.read.option('header', True).schema(ddl).csv(path). CSV and JSON are text formats that need parsing and, without a schema, an extra pass to infer types. Parquet and ORC are columnar binary formats that store the schema, compress well and let Spark read only the needed columns and skip row groups using statistics, so they are the normal choice for analytics.

When writing, the save mode decides what happens if the path already exists: errorifexists (the default) fails, overwrite replaces, append adds files and ignore does nothing. partitionBy('region') writes one sub-folder per value (region=North/, region=West/), so queries filtering on region read only matching folders. Table formats such as Delta Lake and Apache Iceberg add ACID transactions, MERGE and time travel on top of Parquet; they need their own packages and are written with format('delta') or format('iceberg').

Why we use it

Almost every Spark job starts with a read and ends with a write. Choosing the right format, schema, save mode and partitioning decides how fast queries are, how much storage you use and whether a rerun duplicates or destroys data.

When to use it

Read CSV/JSON at the edge of the pipeline, then store curated data as Parquet, ORC or a table format (Delta, Iceberg). Partition by low-cardinality columns that queries filter on, such as date or region; never by IDs. Use overwrite for full refreshes and append for new batches.

How it works

  1. spark.read.format(...).option(...).schema(...)
  2. Text format (CSV/JSON)? Yes: Parse rows using the given schema. No: Read schema and only needed columns from Parquet/ORC.
  3. Transform the DataFrame
  4. df.write.mode(...).partitionBy(...)
  5. Does the target path exist? Yes: Apply mode: error, overwrite, append or ignore. No: Create it.
  6. Files are written, one or more per task

Syntax

spark.read.format("csv").option("header", True).schema(ddl).load(path)
spark.read.json(path); spark.read.parquet(path); spark.read.orc(path)
df.write.mode("overwrite").partitionBy("region").parquet(path)
modes: errorifexists (default) | overwrite | append | ignore
df.write.format("delta").save(path)   # needs the Delta Lake package

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

import os, tempfile
base = tempfile.mkdtemp()

# CSV with header; read back with an explicit schema
orders.write.mode("overwrite").option("header", True).csv(f"{base}/csv")
ddl = ("order_id INT, customer_id INT, region STRING, product STRING, "
       "qty INT, amount DOUBLE, order_date DATE")
csv_df = spark.read.option("header", True).schema(ddl).csv(f"{base}/csv")
print("csv rows:", csv_df.count())

# Parquet partitioned by region: one folder per value
orders.write.mode("overwrite").partitionBy("region").parquet(f"{base}/pq")
print(sorted(d for d in os.listdir(f"{base}/pq") if d.startswith("region=")))
north = spark.read.parquet(f"{base}/pq").filter("region = 'North'")
print("north rows:", north.count())

# Save modes
orders.limit(2).write.mode("append").partitionBy("region").parquet(f"{base}/pq")
print("after append:", spark.read.parquet(f"{base}/pq").count())
try:
    orders.write.parquet(f"{base}/pq")          # default mode = errorifexists
except Exception as e:
    print("default mode failed:", type(e).__name__)

# JSON and ORC work the same way
orders.write.mode("overwrite").json(f"{base}/json")
orders.write.mode("overwrite").orc(f"{base}/orc")
print(spark.read.json(f"{base}/json").count(), spark.read.orc(f"{base}/orc").count())

# Output:
# csv rows: 8
# ['region=East', 'region=North', 'region=South', 'region=West']
# north rows: 3
# after append: 10
# default mode failed: AnalysisException
# 8 8

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic