What is Apache Spark
Apache Spark · Spark Fundamentals
Theory
Apache Spark is an open-source engine for processing large amounts of data in parallel across many machines. You describe what you want (read files, filter, join, aggregate) with DataFrames, SQL, Python, Scala, Java or R, and Spark splits the data into partitions and runs the work on many CPU cores at once. It keeps intermediate data in memory where it can, which makes multi-step pipelines much faster than engines that write to disk after every step.
The same engine handles batch jobs, streaming, machine learning and SQL analytics. For example, a retail company with 2 billion order rows can run one PySpark job that reads the orders from Parquet files, keeps only 2026 orders, sums amount per region and writes the result back, and Spark spreads that work across a cluster of executors so it finishes in minutes instead of hours.
Why we use it
One computer runs out of memory and time when data grows to hundreds of gigabytes or terabytes. Spark lets you write simple DataFrame or SQL code once and run it in parallel on a laptop or on a cluster of hundreds of machines, with automatic recovery when a machine fails.
When to use it
Use Spark when data is too big or too slow for a single machine, or when you need distributed ETL, streaming or ML on a data lake. For a few hundred MB that fits in memory, pandas, Polars or DuckDB on one machine is simpler and often faster.
How it works
- Write DataFrame or SQL code
- Spark builds a plan of the steps
- Optimizer rewrites the plan to be efficient
- Data is split into partitions
- Executors process partitions in parallel
- Did a task fail? Yes: Spark re-runs only that task from lineage. No: Results are combined.
- Result is shown or written to storage
Syntax
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("app").getOrCreate()
df = spark.read.parquet("path/") # read
result = df.filter(...).groupBy(...).agg(...) # transform
result.write.parquet("out/") # write (action)Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
# Total sales per region: the classic first Spark job
result = (orders
.filter(F.col("amount") > 10000)
.groupBy("region")
.agg(F.sum("amount").alias("total_sales"),
F.count("*").alias("orders"))
.orderBy(F.desc("total_sales")))
result.show()
# Output:
# +------+-----------+------+
# |region|total_sales|orders|
# +------+-----------+------+
# | North| 103000.0| 2|
# | West| 55000.0| 2|
# | South| 52000.0| 1|
# | East| 15000.0| 1|
# +------+-----------+------+
Key points
- Spark is a distributed processing engine, not a database: data lives in files, tables or streams
- Work is split into partitions that run in parallel on executor cores
- In-memory processing makes multi-step jobs much faster than disk-based MapReduce
- One engine for batch, SQL, streaming and ML; APIs in Python, Scala, Java, R and SQL
- PySpark is the Python API; your Python code drives the JVM-based Spark engine