What is Apache Spark

Apache Spark · Spark Fundamentals

Theory

Apache Spark is an open-source engine for processing large amounts of data in parallel across many machines. You describe what you want (read files, filter, join, aggregate) with DataFrames, SQL, Python, Scala, Java or R, and Spark splits the data into partitions and runs the work on many CPU cores at once. It keeps intermediate data in memory where it can, which makes multi-step pipelines much faster than engines that write to disk after every step.

The same engine handles batch jobs, streaming, machine learning and SQL analytics. For example, a retail company with 2 billion order rows can run one PySpark job that reads the orders from Parquet files, keeps only 2026 orders, sums amount per region and writes the result back, and Spark spreads that work across a cluster of executors so it finishes in minutes instead of hours.

Why we use it

One computer runs out of memory and time when data grows to hundreds of gigabytes or terabytes. Spark lets you write simple DataFrame or SQL code once and run it in parallel on a laptop or on a cluster of hundreds of machines, with automatic recovery when a machine fails.

When to use it

Use Spark when data is too big or too slow for a single machine, or when you need distributed ETL, streaming or ML on a data lake. For a few hundred MB that fits in memory, pandas, Polars or DuckDB on one machine is simpler and often faster.

How it works

  1. Write DataFrame or SQL code
  2. Spark builds a plan of the steps
  3. Optimizer rewrites the plan to be efficient
  4. Data is split into partitions
  5. Executors process partitions in parallel
  6. Did a task fail? Yes: Spark re-runs only that task from lineage. No: Results are combined.
  7. Result is shown or written to storage

Syntax

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("app").getOrCreate()
df = spark.read.parquet("path/")      # read
result = df.filter(...).groupBy(...).agg(...)   # transform
result.write.parquet("out/")          # write (action)

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

# Total sales per region: the classic first Spark job
result = (orders
          .filter(F.col("amount") > 10000)
          .groupBy("region")
          .agg(F.sum("amount").alias("total_sales"),
               F.count("*").alias("orders"))
          .orderBy(F.desc("total_sales")))
result.show()

# Output:
# +------+-----------+------+
# |region|total_sales|orders|
# +------+-----------+------+
# | North|   103000.0|     2|
# |  West|    55000.0|     2|
# | South|    52000.0|     1|
# |  East|    15000.0|     1|
# +------+-----------+------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic