Spark architecture (driver/executors, jobs/stages/tasks, lazy evaluation)

Databricks · Apache Spark on Databricks

Theory

Apache Spark is a distributed engine. Each Spark application has one driver, which runs your notebook code, builds an execution plan and schedules work, and many executors on worker nodes, which hold data partitions in memory and run tasks in parallel. On Databricks the cluster's driver node runs the driver; on serverless, Databricks manages both sides for you.

Spark is lazy. Transformations such as filter, select, groupBy and join only add steps to a logical plan. Nothing runs until an action such as count(), show(), display(), collect() or a write is called. The action becomes a job; the job is split into stages at every shuffle boundary (for example a groupBy), and each stage runs one task per partition. For example, reading orders, filtering amount > 900 and calling count() gives one job; adding groupBy("region") before count() adds a shuffle and a second stage.

Why we use it

Knowing what runs where explains performance: why collect() can crash the driver, why shuffles are slow, and why nothing seems to happen until an action. It is also a core topic in the Data Engineer Associate exam.

When to use it

Think in this model whenever you tune a job or read the Spark UI or query profile. For small data that fits on one machine, plain pandas may be simpler; for anything that grows, let Spark distribute it.

How it works

  1. Write transformations in the notebook
  2. Spark adds each step to a logical plan
  3. Did the code call an action (count, show, write)? Yes: Catalyst optimizes the plan and Spark creates a job. No: Nothing runs yet.
  4. Job is split into stages at shuffle boundaries
  5. Each stage runs one task per partition on executors
  6. Results go to the driver or are written to storage

Syntax

Transformations (lazy): select, filter/where, withColumn, groupBy, join, orderBy
Actions (run a job): count, show, display, collect, take, write.saveAsTable
Plan: df.explain() | df.explain("formatted")

Example

orders = spark.read.table("main.sales.orders")       # lazy
big = orders.filter("amount > 900")                   # lazy
by_region = big.groupBy("region").count()             # lazy (wide: needs a shuffle)

by_region.explain()     # show the plan, still no job
by_region.show()        # ACTION: Spark creates a job -> 2 stages -> tasks

# Avoid pulling big data to the driver
rows = by_region.collect()   # fine here: only a few rows

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic