Shared variables: broadcast variables & accumulators

Apache Spark · RDDs

Theory

Normally every variable your function uses is copied into each task's closure. Spark offers two special shared variables for the common exceptions. A broadcast variable, created with sc.broadcast(value), sends a read-only value such as a lookup dictionary to each executor once, where all tasks read it through .value. That beats shipping a 1 MB dictionary inside every one of thousands of tasks.

An accumulator, created with sc.accumulator(0), is a write-only counter for tasks: they can only call add(), and only the driver reads .value after an action. It is handy for counting bad records or skipped rows. Updates made inside actions such as foreach() are applied exactly once, but updates inside transformations such as map() are applied again whenever that stage is recomputed or retried. For example, a map that counts unknown cities reports 2 after the first action and 4 after a second action on the same uncached RDD.

Why we use it

Broadcasting a lookup table is a simple, big speed-up for RDD code and UDFs, and accumulators give cheap job-level counters without an extra pass. Both are standard interview questions, especially the accumulator double-counting gotcha.

When to use it

Broadcast small, read-only lookups (a few MB) used by many tasks; for joining two DataFrames use a broadcast join instead. Use accumulators for diagnostics and counters, updated inside actions when the number must be exact. Never use them to pass data between tasks.

How it works

  1. Driver creates bc = sc.broadcast(lookup) and acc = sc.accumulator(0)
  2. Tasks ship with a small handle, not the whole lookup
  3. Each executor fetches the broadcast once and caches it
  4. Tasks read bc.value and call acc.add() for bad rows
  5. Stage recomputed or retried? Yes: map() updates are added again. No: Each update counted once.
  6. Driver reads acc.value after the action

Syntax

bc = sc.broadcast({"Jaipur": "Rajasthan"})
rdd.map(lambda c: bc.value.get(c))
bc.unpersist(); bc.destroy()
acc = sc.accumulator(0)
rdd.foreach(lambda x: acc.add(1))
print(acc.value)            # driver only, after an action

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())
sc = spark.sparkContext

# 1) Broadcast variable: one read-only copy per executor instead of one per task
city_state = sc.broadcast({"Jaipur": "Rajasthan", "Pune": "Maharashtra", "Delhi": "Delhi"})

# 2) Accumulator: tasks can only add; the driver reads the total
bad_rows = sc.accumulator(0)

def parse(line):
    city, amount = line.split(",")
    state = city_state.value.get(city)          # .value reads the local copy
    if state is None:
        bad_rows.add(1)
    return city, state, float(amount)

lines = sc.parallelize(["Jaipur,55000", "Pune,30000", "Surat,1500", "Delhi,52000", "Agra,900"], 2)
rows = lines.map(parse).filter(lambda r: r[1] is not None)
print(rows.collect())                            # the action runs the tasks
print("unknown cities:", bad_rows.value)         # read only after an action

# 3) Gotcha: an accumulator updated inside a transformation counts again on recompute
rows.count()
print("after a second action:", bad_rows.value)

# 4) Safe pattern: update accumulators inside an action such as foreach()
seen = sc.accumulator(0)
lines.foreach(lambda _: seen.add(1))
print("lines seen:", seen.value)

# 5) Broadcasts also work inside DataFrame UDFs
@F.udf("string")
def to_state(city):
    return city_state.value.get(city, "Unknown")

df = spark.createDataFrame([("Jaipur",), ("Agra",)], "city STRING")
df.withColumn("state", to_state("city")).show()

city_state.unpersist()     # drop executor copies (destroy() also frees the driver copy)

# Output:
# [('Jaipur', 'Rajasthan', 55000.0), ('Pune', 'Maharashtra', 30000.0), ('Delhi', 'Delhi', 52000.0)]
# unknown cities: 2
# after a second action: 4
# lines seen: 5
# +------+---------+
# |  city|    state|
# +------+---------+
# |Jaipur|Rajasthan|
# |  Agra|  Unknown|
# +------+---------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic