Shared variables: broadcast variables & accumulators
Apache Spark · RDDs
Theory
Normally every variable your function uses is copied into each task's closure. Spark offers two special shared variables for the common exceptions. A broadcast variable, created with sc.broadcast(value), sends a read-only value such as a lookup dictionary to each executor once, where all tasks read it through .value. That beats shipping a 1 MB dictionary inside every one of thousands of tasks.
An accumulator, created with sc.accumulator(0), is a write-only counter for tasks: they can only call add(), and only the driver reads .value after an action. It is handy for counting bad records or skipped rows. Updates made inside actions such as foreach() are applied exactly once, but updates inside transformations such as map() are applied again whenever that stage is recomputed or retried. For example, a map that counts unknown cities reports 2 after the first action and 4 after a second action on the same uncached RDD.
Why we use it
Broadcasting a lookup table is a simple, big speed-up for RDD code and UDFs, and accumulators give cheap job-level counters without an extra pass. Both are standard interview questions, especially the accumulator double-counting gotcha.
When to use it
Broadcast small, read-only lookups (a few MB) used by many tasks; for joining two DataFrames use a broadcast join instead. Use accumulators for diagnostics and counters, updated inside actions when the number must be exact. Never use them to pass data between tasks.
How it works
- Driver creates bc = sc.broadcast(lookup) and acc = sc.accumulator(0)
- Tasks ship with a small handle, not the whole lookup
- Each executor fetches the broadcast once and caches it
- Tasks read bc.value and call acc.add() for bad rows
- Stage recomputed or retried? Yes: map() updates are added again. No: Each update counted once.
- Driver reads acc.value after the action
Syntax
bc = sc.broadcast({"Jaipur": "Rajasthan"})
rdd.map(lambda c: bc.value.get(c))
bc.unpersist(); bc.destroy()
acc = sc.accumulator(0)
rdd.foreach(lambda x: acc.add(1))
print(acc.value) # driver only, after an actionExample
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.getOrCreate())
sc = spark.sparkContext
# 1) Broadcast variable: one read-only copy per executor instead of one per task
city_state = sc.broadcast({"Jaipur": "Rajasthan", "Pune": "Maharashtra", "Delhi": "Delhi"})
# 2) Accumulator: tasks can only add; the driver reads the total
bad_rows = sc.accumulator(0)
def parse(line):
city, amount = line.split(",")
state = city_state.value.get(city) # .value reads the local copy
if state is None:
bad_rows.add(1)
return city, state, float(amount)
lines = sc.parallelize(["Jaipur,55000", "Pune,30000", "Surat,1500", "Delhi,52000", "Agra,900"], 2)
rows = lines.map(parse).filter(lambda r: r[1] is not None)
print(rows.collect()) # the action runs the tasks
print("unknown cities:", bad_rows.value) # read only after an action
# 3) Gotcha: an accumulator updated inside a transformation counts again on recompute
rows.count()
print("after a second action:", bad_rows.value)
# 4) Safe pattern: update accumulators inside an action such as foreach()
seen = sc.accumulator(0)
lines.foreach(lambda _: seen.add(1))
print("lines seen:", seen.value)
# 5) Broadcasts also work inside DataFrame UDFs
@F.udf("string")
def to_state(city):
return city_state.value.get(city, "Unknown")
df = spark.createDataFrame([("Jaipur",), ("Agra",)], "city STRING")
df.withColumn("state", to_state("city")).show()
city_state.unpersist() # drop executor copies (destroy() also frees the driver copy)
# Output:
# [('Jaipur', 'Rajasthan', 55000.0), ('Pune', 'Maharashtra', 30000.0), ('Delhi', 'Delhi', 52000.0)]
# unknown cities: 2
# after a second action: 4
# lines seen: 5
# +------+---------+
# | city| state|
# +------+---------+
# |Jaipur|Rajasthan|
# | Agra| Unknown|
# +------+---------+
Key points
- Broadcast: one read-only copy per executor, read with .value
- Accumulator: tasks add, the driver reads .value after an action
- Accumulator updates in transformations can be counted twice on recompute
- Update accumulators inside actions (foreach) when the count must be exact
- Broadcast DataFrame joins use F.broadcast(df), not sc.broadcast
- unpersist() frees executor copies; destroy() frees everything