Serialization: Java vs Kryo
Apache Spark · Performance Tuning
Theory
Serialization turns objects into bytes so they can be sent over the network, written in a shuffle or cached in serialized form. On the JVM, Spark uses Java serialization by default (spark.serializer=JavaSerializer); setting it to KryoSerializer is usually faster and more compact for RDDs of JVM objects, especially if classes are registered. In PySpark, Python data is always pickled: functions and the variables they capture are serialized with cloudpickle and sent with every task.
DataFrames avoid most of this cost. Their rows are stored in Spark's own compact binary format (Tungsten UnsafeRow) and the shuffle writes those bytes directly, so the Java vs Kryo choice matters little for DataFrame code. Serialization still bites in PySpark through closures: a lambda that captures a 50,000-entry dictionary pickles to almost 1 MB per task, while capturing a broadcast handle is under 1 KB. Captured objects must be picklable; a lock or open connection raises a PicklingError.
Why we use it
Serialization overhead and 'Task not serializable' / PicklingError failures are common in RDD and UDF code, and interviewers ask why Kryo is faster and why DataFrames are faster than RDDs. Understanding it also explains why broadcasts help.
When to use it
Switch to Kryo for heavy RDD workloads on JVM objects. In PySpark, focus on closures instead: broadcast big lookups, create connections inside mapPartitions rather than capturing them, and prefer DataFrame functions so rows stay in Tungsten format. Kryo settings must be set before the session starts.
How it works
- Task needs to be shipped to an executor
- Driver pickles the Python function and everything it captures
- Captures a large object? Yes: Broadcast it: shipped once per executor. No: Small closure sent.
- DataFrame with built-in functions? Yes: Rows stay in Tungsten format. No: Rows pickled in batches.
- Shuffles and serialized caches of JVM objects use spark.serializer
- Executor deserializes and runs the task
Syntax
SparkSession.builder
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.kryoserializer.buffer.max", "128m")
.config("spark.kryo.registrationRequired", "false")
from pyspark import cloudpickle; len(cloudpickle.dumps(fn)) # closure size
bc = sc.broadcast(big_lookup) # ship onceExample
from pyspark import cloudpickle
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.kryoserializer.buffer.max", "128m")
.getOrCreate())
sc = spark.sparkContext
# 1) JVM side: which serializer is used for shuffles / serialized caching of JVM objects
print("JVM serializer:", sc.getConf().get("spark.serializer").split(".")[-1])
# 2) Python side: Python objects are always pickled (cloudpickle for functions)
print("Python serializer:", type(sc.serializer).__name__)
# 3) Closures are serialized and sent with EVERY task. A big captured object is costly.
prices = {f"SKU{i}": float(i) for i in range(50000)}
with_dict = lambda sku: prices.get(sku, 0.0)
print("closure that captures the dict:", len(cloudpickle.dumps(with_dict)) // 1024, "KB")
bc = sc.broadcast(prices) # sent once per executor instead
with_bc = lambda sku: bc.value.get(sku, 0.0)
print("closure that captures a broadcast is under 1 KB:", len(cloudpickle.dumps(with_bc)) < 1024)
print(sc.parallelize(["SKU7", "SKU42"]).map(with_bc).collect())
# 4) DataFrames keep rows in Spark's binary (Tungsten) format inside the JVM.
# Built-in functions never pickle rows; only Python UDFs ship rows to Python.
df = spark.createDataFrame([(1, "Jaipur", 55000.0), (2, "Pune", 30000.0)],
"id INT, city STRING, amount DOUBLE")
def pickles_rows(d):
return "EvalPython" in d._jdf.queryExecution().executedPlan().toString()
print("built-in upper() pickles rows:", pickles_rows(df.select(F.upper("city"))))
print("Python UDF pickles rows :", pickles_rows(df.select(F.udf(str.upper, "string")("city"))))
# 5) Things that must be picklable: functions, captured objects, RDD elements
try:
import threading
lock = threading.Lock()
sc.parallelize([1]).map(lambda x: (x, lock)).collect()
except Exception as e:
print("unpicklable object in a closure ->", type(e).__name__)
# Output:
# JVM serializer: KryoSerializer
# Python serializer: AutoBatchedSerializer
# closure that captures the dict: 966 KB
# closure that captures a broadcast is under 1 KB: True
# [7.0, 42.0]
# built-in upper() pickles rows: False
# Python UDF pickles rows : True
# unpicklable object in a closure -> PicklingError
Key points
- JVM default is Java serialization; Kryo is faster and smaller
- Serializer is a startup setting: set it before the session is created
- PySpark always pickles Python objects; closures use cloudpickle
- DataFrames keep rows in Tungsten binary format: little serializer impact
- Big captured objects are re-sent with every task: broadcast them
- Unpicklable captures (locks, connections) raise PicklingError