Serialization: Java vs Kryo

Apache Spark · Performance Tuning

Theory

Serialization turns objects into bytes so they can be sent over the network, written in a shuffle or cached in serialized form. On the JVM, Spark uses Java serialization by default (spark.serializer=JavaSerializer); setting it to KryoSerializer is usually faster and more compact for RDDs of JVM objects, especially if classes are registered. In PySpark, Python data is always pickled: functions and the variables they capture are serialized with cloudpickle and sent with every task.

DataFrames avoid most of this cost. Their rows are stored in Spark's own compact binary format (Tungsten UnsafeRow) and the shuffle writes those bytes directly, so the Java vs Kryo choice matters little for DataFrame code. Serialization still bites in PySpark through closures: a lambda that captures a 50,000-entry dictionary pickles to almost 1 MB per task, while capturing a broadcast handle is under 1 KB. Captured objects must be picklable; a lock or open connection raises a PicklingError.

Why we use it

Serialization overhead and 'Task not serializable' / PicklingError failures are common in RDD and UDF code, and interviewers ask why Kryo is faster and why DataFrames are faster than RDDs. Understanding it also explains why broadcasts help.

When to use it

Switch to Kryo for heavy RDD workloads on JVM objects. In PySpark, focus on closures instead: broadcast big lookups, create connections inside mapPartitions rather than capturing them, and prefer DataFrame functions so rows stay in Tungsten format. Kryo settings must be set before the session starts.

How it works

  1. Task needs to be shipped to an executor
  2. Driver pickles the Python function and everything it captures
  3. Captures a large object? Yes: Broadcast it: shipped once per executor. No: Small closure sent.
  4. DataFrame with built-in functions? Yes: Rows stay in Tungsten format. No: Rows pickled in batches.
  5. Shuffles and serialized caches of JVM objects use spark.serializer
  6. Executor deserializes and runs the task

Syntax

SparkSession.builder
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .config("spark.kryoserializer.buffer.max", "128m")
  .config("spark.kryo.registrationRequired", "false")
from pyspark import cloudpickle; len(cloudpickle.dumps(fn))   # closure size
bc = sc.broadcast(big_lookup)                                  # ship once

Example

from pyspark import cloudpickle
from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
         .config("spark.kryoserializer.buffer.max", "128m")
         .getOrCreate())
sc = spark.sparkContext

# 1) JVM side: which serializer is used for shuffles / serialized caching of JVM objects
print("JVM serializer:", sc.getConf().get("spark.serializer").split(".")[-1])

# 2) Python side: Python objects are always pickled (cloudpickle for functions)
print("Python serializer:", type(sc.serializer).__name__)

# 3) Closures are serialized and sent with EVERY task. A big captured object is costly.
prices = {f"SKU{i}": float(i) for i in range(50000)}
with_dict = lambda sku: prices.get(sku, 0.0)
print("closure that captures the dict:", len(cloudpickle.dumps(with_dict)) // 1024, "KB")

bc = sc.broadcast(prices)                       # sent once per executor instead
with_bc = lambda sku: bc.value.get(sku, 0.0)
print("closure that captures a broadcast is under 1 KB:", len(cloudpickle.dumps(with_bc)) < 1024)
print(sc.parallelize(["SKU7", "SKU42"]).map(with_bc).collect())

# 4) DataFrames keep rows in Spark's binary (Tungsten) format inside the JVM.
#    Built-in functions never pickle rows; only Python UDFs ship rows to Python.
df = spark.createDataFrame([(1, "Jaipur", 55000.0), (2, "Pune", 30000.0)],
                           "id INT, city STRING, amount DOUBLE")
def pickles_rows(d):
    return "EvalPython" in d._jdf.queryExecution().executedPlan().toString()
print("built-in upper() pickles rows:", pickles_rows(df.select(F.upper("city"))))
print("Python UDF pickles rows     :", pickles_rows(df.select(F.udf(str.upper, "string")("city"))))

# 5) Things that must be picklable: functions, captured objects, RDD elements
try:
    import threading
    lock = threading.Lock()
    sc.parallelize([1]).map(lambda x: (x, lock)).collect()
except Exception as e:
    print("unpicklable object in a closure ->", type(e).__name__)

# Output:
# JVM serializer: KryoSerializer
# Python serializer: AutoBatchedSerializer
# closure that captures the dict: 966 KB
# closure that captures a broadcast is under 1 KB: True
# [7.0, 42.0]
# built-in upper() pickles rows: False
# Python UDF pickles rows     : True
# unpicklable object in a closure -> PicklingError

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic