How PySpark works: Py4J, the JVM & Python workers

Apache Spark · Spark Architecture

Theory

Spark itself is written in Scala and runs on the Java Virtual Machine (JVM). When you start PySpark, your Python process launches a JVM driver and talks to it through Py4J, a bridge that lets Python call Java objects. Your SparkSession and DataFrames in Python are thin wrappers: df.filter(...) just asks the JVM to add a step to the plan, so built-in DataFrame functions run entirely inside the JVM at full speed.

Python code only runs on the cluster when you ship it there: RDD lambdas, Python UDFs and pandas UDFs. For those, each executor starts Python worker processes, sends them rows, and reads the results back. Plain Python UDFs move rows in pickled batches; pandas UDFs, toPandas() and createDataFrame(pandas_df) can use Apache Arrow, a columnar format that both sides read without converting row by row. For example, F.upper('city') never leaves the JVM, while a udf(lambda s: s.upper()) sends every city to Python and back.

Why we use it

This explains the single most important PySpark performance rule: prefer built-in functions over Python UDFs and RDD lambdas. It also explains errors you will meet, such as Py4JJavaError (a JVM exception surfaced in Python) and Python worker crashes caused by memory overhead.

When to use it

Think about it whenever you write a UDF, call toPandas(), or see Py4JJavaError in a stack trace. Use built-in functions first, pandas UDFs when you truly need Python logic, and enable Arrow for pandas conversions. With Spark Connect the client talks gRPC instead of Py4J.

How it works

  1. Python script calls a DataFrame method
  2. Py4J forwards the call to the JVM driver
  3. JVM builds and optimises the plan; an action runs tasks on executors
  4. Plan has Python code (UDF/lambda)? Yes: Rows go to Python workers (pickle/Arrow). No: Tasks stay in the JVM.
  5. Results come back to the JVM driver
  6. Py4J hands the result objects to Python

Syntax

sc._jvm.java.lang.Math.max(3, 7)        # call Java via Py4J (internal API)
df._jdf                                   # the JVM Dataset behind a DataFrame
df.explain()   # BatchEvalPython = Python UDF, ArrowEvalPython = pandas UDF
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

Example

import os
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import udf, pandas_udf

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())
sc = spark.sparkContext

orders = spark.createDataFrame(
    [(101, "North", 55000.0), (102, "West", 30000.0), (103, "North", 1500.0), (104, "South", 52000.0)],
    "order_id INT, region STRING, amount DOUBLE")

# 1) Py4J: the Python SparkSession is a thin proxy over JVM objects
print("Java Math.max(3, 7) via Py4J:", sc._jvm.java.lang.Math.max(3, 7))
print("JVM class behind the DataFrame:", orders._jdf.getClass().getName())

# 2) DataFrame functions run inside the JVM; no Python process touches the rows
plan = orders.withColumn("gst", F.col("amount") * 0.18)._jdf.queryExecution().executedPlan().toString()
print("built-in expression needs Python?", "Python" in plan)

# 3) A Python UDF ships rows to separate Python worker processes (pickled batches)
gst_udf = udf(lambda a: a * 0.18, "double")
plan = orders.withColumn("gst", gst_udf("amount"))._jdf.queryExecution().executedPlan().toString()
print("Python UDF operator:", "BatchEvalPython" in plan)

# 4) A pandas UDF moves the same data with Apache Arrow (columnar batches)
@pandas_udf("double")
def gst_vec(a):
    return a * 0.18
plan = orders.withColumn("gst", gst_vec("amount"))._jdf.queryExecution().executedPlan().toString()
print("pandas UDF operator:", "ArrowEvalPython" in plan)

# 5) RDD lambdas also run in Python workers, not in the driver process
worker_pids = set(sc.parallelize(range(4), 2).map(lambda _: os.getpid()).collect())
print("lambdas ran outside the driver process:", os.getpid() not in worker_pids)

# 6) Arrow also speeds up toPandas() / createDataFrame(pandas_df)
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
print(orders.toPandas().groupby("region")["amount"].sum().sort_index().to_dict())

# Output:
# Java Math.max(3, 7) via Py4J: 7
# JVM class behind the DataFrame: org.apache.spark.sql.classic.Dataset
# built-in expression needs Python? False
# Python UDF operator: True
# pandas UDF operator: True
# lambdas ran outside the driver process: True
# {'North': 56500.0, 'South': 52000.0, 'West': 30000.0}

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic