Python UDFs & Why They Are Slow
Apache Spark · UDFs
Theory
A user-defined function (UDF) lets you run your own Python code on each row of a DataFrame. You wrap a normal function with F.udf(func, returnType) or the @F.udf("string") decorator,, then use it like a built-in. The return type must be declared; if the function returns a different type, a classic UDF can silently give null.
The cost is in how it runs. Spark's engine lives in the JVM, but your function lives in Python. For a classic UDF, each executor starts Python worker processes, serialises rows with pickle, sends them across, calls your function one row at a time, then serialises results back. Catalyst cannot see inside the function, so it cannot push filters through it or generate optimised code for it. Arrow-optimised Python UDFs (useArrow=True, or the spark.sql.execution.pythonUDF.arrow.enabled setting) cut the transfer cost by sending columnar Arrow batches, but your function is still called once per row.
Why we use it
UDFs exist for logic Spark has no built-in for, such as a custom checksum or a call into a Python library. Knowing why they are slow (serialisation, row-by-row calls, an optimiser black box) helps you decide when they are worth it and is a very common interview question.
When to use it
Use a Python UDF only when no built-in function or SQL expression can do the job, and the logic is genuinely row-level Python. If the work can be vectorised with pandas or NumPy, use a pandas UDF instead. For simple if/else, string or date logic, use F.when and built-ins.
How it works
- Query reaches a column computed by a Python UDF
- Executor sends a batch of rows to a Python worker (pickle or Arrow)
- Python calls your function once for every row
- Results are serialised and sent back to the JVM
- Could a built-in or pandas UDF do this? Yes: Replace the UDF and let Catalyst optimise. No: Keep the UDF, but filter rows before it.
- Query continues with the UDF column
Syntax
@F.udf(returnType="string")
def band(amount):
return None if amount is None else ("High" if amount >= 50000 else "Low")
df.withColumn("band", band("amount"))
@F.udf("string", useArrow=True) # Arrow-optimised transfer
spark.udf.register("band", band) # use in SQLExample
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
from pyspark.sql.types import StringType
@F.udf(returnType=StringType())
def price_band(amount):
if amount is None: # UDFs must handle nulls themselves
return None
if amount >= 50000:
return "Premium"
return "Standard" if amount >= 10000 else "Budget"
# Same function, but rows are moved to Python as Arrow batches
price_band_arrow = F.udf(price_band.func, StringType(), useArrow=True)
result = (orders
.withColumn("band", price_band("amount"))
.withColumn("band_arrow", price_band_arrow("amount"))
.select("order_id", "product", "amount", "band", "band_arrow"))
result.show()
# The plan shows a BatchEvalPython / ArrowEvalPython step: a black box to the optimiser
result.explain()
# Register for SQL
spark.udf.register("price_band", price_band)
orders.createOrReplaceTempView("orders")
spark.sql("SELECT price_band(amount) AS band, COUNT(*) AS n FROM orders GROUP BY 1 ORDER BY 1").show()
# Output:
# +--------+-------+-------+--------+----------+
# |order_id|product| amount| band|band_arrow|
# +--------+-------+-------+--------+----------+
# | 1| Laptop|55000.0| Premium| Premium|
# | 2| Phone|30000.0|Standard| Standard|
# | 3| Mouse| 1500.0| Budget| Budget|
# | 4| Laptop|52000.0| Premium| Premium|
# | 5| Tablet|25000.0|Standard| Standard|
# | 6| Phone|18000.0|Standard| Standard|
# | 7| Mouse| 1000.0| Budget| Budget|
# | 8| Laptop|60000.0| Premium| Premium|
# +--------+-------+-------+--------+----------+
#
# == Physical Plan ==
# *(2) Project [order_id#0, product#3, amount#5, pythonUDF0#30 AS band#9, pythonUDF0#31 AS band_arrow#11]
# +- ArrowEvalPython [price_band(amount#5)#10], [pythonUDF0#31], 101
# +- BatchEvalPython [price_band(amount#5)#8], [pythonUDF0#30]
# +- *(1) Project [order_id#0, product#3, amount#5]
# +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# +--------+---+
# | band| n|
# +--------+---+
# | Budget| 2|
# | Premium| 3|
# |Standard| 3|
# +--------+---+
Key points
- Rows move JVM -> Python worker -> JVM, with serialisation both ways
- The function runs once per row, and Catalyst cannot optimise inside it
- Declare the right return type: a mismatch gives null, not an error
- Handle None inside the function; nulls are passed in as None
- useArrow=True speeds up the data transfer but not the per-row calls