Python UDFs & Why They Are Slow

Apache Spark · UDFs

Theory

A user-defined function (UDF) lets you run your own Python code on each row of a DataFrame. You wrap a normal function with F.udf(func, returnType) or the @F.udf("string") decorator,, then use it like a built-in. The return type must be declared; if the function returns a different type, a classic UDF can silently give null.

The cost is in how it runs. Spark's engine lives in the JVM, but your function lives in Python. For a classic UDF, each executor starts Python worker processes, serialises rows with pickle, sends them across, calls your function one row at a time, then serialises results back. Catalyst cannot see inside the function, so it cannot push filters through it or generate optimised code for it. Arrow-optimised Python UDFs (useArrow=True, or the spark.sql.execution.pythonUDF.arrow.enabled setting) cut the transfer cost by sending columnar Arrow batches, but your function is still called once per row.

Why we use it

UDFs exist for logic Spark has no built-in for, such as a custom checksum or a call into a Python library. Knowing why they are slow (serialisation, row-by-row calls, an optimiser black box) helps you decide when they are worth it and is a very common interview question.

When to use it

Use a Python UDF only when no built-in function or SQL expression can do the job, and the logic is genuinely row-level Python. If the work can be vectorised with pandas or NumPy, use a pandas UDF instead. For simple if/else, string or date logic, use F.when and built-ins.

How it works

  1. Query reaches a column computed by a Python UDF
  2. Executor sends a batch of rows to a Python worker (pickle or Arrow)
  3. Python calls your function once for every row
  4. Results are serialised and sent back to the JVM
  5. Could a built-in or pandas UDF do this? Yes: Replace the UDF and let Catalyst optimise. No: Keep the UDF, but filter rows before it.
  6. Query continues with the UDF column

Syntax

@F.udf(returnType="string")
def band(amount):
    return None if amount is None else ("High" if amount >= 50000 else "Low")

df.withColumn("band", band("amount"))
@F.udf("string", useArrow=True)        # Arrow-optimised transfer
spark.udf.register("band", band)        # use in SQL

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

from pyspark.sql.types import StringType

@F.udf(returnType=StringType())
def price_band(amount):
    if amount is None:                 # UDFs must handle nulls themselves
        return None
    if amount >= 50000:
        return "Premium"
    return "Standard" if amount >= 10000 else "Budget"

# Same function, but rows are moved to Python as Arrow batches
price_band_arrow = F.udf(price_band.func, StringType(), useArrow=True)

result = (orders
    .withColumn("band", price_band("amount"))
    .withColumn("band_arrow", price_band_arrow("amount"))
    .select("order_id", "product", "amount", "band", "band_arrow"))
result.show()

# The plan shows a BatchEvalPython / ArrowEvalPython step: a black box to the optimiser
result.explain()

# Register for SQL
spark.udf.register("price_band", price_band)
orders.createOrReplaceTempView("orders")
spark.sql("SELECT price_band(amount) AS band, COUNT(*) AS n FROM orders GROUP BY 1 ORDER BY 1").show()

# Output:
# +--------+-------+-------+--------+----------+
# |order_id|product| amount|    band|band_arrow|
# +--------+-------+-------+--------+----------+
# |       1| Laptop|55000.0| Premium|   Premium|
# |       2|  Phone|30000.0|Standard|  Standard|
# |       3|  Mouse| 1500.0|  Budget|    Budget|
# |       4| Laptop|52000.0| Premium|   Premium|
# |       5| Tablet|25000.0|Standard|  Standard|
# |       6|  Phone|18000.0|Standard|  Standard|
# |       7|  Mouse| 1000.0|  Budget|    Budget|
# |       8| Laptop|60000.0| Premium|   Premium|
# +--------+-------+-------+--------+----------+
#
# == Physical Plan ==
# *(2) Project [order_id#0, product#3, amount#5, pythonUDF0#30 AS band#9, pythonUDF0#31 AS band_arrow#11]
# +- ArrowEvalPython [price_band(amount#5)#10], [pythonUDF0#31], 101
#    +- BatchEvalPython [price_band(amount#5)#8], [pythonUDF0#30]
#       +- *(1) Project [order_id#0, product#3, amount#5]
#          +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# +--------+---+
# |    band|  n|
# +--------+---+
# |  Budget|  2|
# | Premium|  3|
# |Standard|  3|
# +--------+---+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic