UDFs & pandas UDFs
Databricks · Apache Spark on Databricks
Theory
A user-defined function (UDF) lets you apply your own Python logic to a column when no built-in function fits. A plain Python UDF (@F.udf or F.udf(func, returnType)) is called once per row: Spark serializes each value, sends it to a Python process, runs your function and sends the result back, which is slow for large data and blocks Photon optimizations. A pandas UDF (@F.pandas_udf) receives whole batches as a pandas Series using Apache Arrow and returns a Series, so it is usually many times faster.
Unity Catalog also supports SQL and Python UDFs created with CREATE FUNCTION main.sales.fn(...), which can be governed and reused in SQL. Always check pyspark.sql.functions first: F.when, F.round, F.regexp_extract and hundreds of others run natively. For example, labelling orders as High when amount >= 1000 is a one-line F.when, not a UDF.
Why we use it
Some business rules are too complex for built-in functions, or reuse existing Python code. UDFs make that possible inside Spark, and pandas UDFs keep it reasonably fast by working on batches.
When to use it
Prefer built-in functions first, then a SQL UDF or pandas UDF, and a row-by-row Python UDF only as a last resort. Use a Unity Catalog function when the logic must be shared and governed across teams.
How it works
- Need custom column logic
- Can built-in functions (F.when, F.regexp_extract ...) do it? Yes: Use built-in functions. No: Write a UDF.
- Can the logic work on a whole pandas Series? Yes: Use a pandas UDF. No: Use a Python UDF (slowest).
- Declare the return type and apply it in select or withColumn
- Register it in Unity Catalog if others need it
Syntax
@F.udf(returnType="string") def f(x): ...
@F.pandas_udf("string") def f(s: pd.Series) -> pd.Series: ...
spark.udf.register("name", f)
CREATE FUNCTION main.sales.fn(x DOUBLE) RETURNS STRING RETURN ...Example
import pandas as pd
from pyspark.sql import functions as F
@F.pandas_udf("string")
def amount_band(amount: pd.Series) -> pd.Series:
return amount.apply(lambda a: "High" if a >= 1000 else "Normal")
orders = spark.read.table("main.sales.orders")
display(orders.select("order_id", "amount", amount_band("amount").alias("band")))
# Same result with a built-in (fastest)
orders.withColumn("band", F.when(F.col("amount") >= 1000, "High").otherwise("Normal"))
-- Governed SQL UDF in Unity Catalog
CREATE OR REPLACE FUNCTION main.sales.amount_band(amount DOUBLE)
RETURNS STRING
RETURN CASE WHEN amount >= 1000 THEN 'High' ELSE 'Normal' END;Key points
- Built-in functions are fastest and work with Photon; check them before writing a UDF
- Python UDFs run row by row and serialize data to a Python process
- pandas UDFs process Arrow batches as pandas Series and are much faster
- Always declare the return type
- Unity Catalog functions (CREATE FUNCTION) are reusable and permissioned