UDFs & pandas UDFs

Databricks · Apache Spark on Databricks

Theory

A user-defined function (UDF) lets you apply your own Python logic to a column when no built-in function fits. A plain Python UDF (@F.udf or F.udf(func, returnType)) is called once per row: Spark serializes each value, sends it to a Python process, runs your function and sends the result back, which is slow for large data and blocks Photon optimizations. A pandas UDF (@F.pandas_udf) receives whole batches as a pandas Series using Apache Arrow and returns a Series, so it is usually many times faster.

Unity Catalog also supports SQL and Python UDFs created with CREATE FUNCTION main.sales.fn(...), which can be governed and reused in SQL. Always check pyspark.sql.functions first: F.when, F.round, F.regexp_extract and hundreds of others run natively. For example, labelling orders as High when amount >= 1000 is a one-line F.when, not a UDF.

Why we use it

Some business rules are too complex for built-in functions, or reuse existing Python code. UDFs make that possible inside Spark, and pandas UDFs keep it reasonably fast by working on batches.

When to use it

Prefer built-in functions first, then a SQL UDF or pandas UDF, and a row-by-row Python UDF only as a last resort. Use a Unity Catalog function when the logic must be shared and governed across teams.

How it works

  1. Need custom column logic
  2. Can built-in functions (F.when, F.regexp_extract ...) do it? Yes: Use built-in functions. No: Write a UDF.
  3. Can the logic work on a whole pandas Series? Yes: Use a pandas UDF. No: Use a Python UDF (slowest).
  4. Declare the return type and apply it in select or withColumn
  5. Register it in Unity Catalog if others need it

Syntax

@F.udf(returnType="string")  def f(x): ...
@F.pandas_udf("string")  def f(s: pd.Series) -> pd.Series: ...
spark.udf.register("name", f)
CREATE FUNCTION main.sales.fn(x DOUBLE) RETURNS STRING RETURN ...

Example

import pandas as pd
from pyspark.sql import functions as F

@F.pandas_udf("string")
def amount_band(amount: pd.Series) -> pd.Series:
    return amount.apply(lambda a: "High" if a >= 1000 else "Normal")

orders = spark.read.table("main.sales.orders")
display(orders.select("order_id", "amount", amount_band("amount").alias("band")))

# Same result with a built-in (fastest)
orders.withColumn("band", F.when(F.col("amount") >= 1000, "High").otherwise("Normal"))

-- Governed SQL UDF in Unity Catalog
CREATE OR REPLACE FUNCTION main.sales.amount_band(amount DOUBLE)
RETURNS STRING
RETURN CASE WHEN amount >= 1000 THEN 'High' ELSE 'Normal' END;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic