Pandas UDFs (Vectorized) & Arrow
Apache Spark · UDFs
Theory
A pandas UDF, also called a vectorized UDF, receives whole batches of rows as pandas Series instead of one value at a time. You write it with the @F.pandas_udf("double") decorator and Python type hints: a Series-to-Series function such as add_gst(amount: pd.Series) -> pd.Series multiplies thousands of values in one NumPy operation. Spark moves data between the JVM and Python with Apache Arrow, a columnar format that avoids per-row pickling.
The type hints choose the flavour. Series -> Series transforms a column (output must have the same length). Iterator[pd.Series] -> Iterator[pd.Series] works on a stream of batches and is useful when you load something expensive once, like a model. Series -> scalar is a grouped aggregate usable in groupBy().agg() and over windows. Batch size is set by spark.sql.execution.arrow.maxRecordsPerBatch. Arrow also speeds up toPandas() when spark.sql.execution.arrow.pyspark.enabled is true.
Why we use it
Pandas UDFs keep the flexibility of Python while cutting most of the overhead of classic UDFs: Arrow moves columns in bulk and vectorised pandas/NumPy code does the maths in C. They are the standard way to apply Python libraries, such as scoring an ML model, at scale.
When to use it
Use a pandas UDF when you need Python logic that can be written with vectorised pandas or NumPy, or must call a Python library on many rows. Use the iterator form to load a heavy resource once per batch stream. Still prefer built-in functions when they can express the logic.
How it works
- Column needs Python logic
- Can it be written with vectorised pandas / NumPy? Yes: Write a pandas UDF with type hints. No: Fall back to a row-level Python UDF.
- Spark slices the column into Arrow batches
- Python receives each batch as a pandas Series
- Your function returns a Series (or a scalar for aggregates)
- Arrow batches come back as a Spark column
Syntax
import pandas as pd
@F.pandas_udf("double")
def add_gst(amount: pd.Series) -> pd.Series:
return amount * 1.18
@F.pandas_udf("double") # Series -> scalar = grouped aggregate
def median_amt(a: pd.Series) -> float:
return float(a.median())
df.groupBy("region").agg(median_amt("amount"))Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
import pandas as pd
from typing import Iterator
@F.pandas_udf("double")
def add_gst(amount: pd.Series) -> pd.Series: # Series -> Series
return (amount * 1.18).round(2)
@F.pandas_udf("double")
def median_amount(amount: pd.Series) -> float: # Series -> scalar (aggregate)
return float(amount.median())
@F.pandas_udf("double")
def discounted(batches: Iterator[pd.Series]) -> Iterator[pd.Series]:
rate = 0.10 # load expensive things once here
for amount in batches:
yield amount * (1 - rate)
(orders.select("order_id", "amount",
add_gst("amount").alias("with_gst"),
discounted("amount").alias("after_10pct_off"))
.show())
orders.groupBy("region").agg(median_amount("amount").alias("median_amount")).orderBy("region").show()
# Arrow also speeds up toPandas() when this setting is on
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
pdf = orders.groupBy("product").agg(F.sum("qty").alias("units")).orderBy("product").toPandas()
print(pdf)
# Output:
# +--------+-------+--------+---------------+
# |order_id| amount|with_gst|after_10pct_off|
# +--------+-------+--------+---------------+
# | 1|55000.0| 64900.0| 49500.0|
# | 2|30000.0| 35400.0| 27000.0|
# | 3| 1500.0| 1770.0| 1350.0|
# | 4|52000.0| 61360.0| 46800.0|
# | 5|25000.0| 29500.0| 22500.0|
# | 6|18000.0| 21240.0| 16200.0|
# | 7| 1000.0| 1180.0| 900.0|
# | 8|60000.0| 70800.0| 54000.0|
# +--------+-------+--------+---------------+
#
# +------+-------------+
# |region|median_amount|
# +------+-------------+
# | North| 18000.0|
# | South| 26500.0|
# | West| 30000.0|
# +------+-------------+
#
# product units
# 0 Laptop 3
# 1 Mouse 5
# 2 Phone 3
# 3 Tablet 1
Key points
- Pandas UDFs get pandas Series batches via Arrow, not single rows
- Type hints pick the kind: Series->Series, Iterator->Iterator, Series->scalar
- Series->Series output must have the same length as the input
- Iterator form lets you load a model or lookup once and reuse it
- Needs pyarrow and pandas installed on the driver and every executor