Pandas UDFs (Vectorized) & Arrow

Apache Spark · UDFs

Theory

A pandas UDF, also called a vectorized UDF, receives whole batches of rows as pandas Series instead of one value at a time. You write it with the @F.pandas_udf("double") decorator and Python type hints: a Series-to-Series function such as add_gst(amount: pd.Series) -> pd.Series multiplies thousands of values in one NumPy operation. Spark moves data between the JVM and Python with Apache Arrow, a columnar format that avoids per-row pickling.

The type hints choose the flavour. Series -> Series transforms a column (output must have the same length). Iterator[pd.Series] -> Iterator[pd.Series] works on a stream of batches and is useful when you load something expensive once, like a model. Series -> scalar is a grouped aggregate usable in groupBy().agg() and over windows. Batch size is set by spark.sql.execution.arrow.maxRecordsPerBatch. Arrow also speeds up toPandas() when spark.sql.execution.arrow.pyspark.enabled is true.

Why we use it

Pandas UDFs keep the flexibility of Python while cutting most of the overhead of classic UDFs: Arrow moves columns in bulk and vectorised pandas/NumPy code does the maths in C. They are the standard way to apply Python libraries, such as scoring an ML model, at scale.

When to use it

Use a pandas UDF when you need Python logic that can be written with vectorised pandas or NumPy, or must call a Python library on many rows. Use the iterator form to load a heavy resource once per batch stream. Still prefer built-in functions when they can express the logic.

How it works

  1. Column needs Python logic
  2. Can it be written with vectorised pandas / NumPy? Yes: Write a pandas UDF with type hints. No: Fall back to a row-level Python UDF.
  3. Spark slices the column into Arrow batches
  4. Python receives each batch as a pandas Series
  5. Your function returns a Series (or a scalar for aggregates)
  6. Arrow batches come back as a Spark column

Syntax

import pandas as pd
@F.pandas_udf("double")
def add_gst(amount: pd.Series) -> pd.Series:
    return amount * 1.18

@F.pandas_udf("double")              # Series -> scalar = grouped aggregate
def median_amt(a: pd.Series) -> float:
    return float(a.median())
df.groupBy("region").agg(median_amt("amount"))

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

import pandas as pd
from typing import Iterator

@F.pandas_udf("double")
def add_gst(amount: pd.Series) -> pd.Series:          # Series -> Series
    return (amount * 1.18).round(2)

@F.pandas_udf("double")
def median_amount(amount: pd.Series) -> float:        # Series -> scalar (aggregate)
    return float(amount.median())

@F.pandas_udf("double")
def discounted(batches: Iterator[pd.Series]) -> Iterator[pd.Series]:
    rate = 0.10                                       # load expensive things once here
    for amount in batches:
        yield amount * (1 - rate)

(orders.select("order_id", "amount",
               add_gst("amount").alias("with_gst"),
               discounted("amount").alias("after_10pct_off"))
       .show())

orders.groupBy("region").agg(median_amount("amount").alias("median_amount")).orderBy("region").show()

# Arrow also speeds up toPandas() when this setting is on
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
pdf = orders.groupBy("product").agg(F.sum("qty").alias("units")).orderBy("product").toPandas()
print(pdf)

# Output:
# +--------+-------+--------+---------------+
# |order_id| amount|with_gst|after_10pct_off|
# +--------+-------+--------+---------------+
# |       1|55000.0| 64900.0|        49500.0|
# |       2|30000.0| 35400.0|        27000.0|
# |       3| 1500.0|  1770.0|         1350.0|
# |       4|52000.0| 61360.0|        46800.0|
# |       5|25000.0| 29500.0|        22500.0|
# |       6|18000.0| 21240.0|        16200.0|
# |       7| 1000.0|  1180.0|          900.0|
# |       8|60000.0| 70800.0|        54000.0|
# +--------+-------+--------+---------------+
#
# +------+-------------+
# |region|median_amount|
# +------+-------------+
# | North|      18000.0|
# | South|      26500.0|
# |  West|      30000.0|
# +------+-------------+
#
#   product  units
# 0  Laptop      3
# 1   Mouse      5
# 2   Phone      3
# 3  Tablet      1

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic