mapInPandas & applyInPandas
Apache Spark · UDFs
Theory
Pandas UDFs work on columns; these two APIs hand your function whole pandas DataFrames. df.mapInPandas(func, schema) passes an iterator of pandas DataFrame batches from each partition, and your function yields DataFrames of any length with the declared output schema. It suits row-wise or batch-wise transforms that need several columns at once, filtering, or calling a model on a batch.
df.groupBy("region").applyInPandas(func, schema) is grouped map: Spark shuffles so each group's rows are together, converts each group to one pandas DataFrame, calls func(pdf) and combines the returned DataFrames. That lets you use any pandas logic per group, for example each order's share of its region's sales or fitting a small model per store. All rows of a group must fit in one executor's memory, so a very large or skewed group can cause out-of-memory errors. There is also groupBy().cogroup(other.groupBy()).applyInPandas for two DataFrames at once.
Why we use it
Some logic is far easier in pandas than in Spark expressions: per-group interpolation, custom forecasting, or library calls that need a whole DataFrame. These APIs let you reuse that pandas code while Spark runs it in parallel across partitions or groups.
When to use it
Use mapInPandas for batch transforms whose output size can differ from the input. Use applyInPandas when the logic needs all rows of a group together and groups are reasonably small. If built-in aggregates or window functions can do it, use those instead: they avoid Python and per-group memory limits.
How it works
- Logic easiest to write in pandas on whole DataFrames
- Does it need all rows of a group together? Yes: groupBy(key).applyInPandas(func, schema). No: df.mapInPandas(func, schema) on batches.
- Spark converts data to pandas DataFrames with Arrow
- Your function returns pandas DataFrames matching the schema
- Results are combined into a Spark DataFrame
Syntax
def f(batches): # Iterator[pd.DataFrame]
for pdf in batches:
yield pdf[pdf.amount > 20000]
df.mapInPandas(f, schema=df.schema)
def g(pdf): # one group as pd.DataFrame
return pdf.assign(share=pdf.amount / pdf.amount.sum())
df.groupBy("region").applyInPandas(g, schema="region string, amount double, share double")Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
import pandas as pd
from typing import Iterator
# mapInPandas: batch transform, output length can change
def big_orders_with_gst(batches: Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]:
for pdf in batches:
out = pdf[pdf["amount"] >= 20000][["order_id", "product", "amount"]].copy()
out["with_gst"] = (out["amount"] * 1.18).round(2)
yield out
orders.mapInPandas(big_orders_with_gst,
schema="order_id int, product string, amount double, with_gst double").show()
# applyInPandas: one pandas DataFrame per region
def region_share(pdf: pd.DataFrame) -> pd.DataFrame:
pdf = pdf[["region", "order_id", "amount"]].copy()
pdf["share_pct"] = (100 * pdf["amount"] / pdf["amount"].sum()).round(1)
pdf["rank_in_region"] = pdf["amount"].rank(ascending=False).astype("int32")
return pdf
(orders.groupBy("region")
.applyInPandas(region_share,
schema="region string, order_id int, amount double, share_pct double, rank_in_region int")
.orderBy("region", "rank_in_region")
.show())
# Output:
# +--------+-------+-------+--------+
# |order_id|product| amount|with_gst|
# +--------+-------+-------+--------+
# | 1| Laptop|55000.0| 64900.0|
# | 2| Phone|30000.0| 35400.0|
# | 4| Laptop|52000.0| 61360.0|
# | 5| Tablet|25000.0| 29500.0|
# | 8| Laptop|60000.0| 70800.0|
# +--------+-------+-------+--------+
#
# +------+--------+-------+---------+--------------+
# |region|order_id| amount|share_pct|rank_in_region|
# +------+--------+-------+---------+--------------+
# | North| 1|55000.0| 73.8| 1|
# | North| 6|18000.0| 24.2| 2|
# | North| 3| 1500.0| 2.0| 3|
# | South| 4|52000.0| 98.1| 1|
# | South| 7| 1000.0| 1.9| 2|
# | West| 8|60000.0| 52.2| 1|
# | West| 2|30000.0| 26.1| 2|
# | West| 5|25000.0| 21.7| 3|
# +------+--------+-------+---------+--------------+
Key points
- mapInPandas: iterator of DataFrame batches in, any number of rows out
- applyInPandas: one full pandas DataFrame per group, after a shuffle
- You must declare the output schema; column names must match it
- Each group must fit in one executor's memory: watch out for skewed groups
- Prefer built-in aggregates and windows when they can do the same job