mapInPandas & applyInPandas

Apache Spark · UDFs

Theory

Pandas UDFs work on columns; these two APIs hand your function whole pandas DataFrames. df.mapInPandas(func, schema) passes an iterator of pandas DataFrame batches from each partition, and your function yields DataFrames of any length with the declared output schema. It suits row-wise or batch-wise transforms that need several columns at once, filtering, or calling a model on a batch.

df.groupBy("region").applyInPandas(func, schema) is grouped map: Spark shuffles so each group's rows are together, converts each group to one pandas DataFrame, calls func(pdf) and combines the returned DataFrames. That lets you use any pandas logic per group, for example each order's share of its region's sales or fitting a small model per store. All rows of a group must fit in one executor's memory, so a very large or skewed group can cause out-of-memory errors. There is also groupBy().cogroup(other.groupBy()).applyInPandas for two DataFrames at once.

Why we use it

Some logic is far easier in pandas than in Spark expressions: per-group interpolation, custom forecasting, or library calls that need a whole DataFrame. These APIs let you reuse that pandas code while Spark runs it in parallel across partitions or groups.

When to use it

Use mapInPandas for batch transforms whose output size can differ from the input. Use applyInPandas when the logic needs all rows of a group together and groups are reasonably small. If built-in aggregates or window functions can do it, use those instead: they avoid Python and per-group memory limits.

How it works

  1. Logic easiest to write in pandas on whole DataFrames
  2. Does it need all rows of a group together? Yes: groupBy(key).applyInPandas(func, schema). No: df.mapInPandas(func, schema) on batches.
  3. Spark converts data to pandas DataFrames with Arrow
  4. Your function returns pandas DataFrames matching the schema
  5. Results are combined into a Spark DataFrame

Syntax

def f(batches):                     # Iterator[pd.DataFrame]
    for pdf in batches:
        yield pdf[pdf.amount > 20000]
df.mapInPandas(f, schema=df.schema)

def g(pdf):                         # one group as pd.DataFrame
    return pdf.assign(share=pdf.amount / pdf.amount.sum())
df.groupBy("region").applyInPandas(g, schema="region string, amount double, share double")

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

import pandas as pd
from typing import Iterator

# mapInPandas: batch transform, output length can change
def big_orders_with_gst(batches: Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]:
    for pdf in batches:
        out = pdf[pdf["amount"] >= 20000][["order_id", "product", "amount"]].copy()
        out["with_gst"] = (out["amount"] * 1.18).round(2)
        yield out

orders.mapInPandas(big_orders_with_gst,
                   schema="order_id int, product string, amount double, with_gst double").show()

# applyInPandas: one pandas DataFrame per region
def region_share(pdf: pd.DataFrame) -> pd.DataFrame:
    pdf = pdf[["region", "order_id", "amount"]].copy()
    pdf["share_pct"] = (100 * pdf["amount"] / pdf["amount"].sum()).round(1)
    pdf["rank_in_region"] = pdf["amount"].rank(ascending=False).astype("int32")
    return pdf

(orders.groupBy("region")
       .applyInPandas(region_share,
                      schema="region string, order_id int, amount double, share_pct double, rank_in_region int")
       .orderBy("region", "rank_in_region")
       .show())

# Output:
# +--------+-------+-------+--------+
# |order_id|product| amount|with_gst|
# +--------+-------+-------+--------+
# |       1| Laptop|55000.0| 64900.0|
# |       2|  Phone|30000.0| 35400.0|
# |       4| Laptop|52000.0| 61360.0|
# |       5| Tablet|25000.0| 29500.0|
# |       8| Laptop|60000.0| 70800.0|
# +--------+-------+-------+--------+
#
# +------+--------+-------+---------+--------------+
# |region|order_id| amount|share_pct|rank_in_region|
# +------+--------+-------+---------+--------------+
# | North|       1|55000.0|     73.8|             1|
# | North|       6|18000.0|     24.2|             2|
# | North|       3| 1500.0|      2.0|             3|
# | South|       4|52000.0|     98.1|             1|
# | South|       7| 1000.0|      1.9|             2|
# |  West|       8|60000.0|     52.2|             1|
# |  West|       2|30000.0|     26.1|             2|
# |  West|       5|25000.0|     21.7|             3|
# +------+--------+-------+---------+--------------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic