Built-in Functions First

Apache Spark · UDFs

Theory

Before writing any UDF, check pyspark.sql.functions and Spark SQL's built-in functions. They cover conditions, maths, strings, regex, dates, arrays, maps, JSON, hashing and many aggregates. Built-ins run inside the JVM on Spark's internal row format, so there is no Python process or serialisation, and Catalyst understands them: it can fold constants, push filters down to the data source, prune columns and generate compiled code for whole stages.

A typical case: a UDF that labels orders "Premium", "Standard" or "Budget" by amount is the same as F.when(F.col("amount") >= 50000, "Premium").when(F.col("amount") >= 10000, "Standard").otherwise("Budget"). The built-in version handles nulls, shows up in explain() as a normal expression and is usually many times faster. Higher-order functions (transform, filter, exists, aggregate) and F.expr("SQL text") cover a lot of logic people reach for UDFs to solve.

Why we use it

Replacing UDFs with built-ins is one of the easiest big performance wins in PySpark. It also keeps code simpler, null handling consistent and plans visible in explain(), so the optimiser and readers can both understand the query.

When to use it

Always try built-ins, F.expr SQL expressions and higher-order functions first. Reach for a pandas UDF only when the logic needs a Python library or cannot be expressed in Spark functions, and use a plain Python UDF as the last option.

How it works

  1. You need a new derived column
  2. Is there a built-in function or SQL expression for it? Yes: Use it (F.*, F.expr, higher-order functions). No: Check if it can be vectorised.
  3. Can pandas / NumPy do it on whole columns? Yes: Write a pandas UDF. No: Write a Python UDF as the last option.
  4. Check explain(): built-ins appear as normal expressions
  5. Fast, optimisable column

Syntax

F.when(cond, value).when(cond2, value2).otherwise(default)
F.expr("CASE WHEN amount >= 50000 THEN 'Premium' ELSE 'Other' END")
F.transform("arr", lambda x: x * 2); F.exists("arr", lambda x: x > 5)
F.sha2(F.col("email"), 256); F.regexp_extract(c, pattern, 1)

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

import time

@F.udf("string")
def band_udf(amount):
    if amount is None:
        return None
    return "Premium" if amount >= 50000 else ("Standard" if amount >= 10000 else "Budget")

band_builtin = (F.when(F.col("amount") >= 50000, "Premium")
                 .when(F.col("amount") >= 10000, "Standard")
                 .otherwise("Budget"))

orders.select("order_id", "amount",
              band_udf("amount").alias("udf_band"),
              band_builtin.alias("builtin_band")).show()

# Same logic, timed on a bigger generated table
big = spark.range(300_000).withColumn("amount", (F.col("id") % 70000).cast("double"))
for label, col in [("python udf", band_udf("amount")), ("built-in", band_builtin)]:
    start = time.time()
    big.select(col.alias("band")).groupBy("band").count().collect()
    print(f"{label:10s}: {time.time() - start:.2f}s")

# A SQL expression string works too
orders.select("order_id", F.expr("CASE WHEN qty > 1 THEN 'multi' ELSE 'single' END").alias("basket")).show(3)

# Output:
# +--------+-------+--------+------------+
# |order_id| amount|udf_band|builtin_band|
# +--------+-------+--------+------------+
# |       1|55000.0| Premium|     Premium|
# |       2|30000.0|Standard|    Standard|
# |       3| 1500.0|  Budget|      Budget|
# |       4|52000.0| Premium|     Premium|
# |       5|25000.0|Standard|    Standard|
# |       6|18000.0|Standard|    Standard|
# |       7| 1000.0|  Budget|      Budget|
# |       8|60000.0| Premium|     Premium|
# +--------+-------+--------+------------+
#
# python udf: 4.07s
# built-in  : 0.93s
# +--------+------+
# |order_id|basket|
# +--------+------+
# |       1|single|
# |       2| multi|
# |       3| multi|
# +--------+------+
# only showing top 3 rows

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic