Built-in Functions First
Apache Spark · UDFs
Theory
Before writing any UDF, check pyspark.sql.functions and Spark SQL's built-in functions. They cover conditions, maths, strings, regex, dates, arrays, maps, JSON, hashing and many aggregates. Built-ins run inside the JVM on Spark's internal row format, so there is no Python process or serialisation, and Catalyst understands them: it can fold constants, push filters down to the data source, prune columns and generate compiled code for whole stages.
A typical case: a UDF that labels orders "Premium", "Standard" or "Budget" by amount is the same as F.when(F.col("amount") >= 50000, "Premium").when(F.col("amount") >= 10000, "Standard").otherwise("Budget"). The built-in version handles nulls, shows up in explain() as a normal expression and is usually many times faster. Higher-order functions (transform, filter, exists, aggregate) and F.expr("SQL text") cover a lot of logic people reach for UDFs to solve.
Why we use it
Replacing UDFs with built-ins is one of the easiest big performance wins in PySpark. It also keeps code simpler, null handling consistent and plans visible in explain(), so the optimiser and readers can both understand the query.
When to use it
Always try built-ins, F.expr SQL expressions and higher-order functions first. Reach for a pandas UDF only when the logic needs a Python library or cannot be expressed in Spark functions, and use a plain Python UDF as the last option.
How it works
- You need a new derived column
- Is there a built-in function or SQL expression for it? Yes: Use it (F.*, F.expr, higher-order functions). No: Check if it can be vectorised.
- Can pandas / NumPy do it on whole columns? Yes: Write a pandas UDF. No: Write a Python UDF as the last option.
- Check explain(): built-ins appear as normal expressions
- Fast, optimisable column
Syntax
F.when(cond, value).when(cond2, value2).otherwise(default)
F.expr("CASE WHEN amount >= 50000 THEN 'Premium' ELSE 'Other' END")
F.transform("arr", lambda x: x * 2); F.exists("arr", lambda x: x > 5)
F.sha2(F.col("email"), 256); F.regexp_extract(c, pattern, 1)Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
import time
@F.udf("string")
def band_udf(amount):
if amount is None:
return None
return "Premium" if amount >= 50000 else ("Standard" if amount >= 10000 else "Budget")
band_builtin = (F.when(F.col("amount") >= 50000, "Premium")
.when(F.col("amount") >= 10000, "Standard")
.otherwise("Budget"))
orders.select("order_id", "amount",
band_udf("amount").alias("udf_band"),
band_builtin.alias("builtin_band")).show()
# Same logic, timed on a bigger generated table
big = spark.range(300_000).withColumn("amount", (F.col("id") % 70000).cast("double"))
for label, col in [("python udf", band_udf("amount")), ("built-in", band_builtin)]:
start = time.time()
big.select(col.alias("band")).groupBy("band").count().collect()
print(f"{label:10s}: {time.time() - start:.2f}s")
# A SQL expression string works too
orders.select("order_id", F.expr("CASE WHEN qty > 1 THEN 'multi' ELSE 'single' END").alias("basket")).show(3)
# Output:
# +--------+-------+--------+------------+
# |order_id| amount|udf_band|builtin_band|
# +--------+-------+--------+------------+
# | 1|55000.0| Premium| Premium|
# | 2|30000.0|Standard| Standard|
# | 3| 1500.0| Budget| Budget|
# | 4|52000.0| Premium| Premium|
# | 5|25000.0|Standard| Standard|
# | 6|18000.0|Standard| Standard|
# | 7| 1000.0| Budget| Budget|
# | 8|60000.0| Premium| Premium|
# +--------+-------+--------+------------+
#
# python udf: 4.07s
# built-in : 0.93s
# +--------+------+
# |order_id|basket|
# +--------+------+
# | 1|single|
# | 2| multi|
# | 3| multi|
# +--------+------+
# only showing top 3 rows
Key points
- Built-ins run in the JVM: no Python workers, no serialisation
- Catalyst can push filters and prune columns through built-ins, not through UDFs
- F.when().otherwise() replaces most if/else UDFs
- transform / filter / exists / aggregate handle array logic without UDFs
- F.expr lets you drop any Spark SQL expression into DataFrame code