What's new in Spark 4
Apache Spark · Spark Fundamentals
Theory
Spark 4 is the major release line that followed Spark 3.5. The biggest behaviour change is that ANSI SQL mode is on by default: invalid operations such as dividing by zero or casting 'abc' to INT now raise an error instead of silently returning NULL, and you use try_divide, try_cast and similar try_ functions when you want NULL. Spark Connect is much more complete, with a lightweight pyspark-client package and a spark.api.mode setting to switch an application to Connect.
Spark 4.0 also added the VARIANT type for semi-structured JSON (parse_json, variant_get), SQL pipe syntax (FROM orders |> WHERE ... |> AGGREGATE ...), the Python Data Source API for writing custom readers and writers in pure Python, string collations, SQL user-defined functions and session variables. Later 4.x releases added Spark Declarative Pipelines, a real-time mode for Structured Streaming and made SQL scripting and VARIANT generally available. Spark 4 requires Java 17 or 21 and Scala 2.13.
Why we use it
Code written for Spark 3 can behave differently on Spark 4, mainly because of ANSI mode. Knowing the new defaults and features helps you upgrade safely, explain errors like DIVIDE_BY_ZERO or CAST_INVALID_INPUT, and answer interview questions about modern Spark.
When to use it
Learn these when starting new Spark 4 projects or upgrading from 3.x. Use try_ functions where bad data is expected, VARIANT for flexible JSON, and pipe syntax when it reads better. Set spark.sql.ansi.enabled=false only as a temporary migration aid.
How it works
- Upgrading a job from Spark 3.x
- Run the tests on Spark 4 with defaults
- Do queries fail with ANSI errors (cast, divide, overflow)? Yes: Fix the data or use try_ functions. No: Keep the code as it is.
- Move JSON string columns to VARIANT where it helps
- Try Spark Connect for thin clients
- Deploy on Java 17 or 21
Syntax
F.try_divide(a, b) / try_cast(col AS INT)
parse_json(json_str) / variant_get(v, '$.path', 'type')
FROM t |> WHERE ... |> AGGREGATE SUM(x) GROUP BY k
from pyspark.sql.datasource import DataSource, DataSourceReader
spark.dataSource.register(MySource)Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
from pyspark.sql.datasource import DataSource, DataSourceReader
print("ANSI mode:", spark.conf.get("spark.sql.ansi.enabled"))
# ANSI: 1/0 raises DIVIDE_BY_ZERO, so use try_ functions when NULL is wanted
spark.sql("SELECT try_divide(100, 0) AS safe_div, "
"try_cast('abc' AS INT) AS safe_cast").show()
# VARIANT: store any JSON, extract fields later
spark.sql("""SELECT variant_get(parse_json('{"city":"Jaipur","pin":302001}'),
'$.city', 'string') AS city""").show()
# SQL pipe syntax
orders.createOrReplaceTempView("orders")
spark.sql("""FROM orders
|> WHERE amount > 20000
|> AGGREGATE SUM(amount) AS total GROUP BY region
|> ORDER BY total DESC""").show()
# Python Data Source API: a custom reader in pure Python
class CityReader(DataSourceReader):
def read(self, partition):
yield ("Jaipur", "Rajasthan")
yield ("Pune", "Maharashtra")
class CitySource(DataSource):
@classmethod
def name(cls):
return "cities"
def schema(self):
return "city STRING, state STRING"
def reader(self, schema):
return CityReader()
spark.dataSource.register(CitySource)
spark.read.format("cities").load().show()
# Output:
# ANSI mode: true
# +--------+---------+
# |safe_div|safe_cast|
# +--------+---------+
# | NULL| NULL|
# +--------+---------+
#
# +------+
# | city|
# +------+
# |Jaipur|
# +------+
#
# +------+--------+
# |region| total|
# +------+--------+
# | North|103000.0|
# | West| 55000.0|
# | South| 52000.0|
# +------+--------+
#
# +------+-----------+
# | city| state|
# +------+-----------+
# |Jaipur| Rajasthan|
# | Pune|Maharashtra|
# +------+-----------+
Key points
- ANSI mode is on by default: bad casts, overflow and divide-by-zero raise errors
- Use try_cast, try_divide, try_add and friends to get NULL instead of an error
- VARIANT stores semi-structured data; read it with variant_get or the : path syntax in SQL
- Pipe syntax chains SQL steps with |> in the order they run
- Python Data Source API: custom spark.read.format(...) sources written in Python
- Spark 4 needs Java 17 or 21; Java 8 and 11 are no longer supported