What's new in Spark 4

Apache Spark · Spark Fundamentals

Theory

Spark 4 is the major release line that followed Spark 3.5. The biggest behaviour change is that ANSI SQL mode is on by default: invalid operations such as dividing by zero or casting 'abc' to INT now raise an error instead of silently returning NULL, and you use try_divide, try_cast and similar try_ functions when you want NULL. Spark Connect is much more complete, with a lightweight pyspark-client package and a spark.api.mode setting to switch an application to Connect.

Spark 4.0 also added the VARIANT type for semi-structured JSON (parse_json, variant_get), SQL pipe syntax (FROM orders |> WHERE ... |> AGGREGATE ...), the Python Data Source API for writing custom readers and writers in pure Python, string collations, SQL user-defined functions and session variables. Later 4.x releases added Spark Declarative Pipelines, a real-time mode for Structured Streaming and made SQL scripting and VARIANT generally available. Spark 4 requires Java 17 or 21 and Scala 2.13.

Why we use it

Code written for Spark 3 can behave differently on Spark 4, mainly because of ANSI mode. Knowing the new defaults and features helps you upgrade safely, explain errors like DIVIDE_BY_ZERO or CAST_INVALID_INPUT, and answer interview questions about modern Spark.

When to use it

Learn these when starting new Spark 4 projects or upgrading from 3.x. Use try_ functions where bad data is expected, VARIANT for flexible JSON, and pipe syntax when it reads better. Set spark.sql.ansi.enabled=false only as a temporary migration aid.

How it works

  1. Upgrading a job from Spark 3.x
  2. Run the tests on Spark 4 with defaults
  3. Do queries fail with ANSI errors (cast, divide, overflow)? Yes: Fix the data or use try_ functions. No: Keep the code as it is.
  4. Move JSON string columns to VARIANT where it helps
  5. Try Spark Connect for thin clients
  6. Deploy on Java 17 or 21

Syntax

F.try_divide(a, b)  /  try_cast(col AS INT)
parse_json(json_str)  /  variant_get(v, '$.path', 'type')
FROM t |> WHERE ... |> AGGREGATE SUM(x) GROUP BY k
from pyspark.sql.datasource import DataSource, DataSourceReader
spark.dataSource.register(MySource)

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

from pyspark.sql.datasource import DataSource, DataSourceReader

print("ANSI mode:", spark.conf.get("spark.sql.ansi.enabled"))

# ANSI: 1/0 raises DIVIDE_BY_ZERO, so use try_ functions when NULL is wanted
spark.sql("SELECT try_divide(100, 0) AS safe_div, "
          "try_cast('abc' AS INT) AS safe_cast").show()

# VARIANT: store any JSON, extract fields later
spark.sql("""SELECT variant_get(parse_json('{"city":"Jaipur","pin":302001}'),
                                '$.city', 'string') AS city""").show()

# SQL pipe syntax
orders.createOrReplaceTempView("orders")
spark.sql("""FROM orders
             |> WHERE amount > 20000
             |> AGGREGATE SUM(amount) AS total GROUP BY region
             |> ORDER BY total DESC""").show()

# Python Data Source API: a custom reader in pure Python
class CityReader(DataSourceReader):
    def read(self, partition):
        yield ("Jaipur", "Rajasthan")
        yield ("Pune", "Maharashtra")

class CitySource(DataSource):
    @classmethod
    def name(cls):
        return "cities"
    def schema(self):
        return "city STRING, state STRING"
    def reader(self, schema):
        return CityReader()

spark.dataSource.register(CitySource)
spark.read.format("cities").load().show()

# Output:
# ANSI mode: true
# +--------+---------+
# |safe_div|safe_cast|
# +--------+---------+
# |    NULL|     NULL|
# +--------+---------+
#
# +------+
# |  city|
# +------+
# |Jaipur|
# +------+
#
# +------+--------+
# |region|   total|
# +------+--------+
# | North|103000.0|
# |  West| 55000.0|
# | South| 52000.0|
# +------+--------+
#
# +------+-----------+
# |  city|      state|
# +------+-----------+
# |Jaipur|  Rajasthan|
# |  Pune|Maharashtra|
# +------+-----------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic