Working with JSON & VARIANT
Apache Spark · Cleaning & Complex Types
Theory
JSON arrives either as files or as a string column inside another table (a Kafka value, an API payload). For files, spark.read.json(path) infers a schema, and passing .schema(...) skips the inference pass. For a string column, F.from_json(col, schema) parses it into a struct you can query with dot notation; F.schema_of_json(sample) suggests a DDL schema from one example. F.get_json_object(col, "$.address.city") pulls a single value by JSON path without a full schema, F.json_tuple extracts several top-level keys, and F.to_json turns a struct back into a string.
Spark 4 adds the VARIANT type for semi-structured data whose shape changes often. F.parse_json(col) stores the JSON in an efficient binary form, F.variant_get(v, "$.address.city", "string") reads a path and casts it, try_variant_get returns null on a bad cast, and schema_of_variant shows the shape. Unlike from_json, you do not need to fix the schema up front.
Why we use it
Event, log and API data is JSON. Parsing it with built-in functions keeps the work in the JVM and lets you flatten it into typed columns for joins and aggregates, while VARIANT lets you store fast-changing payloads without schema breakages.
When to use it
Use from_json with an explicit schema when the structure is known and stable (best performance, typed columns). Use get_json_object for one or two fields. Use VARIANT (parse_json + variant_get) when fields vary between records or evolve often. Avoid Python json.loads inside UDFs.
How it works
- JSON strings or JSON files
- Is the structure known and stable? Yes: from_json(col, schema) into a struct. No: Store as VARIANT with parse_json.
- Read fields with dot notation or variant_get(v, path, type)
- Flatten into typed columns; explode arrays if needed
- Typed table ready for joins and aggregates
Syntax
schema = "customer_id INT, address STRUCT<city: STRING, pin: STRING>, tags ARRAY<STRING>"
F.from_json("payload", schema)
F.schema_of_json(F.lit(sample_json))
F.get_json_object("payload", "$.address.city")
v = F.parse_json("payload"); F.variant_get(v, "$.address.pin", "string")Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
raw = spark.createDataFrame([
(1, '{"customer_id": 101, "address": {"city": "Jaipur", "pin": "302021"}, "tags": ["new", "app"]}'),
(2, '{"customer_id": 102, "address": {"city": "Pune", "pin": "411001"}, "tags": ["web"]}'),
(3, '{"customer_id": 103, "address": {"city": "Delhi"}, "coupon": "DIWALI10"}'),
], "event_id INT, payload STRING")
# 1. Ask Spark for a schema from a sample record
sample = raw.first()["payload"]
spark.range(1).select(F.schema_of_json(F.lit(sample)).alias("schema")).show(truncate=False)
# 2. from_json with an explicit schema -> typed struct
schema = "customer_id INT, address STRUCT<city: STRING, pin: STRING>, tags ARRAY<STRING>"
parsed = raw.withColumn("j", F.from_json("payload", schema))
parsed.select("event_id", "j.customer_id", "j.address.city", "j.address.pin", "j.tags").show()
# 3. One value by JSON path, no schema needed
raw.select("event_id", F.get_json_object("payload", "$.coupon").alias("coupon")).show()
# 4. VARIANT: store as-is, read paths later
v = raw.withColumn("v", F.parse_json("payload"))
v.select("event_id",
F.variant_get("v", "$.address.city", "string").alias("city"),
F.try_variant_get("v", "$.customer_id", "int").alias("customer_id"),
F.variant_get("v", "$.coupon", "string").alias("coupon")).show()
# Output:
# +--------------------------------------------------------------------------------------------+
# |schema |
# +--------------------------------------------------------------------------------------------+
# |STRUCT<address: STRUCT<city: STRING, pin: STRING>, customer_id: BIGINT, tags: ARRAY<STRING>>|
# +--------------------------------------------------------------------------------------------+
#
# +--------+-----------+------+------+----------+
# |event_id|customer_id| city| pin| tags|
# +--------+-----------+------+------+----------+
# | 1| 101|Jaipur|302021|[new, app]|
# | 2| 102| Pune|411001| [web]|
# | 3| 103| Delhi| NULL| NULL|
# +--------+-----------+------+------+----------+
#
# +--------+--------+
# |event_id| coupon|
# +--------+--------+
# | 1| NULL|
# | 2| NULL|
# | 3|DIWALI10|
# +--------+--------+
#
# +--------+------+-----------+--------+
# |event_id| city|customer_id| coupon|
# +--------+------+-----------+--------+
# | 1|Jaipur| 101| NULL|
# | 2| Pune| 102| NULL|
# | 3| Delhi| 103|DIWALI10|
# +--------+------+-----------+--------+
Key points
- from_json needs a schema; fields not in the schema are ignored, malformed rows give null
- schema_of_json(sample) is a quick way to draft the schema DDL
- get_json_object is handy for one field but reparses the string each call
- VARIANT (parse_json + variant_get) handles changing shapes without a fixed schema
- Give spark.read.json an explicit schema in production to skip inference