Working with JSON & VARIANT

Apache Spark · Cleaning & Complex Types

Theory

JSON arrives either as files or as a string column inside another table (a Kafka value, an API payload). For files, spark.read.json(path) infers a schema, and passing .schema(...) skips the inference pass. For a string column, F.from_json(col, schema) parses it into a struct you can query with dot notation; F.schema_of_json(sample) suggests a DDL schema from one example. F.get_json_object(col, "$.address.city") pulls a single value by JSON path without a full schema, F.json_tuple extracts several top-level keys, and F.to_json turns a struct back into a string.

Spark 4 adds the VARIANT type for semi-structured data whose shape changes often. F.parse_json(col) stores the JSON in an efficient binary form, F.variant_get(v, "$.address.city", "string") reads a path and casts it, try_variant_get returns null on a bad cast, and schema_of_variant shows the shape. Unlike from_json, you do not need to fix the schema up front.

Why we use it

Event, log and API data is JSON. Parsing it with built-in functions keeps the work in the JVM and lets you flatten it into typed columns for joins and aggregates, while VARIANT lets you store fast-changing payloads without schema breakages.

When to use it

Use from_json with an explicit schema when the structure is known and stable (best performance, typed columns). Use get_json_object for one or two fields. Use VARIANT (parse_json + variant_get) when fields vary between records or evolve often. Avoid Python json.loads inside UDFs.

How it works

  1. JSON strings or JSON files
  2. Is the structure known and stable? Yes: from_json(col, schema) into a struct. No: Store as VARIANT with parse_json.
  3. Read fields with dot notation or variant_get(v, path, type)
  4. Flatten into typed columns; explode arrays if needed
  5. Typed table ready for joins and aggregates

Syntax

schema = "customer_id INT, address STRUCT<city: STRING, pin: STRING>, tags ARRAY<STRING>"
F.from_json("payload", schema)
F.schema_of_json(F.lit(sample_json))
F.get_json_object("payload", "$.address.city")
v = F.parse_json("payload"); F.variant_get(v, "$.address.pin", "string")

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

raw = spark.createDataFrame([
    (1, '{"customer_id": 101, "address": {"city": "Jaipur", "pin": "302021"}, "tags": ["new", "app"]}'),
    (2, '{"customer_id": 102, "address": {"city": "Pune", "pin": "411001"}, "tags": ["web"]}'),
    (3, '{"customer_id": 103, "address": {"city": "Delhi"}, "coupon": "DIWALI10"}'),
], "event_id INT, payload STRING")

# 1. Ask Spark for a schema from a sample record
sample = raw.first()["payload"]
spark.range(1).select(F.schema_of_json(F.lit(sample)).alias("schema")).show(truncate=False)

# 2. from_json with an explicit schema -> typed struct
schema = "customer_id INT, address STRUCT<city: STRING, pin: STRING>, tags ARRAY<STRING>"
parsed = raw.withColumn("j", F.from_json("payload", schema))
parsed.select("event_id", "j.customer_id", "j.address.city", "j.address.pin", "j.tags").show()

# 3. One value by JSON path, no schema needed
raw.select("event_id", F.get_json_object("payload", "$.coupon").alias("coupon")).show()

# 4. VARIANT: store as-is, read paths later
v = raw.withColumn("v", F.parse_json("payload"))
v.select("event_id",
         F.variant_get("v", "$.address.city", "string").alias("city"),
         F.try_variant_get("v", "$.customer_id", "int").alias("customer_id"),
         F.variant_get("v", "$.coupon", "string").alias("coupon")).show()

# Output:
# +--------------------------------------------------------------------------------------------+
# |schema                                                                                      |
# +--------------------------------------------------------------------------------------------+
# |STRUCT<address: STRUCT<city: STRING, pin: STRING>, customer_id: BIGINT, tags: ARRAY<STRING>>|
# +--------------------------------------------------------------------------------------------+
#
# +--------+-----------+------+------+----------+
# |event_id|customer_id|  city|   pin|      tags|
# +--------+-----------+------+------+----------+
# |       1|        101|Jaipur|302021|[new, app]|
# |       2|        102|  Pune|411001|     [web]|
# |       3|        103| Delhi|  NULL|      NULL|
# +--------+-----------+------+------+----------+
#
# +--------+--------+
# |event_id|  coupon|
# +--------+--------+
# |       1|    NULL|
# |       2|    NULL|
# |       3|DIWALI10|
# +--------+--------+
#
# +--------+------+-----------+--------+
# |event_id|  city|customer_id|  coupon|
# +--------+------+-----------+--------+
# |       1|Jaipur|        101|    NULL|
# |       2|  Pune|        102|    NULL|
# |       3| Delhi|        103|DIWALI10|
# +--------+------+-----------+--------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic