Data types
Apache Spark · DataFrames
Theory
Every DataFrame column has a Spark SQL data type. The common simple types are STRING, INT (IntegerType), BIGINT (LongType), DOUBLE, DECIMAL(p,s) for exact money values, BOOLEAN, DATE and TIMESTAMP, plus BINARY. Complex types hold nested data: ARRAY<T> (a list), MAP<K,V> (key-value pairs) and STRUCT<...> (a record with named fields). Spark 4 also has VARIANT for semi-structured JSON whose shape varies from row to row.
You change types with col.cast('int') or CAST(x AS DECIMAL(10,2)) in SQL. With ANSI mode on by default, an invalid cast such as 'abc' to INT raises an error instead of quietly returning NULL, so for messy input use try_cast, which returns NULL for bad values. For example, try_cast('1200.50' AS DECIMAL(10,2)) gives 1200.50, while try_cast('2026-02-30' AS DATE) gives NULL because that date does not exist.
Why we use it
Correct types make calculations right (DECIMAL for rupees instead of DOUBLE rounding), make dates sortable and filterable, and let Spark store data compactly. Type mistakes are a common source of wrong results and, under ANSI mode, of job failures.
When to use it
Cast raw string columns to proper types right after reading. Use DECIMAL for currency, DATE/TIMESTAMP for time, ARRAY/MAP/STRUCT for nested data and VARIANT for JSON with changing shape. Use try_cast when the input may contain bad values you want as NULL.
How it works
- Raw columns arrive as strings
- Decide the target type for each column
- Can the input contain bad values? Yes: Use try_cast and check the NULLs. No: Use CAST (fails fast on bad data).
- Use complex types (ARRAY, MAP, STRUCT) for nested data
- printSchema() to confirm
Syntax
F.col("qty").cast("int")
F.col("amount").cast("decimal(10,2)")
df.selectExpr("try_cast(order_date AS DATE) AS order_date")
"tags ARRAY<STRING>, attrs MAP<STRING,STRING>, addr STRUCT<city: STRING, pin: INT>"Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
raw = spark.createDataFrame(
[("1", "1200.50", "2026-01-05"), ("2", "abc", "2026-02-30"), ("3", "999", "2026-03-15")],
"id STRING, amount STRING, order_date STRING")
typed = raw.selectExpr(
"CAST(id AS INT) AS id",
"try_cast(amount AS DECIMAL(10,2)) AS amount", # bad values -> NULL
"try_cast(order_date AS DATE) AS order_date")
typed.show()
typed.printSchema()
nested = spark.createDataFrame(
[(1, ["Laptop", "Mouse"], {"color": "black"}, ("Jaipur", 302001))],
"order_id INT, items ARRAY<STRING>, attrs MAP<STRING,STRING>, "
"ship_to STRUCT<city: STRING, pin: INT>")
nested.select("items", F.col("attrs")["color"].alias("color"),
F.col("ship_to.city").alias("city")).show()
# Output:
# +---+-------+----------+
# | id| amount|order_date|
# +---+-------+----------+
# | 1|1200.50|2026-01-05|
# | 2| NULL| NULL|
# | 3| 999.00|2026-03-15|
# +---+-------+----------+
#
# root
# |-- id: integer (nullable = true)
# |-- amount: decimal(10,2) (nullable = true)
# |-- order_date: date (nullable = true)
#
# +---------------+-----+------+
# | items|color| city|
# +---------------+-----+------+
# |[Laptop, Mouse]|black|Jaipur|
# +---------------+-----+------+
Key points
- Use DECIMAL(p,s) for money; DOUBLE can show rounding errors
- ANSI mode: invalid CAST raises an error; try_cast returns NULL
- Complex types: ARRAY, MAP, STRUCT; access with [index], ['key'] and dot notation
- VARIANT stores semi-structured JSON with flexible shape
- DATE has no time part; TIMESTAMP is shown in the session time zone