Data types

Apache Spark · DataFrames

Theory

Every DataFrame column has a Spark SQL data type. The common simple types are STRING, INT (IntegerType), BIGINT (LongType), DOUBLE, DECIMAL(p,s) for exact money values, BOOLEAN, DATE and TIMESTAMP, plus BINARY. Complex types hold nested data: ARRAY<T> (a list), MAP<K,V> (key-value pairs) and STRUCT<...> (a record with named fields). Spark 4 also has VARIANT for semi-structured JSON whose shape varies from row to row.

You change types with col.cast('int') or CAST(x AS DECIMAL(10,2)) in SQL. With ANSI mode on by default, an invalid cast such as 'abc' to INT raises an error instead of quietly returning NULL, so for messy input use try_cast, which returns NULL for bad values. For example, try_cast('1200.50' AS DECIMAL(10,2)) gives 1200.50, while try_cast('2026-02-30' AS DATE) gives NULL because that date does not exist.

Why we use it

Correct types make calculations right (DECIMAL for rupees instead of DOUBLE rounding), make dates sortable and filterable, and let Spark store data compactly. Type mistakes are a common source of wrong results and, under ANSI mode, of job failures.

When to use it

Cast raw string columns to proper types right after reading. Use DECIMAL for currency, DATE/TIMESTAMP for time, ARRAY/MAP/STRUCT for nested data and VARIANT for JSON with changing shape. Use try_cast when the input may contain bad values you want as NULL.

How it works

  1. Raw columns arrive as strings
  2. Decide the target type for each column
  3. Can the input contain bad values? Yes: Use try_cast and check the NULLs. No: Use CAST (fails fast on bad data).
  4. Use complex types (ARRAY, MAP, STRUCT) for nested data
  5. printSchema() to confirm

Syntax

F.col("qty").cast("int")
F.col("amount").cast("decimal(10,2)")
df.selectExpr("try_cast(order_date AS DATE) AS order_date")
"tags ARRAY<STRING>, attrs MAP<STRING,STRING>, addr STRUCT<city: STRING, pin: INT>"

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

raw = spark.createDataFrame(
    [("1", "1200.50", "2026-01-05"), ("2", "abc", "2026-02-30"), ("3", "999", "2026-03-15")],
    "id STRING, amount STRING, order_date STRING")

typed = raw.selectExpr(
    "CAST(id AS INT) AS id",
    "try_cast(amount AS DECIMAL(10,2)) AS amount",     # bad values -> NULL
    "try_cast(order_date AS DATE) AS order_date")
typed.show()
typed.printSchema()

nested = spark.createDataFrame(
    [(1, ["Laptop", "Mouse"], {"color": "black"}, ("Jaipur", 302001))],
    "order_id INT, items ARRAY<STRING>, attrs MAP<STRING,STRING>, "
    "ship_to STRUCT<city: STRING, pin: INT>")
nested.select("items", F.col("attrs")["color"].alias("color"),
              F.col("ship_to.city").alias("city")).show()

# Output:
# +---+-------+----------+
# | id| amount|order_date|
# +---+-------+----------+
# |  1|1200.50|2026-01-05|
# |  2|   NULL|      NULL|
# |  3| 999.00|2026-03-15|
# +---+-------+----------+
#
# root
#  |-- id: integer (nullable = true)
#  |-- amount: decimal(10,2) (nullable = true)
#  |-- order_date: date (nullable = true)
#
# +---------------+-----+------+
# |          items|color|  city|
# +---------------+-----+------+
# |[Laptop, Mouse]|black|Jaipur|
# +---------------+-----+------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic