Creating DataFrames & schemas (StructType, DDL strings)
Apache Spark · DataFrames
Theory
A DataFrame is a distributed table: rows plus a schema that names each column and gives its data type and nullability. You can create one from Python data with spark.createDataFrame(rows, schema), from a pandas DataFrame, from spark.range(n), or by reading files and tables. The schema can be written as a DDL string such as 'order_id INT, region STRING, amount DOUBLE', or built in code with StructType([StructField('order_id', IntegerType(), True), ...]). If you pass only column names, Spark infers types from the data.
Defining the schema yourself is safer and faster than inference: Spark does not need an extra pass over the data, and a column of IDs never turns into strings by accident. printSchema() shows the tree, df.schema returns the StructType and df.dtypes lists (name, type) pairs. For example, createDataFrame([(1, 'Asha', 'Jaipur')], 'customer_id INT, name STRING, city STRING') gives a three-column DataFrame with exactly those types.
Why we use it
Every Spark pipeline starts with DataFrames, and a correct schema prevents silent type problems later (strings where numbers should be, wrong dates, nullable keys). Explicit schemas also make reading large files faster and more predictable.
When to use it
Use DDL strings for quick, readable schemas; use StructType when building schemas in code, adding metadata or nesting structs. Let Spark infer only for quick exploration. For files in production, pass the schema to the reader instead of inferSchema.
How it works
- Have data (Python list, pandas, files)
- Do you know the columns and types? Yes: Write a DDL string or StructType. No: Let Spark infer (exploration only).
- spark.createDataFrame(data, schema) or spark.read.schema(schema)
- printSchema() to confirm types
- Use the DataFrame in transformations
Syntax
spark.createDataFrame(rows, "id INT, name STRING")
spark.createDataFrame(rows, StructType([StructField("id", IntegerType(), False), ...]))
spark.createDataFrame(pandas_df); spark.range(5)
df.printSchema(); df.schema; df.dtypesExample
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
# 1) DDL string schema
c1 = spark.createDataFrame([(1, "Asha", "Jaipur"), (2, "Ravi", "Pune")],
"customer_id INT, name STRING, city STRING")
c1.printSchema()
# 2) StructType schema (customer_id NOT NULL)
schema = StructType([
StructField("customer_id", IntegerType(), nullable=False),
StructField("name", StringType(), True),
StructField("city", StringType(), True),
])
c2 = spark.createDataFrame([(3, "Meera", "Delhi")], schema)
c2.printSchema() # nullable = false on the key
# 3) Inferred from Row objects / column names only
c3 = spark.createDataFrame([Row(customer_id=4, name="Karan", score=8.5)])
print(c3.dtypes)
spark.range(3).show()
# Output:
# root
# |-- customer_id: integer (nullable = true)
# |-- name: string (nullable = true)
# |-- city: string (nullable = true)
#
# root
# |-- customer_id: integer (nullable = false)
# |-- name: string (nullable = true)
# |-- city: string (nullable = true)
#
# [('customer_id', 'bigint'), ('name', 'string'), ('score', 'double')]
# +---+
# | id|
# +---+
# | 0|
# | 1|
# | 2|
# +---+
Key points
- A DataFrame = distributed rows + schema (column names, types, nullability)
- DDL strings are short and readable; StructType is programmatic and supports metadata
- Explicit schemas avoid an extra inference pass and type surprises
- printSchema(), schema and dtypes let you inspect what Spark thinks the types are
- Python ints are inferred as LONG (bigint) and floats as DOUBLE when no schema is given