Creating DataFrames & schemas (StructType, DDL strings)

Apache Spark · DataFrames

Theory

A DataFrame is a distributed table: rows plus a schema that names each column and gives its data type and nullability. You can create one from Python data with spark.createDataFrame(rows, schema), from a pandas DataFrame, from spark.range(n), or by reading files and tables. The schema can be written as a DDL string such as 'order_id INT, region STRING, amount DOUBLE', or built in code with StructType([StructField('order_id', IntegerType(), True), ...]). If you pass only column names, Spark infers types from the data.

Defining the schema yourself is safer and faster than inference: Spark does not need an extra pass over the data, and a column of IDs never turns into strings by accident. printSchema() shows the tree, df.schema returns the StructType and df.dtypes lists (name, type) pairs. For example, createDataFrame([(1, 'Asha', 'Jaipur')], 'customer_id INT, name STRING, city STRING') gives a three-column DataFrame with exactly those types.

Why we use it

Every Spark pipeline starts with DataFrames, and a correct schema prevents silent type problems later (strings where numbers should be, wrong dates, nullable keys). Explicit schemas also make reading large files faster and more predictable.

When to use it

Use DDL strings for quick, readable schemas; use StructType when building schemas in code, adding metadata or nesting structs. Let Spark infer only for quick exploration. For files in production, pass the schema to the reader instead of inferSchema.

How it works

  1. Have data (Python list, pandas, files)
  2. Do you know the columns and types? Yes: Write a DDL string or StructType. No: Let Spark infer (exploration only).
  3. spark.createDataFrame(data, schema) or spark.read.schema(schema)
  4. printSchema() to confirm types
  5. Use the DataFrame in transformations

Syntax

spark.createDataFrame(rows, "id INT, name STRING")
spark.createDataFrame(rows, StructType([StructField("id", IntegerType(), False), ...]))
spark.createDataFrame(pandas_df);  spark.range(5)
df.printSchema();  df.schema;  df.dtypes

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, IntegerType, StringType

# 1) DDL string schema
c1 = spark.createDataFrame([(1, "Asha", "Jaipur"), (2, "Ravi", "Pune")],
                           "customer_id INT, name STRING, city STRING")
c1.printSchema()

# 2) StructType schema (customer_id NOT NULL)
schema = StructType([
    StructField("customer_id", IntegerType(), nullable=False),
    StructField("name", StringType(), True),
    StructField("city", StringType(), True),
])
c2 = spark.createDataFrame([(3, "Meera", "Delhi")], schema)
c2.printSchema()                            # nullable = false on the key

# 3) Inferred from Row objects / column names only
c3 = spark.createDataFrame([Row(customer_id=4, name="Karan", score=8.5)])
print(c3.dtypes)

spark.range(3).show()

# Output:
# root
#  |-- customer_id: integer (nullable = true)
#  |-- name: string (nullable = true)
#  |-- city: string (nullable = true)
#
# root
#  |-- customer_id: integer (nullable = false)
#  |-- name: string (nullable = true)
#  |-- city: string (nullable = true)
#
# [('customer_id', 'bigint'), ('name', 'string'), ('score', 'double')]
# +---+
# | id|
# +---+
# |  0|
# |  1|
# |  2|
# +---+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic