SparkConf & runtime configuration

Apache Spark · Spark Architecture

Theory

Spark settings are key-value pairs such as spark.executor.memory=8g or spark.sql.shuffle.partitions=200. There are two kinds. Application (static) settings, like executor memory, cores, the master and the serializer, are read once when the SparkContext starts, so you set them with SparkConf, the session builder's .config(), spark-submit --conf or spark-defaults.conf. Runtime SQL settings, such as shuffle partitions, AQE switches or the broadcast threshold, live in the session and can be changed at any time with spark.conf.set() or SQL SET.

If a key is set in several places, values in code (SparkConf / builder) win, then spark-submit flags, then spark-defaults.conf. spark.conf.isModifiable(key) tells you whether a key can change at runtime; trying to set a static one on a running session raises an error. For example, you can lower spark.sql.shuffle.partitions from 200 to 8 for a small notebook dataset with one line, but executor memory needs a restart.

Why we use it

Most tuning is configuration: shuffle partitions, broadcast threshold, memory and cores. Knowing which settings can change at runtime, and which source wins, saves hours of confusion when a value seems to be ignored.

When to use it

Set cluster resources at launch (spark-submit or the platform's cluster settings). Change SQL settings per job or per query with spark.conf.set. Read a value with spark.conf.get(key, default) before assuming it. Keep secrets and environment-specific values out of code.

How it works

  1. Spark needs the value of a setting
  2. Set in code (SparkConf / builder)? Yes: Use the code value. No: Check spark-submit flags.
  3. Set by a spark-submit flag? Yes: Use the flag value. No: Check spark-defaults.conf.
  4. Otherwise use Spark's built-in default
  5. Changed later with spark.conf.set? Yes: Session value wins from now on. No: Startup value stays.
  6. Value used by the next query

Syntax

from pyspark import SparkConf
conf = SparkConf().setAppName("app").set("spark.executor.memory", "8g")
spark = SparkSession.builder.config(conf=conf).getOrCreate()
spark.conf.set("spark.sql.shuffle.partitions", "64")
spark.conf.get("key", "default"); spark.conf.unset("key")
spark.conf.isModifiable("spark.executor.memory")   # False
spark.sql("SET spark.sql.shuffle.partitions = 64")

Example

from pyspark import SparkConf
from pyspark.sql import SparkSession

# 1) SparkConf: settings fixed when the application (SparkContext) starts
conf = (SparkConf()
        .setMaster("local[2]")
        .setAppName("conf-demo")
        .set("spark.ui.enabled", "false")
        .set("spark.sql.shuffle.partitions", "8")
        .set("spark.driver.memory", "1g"))
spark = SparkSession.builder.config(conf=conf).getOrCreate()

print("startup value:", spark.conf.get("spark.sql.shuffle.partitions"))

# 2) Runtime (session) SQL configs can be changed at any time
spark.conf.set("spark.sql.shuffle.partitions", "4")
print("after conf.set:", spark.conf.get("spark.sql.shuffle.partitions"))
spark.sql("SET spark.sql.shuffle.partitions = 2")          # SQL way
print("after SQL SET:", spark.conf.get("spark.sql.shuffle.partitions"))

# 3) Is a key changeable at runtime?
for key in ["spark.sql.shuffle.partitions", "spark.sql.adaptive.enabled", "spark.executor.memory"]:
    print(key, "modifiable:", spark.conf.isModifiable(key))

# 4) Core settings cannot be changed on a running session
try:
    spark.conf.set("spark.executor.memory", "4g")
except Exception as e:
    print("set executor memory ->", type(e).__name__)

# 5) Read with a fallback, and read the startup conf of the SparkContext
print("unset key with default:", spark.conf.get("spark.techtown.team", "data-eng"))
print("driver memory (startup):", spark.sparkContext.getConf().get("spark.driver.memory"))

# 6) unset removes the session value; Spark's built-in default applies again
spark.conf.unset("spark.sql.shuffle.partitions")
print("after unset:", spark.conf.get("spark.sql.shuffle.partitions"))

# Output:
# startup value: 8
# after conf.set: 4
# after SQL SET: 2
# spark.sql.shuffle.partitions modifiable: True
# spark.sql.adaptive.enabled modifiable: True
# spark.executor.memory modifiable: False
# set executor memory -> AnalysisException
# unset key with default: data-eng
# driver memory (startup): 1g
# after unset: 200

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic