SparkConf & runtime configuration
Apache Spark · Spark Architecture
Theory
Spark settings are key-value pairs such as spark.executor.memory=8g or spark.sql.shuffle.partitions=200. There are two kinds. Application (static) settings, like executor memory, cores, the master and the serializer, are read once when the SparkContext starts, so you set them with SparkConf, the session builder's .config(), spark-submit --conf or spark-defaults.conf. Runtime SQL settings, such as shuffle partitions, AQE switches or the broadcast threshold, live in the session and can be changed at any time with spark.conf.set() or SQL SET.
If a key is set in several places, values in code (SparkConf / builder) win, then spark-submit flags, then spark-defaults.conf. spark.conf.isModifiable(key) tells you whether a key can change at runtime; trying to set a static one on a running session raises an error. For example, you can lower spark.sql.shuffle.partitions from 200 to 8 for a small notebook dataset with one line, but executor memory needs a restart.
Why we use it
Most tuning is configuration: shuffle partitions, broadcast threshold, memory and cores. Knowing which settings can change at runtime, and which source wins, saves hours of confusion when a value seems to be ignored.
When to use it
Set cluster resources at launch (spark-submit or the platform's cluster settings). Change SQL settings per job or per query with spark.conf.set. Read a value with spark.conf.get(key, default) before assuming it. Keep secrets and environment-specific values out of code.
How it works
- Spark needs the value of a setting
- Set in code (SparkConf / builder)? Yes: Use the code value. No: Check spark-submit flags.
- Set by a spark-submit flag? Yes: Use the flag value. No: Check spark-defaults.conf.
- Otherwise use Spark's built-in default
- Changed later with spark.conf.set? Yes: Session value wins from now on. No: Startup value stays.
- Value used by the next query
Syntax
from pyspark import SparkConf
conf = SparkConf().setAppName("app").set("spark.executor.memory", "8g")
spark = SparkSession.builder.config(conf=conf).getOrCreate()
spark.conf.set("spark.sql.shuffle.partitions", "64")
spark.conf.get("key", "default"); spark.conf.unset("key")
spark.conf.isModifiable("spark.executor.memory") # False
spark.sql("SET spark.sql.shuffle.partitions = 64")Example
from pyspark import SparkConf
from pyspark.sql import SparkSession
# 1) SparkConf: settings fixed when the application (SparkContext) starts
conf = (SparkConf()
.setMaster("local[2]")
.setAppName("conf-demo")
.set("spark.ui.enabled", "false")
.set("spark.sql.shuffle.partitions", "8")
.set("spark.driver.memory", "1g"))
spark = SparkSession.builder.config(conf=conf).getOrCreate()
print("startup value:", spark.conf.get("spark.sql.shuffle.partitions"))
# 2) Runtime (session) SQL configs can be changed at any time
spark.conf.set("spark.sql.shuffle.partitions", "4")
print("after conf.set:", spark.conf.get("spark.sql.shuffle.partitions"))
spark.sql("SET spark.sql.shuffle.partitions = 2") # SQL way
print("after SQL SET:", spark.conf.get("spark.sql.shuffle.partitions"))
# 3) Is a key changeable at runtime?
for key in ["spark.sql.shuffle.partitions", "spark.sql.adaptive.enabled", "spark.executor.memory"]:
print(key, "modifiable:", spark.conf.isModifiable(key))
# 4) Core settings cannot be changed on a running session
try:
spark.conf.set("spark.executor.memory", "4g")
except Exception as e:
print("set executor memory ->", type(e).__name__)
# 5) Read with a fallback, and read the startup conf of the SparkContext
print("unset key with default:", spark.conf.get("spark.techtown.team", "data-eng"))
print("driver memory (startup):", spark.sparkContext.getConf().get("spark.driver.memory"))
# 6) unset removes the session value; Spark's built-in default applies again
spark.conf.unset("spark.sql.shuffle.partitions")
print("after unset:", spark.conf.get("spark.sql.shuffle.partitions"))
# Output:
# startup value: 8
# after conf.set: 4
# after SQL SET: 2
# spark.sql.shuffle.partitions modifiable: True
# spark.sql.adaptive.enabled modifiable: True
# spark.executor.memory modifiable: False
# set executor memory -> AnalysisException
# unset key with default: data-eng
# driver memory (startup): 1g
# after unset: 200
Key points
- Static (app) settings: memory, cores, master, serializer; fixed at startup
- Runtime SQL settings: spark.conf.set or SET; change any time
- Precedence: code > spark-submit flags > spark-defaults.conf
- isModifiable(key) tells you if a key can change at runtime
- unset() falls back to Spark's built-in default, not your startup value
- Common: spark.sql.shuffle.partitions (200), autoBroadcastJoinThreshold (10 MB)