spark-submit, deploy modes & master URLs
Apache Spark · Spark Architecture
Theory
spark-submit is the command that launches a Spark application. You give it a master URL (where to run), a deploy mode (where the driver runs), resources (executor memory, cores, count), extra settings with --conf, dependencies with --py-files or --packages, then the script and its arguments. The master URL picks the cluster manager: local, local[4] or local[*] for threads on one machine, spark://host:7077 for Spark's Standalone manager, yarn for Hadoop YARN, and k8s://https://host:443 for Kubernetes.
In client mode (the default) the driver runs inside the spark-submit process on the machine you submit from, so its output appears in your terminal. In cluster mode the cluster manager starts the driver on a cluster node, so the job keeps running if your laptop disconnects. For example, spark-submit --master yarn --deploy-mode cluster --executor-memory 8g job.py 2026-01-05 runs job.py on YARN with the date passed to sys.argv.
Why we use it
Notebooks hide how Spark starts, but production jobs, schedulers (Airflow, cron) and CI pipelines all launch Spark through spark-submit. Knowing the flags lets you run the same script on a laptop and on a cluster just by changing the command, not the code.
When to use it
Use spark-submit for scheduled batch jobs and anything you deploy. Choose client mode for interactive work and debugging from a gateway node, cluster mode for production. Keep .master() out of job code so the launcher decides; notebooks and managed platforms create the session for you.
How it works
- Run spark-submit with master, deploy mode and resources
- Parse flags and merge with spark-defaults.conf
- Deploy mode is cluster? Yes: Driver starts on a cluster node. No: Driver runs in the spark-submit process.
- Driver requests executors from the cluster manager
- Script runs; actions become jobs on the executors
- spark.stop() releases executors; exit code reports success
Syntax
spark-submit \
--master local[*] | yarn | k8s://https://host:443 | spark://host:7077 \
--deploy-mode client | cluster \
--name daily-sales \
--num-executors 10 --executor-cores 4 --executor-memory 8g --driver-memory 4g \
--conf spark.sql.shuffle.partitions=200 \
--py-files deps.zip --packages group:artifact:version \
job.py arg1 arg2Example
import os, shutil, subprocess, tempfile, textwrap
# 1) A small job script. It sets NO .master(): spark-submit decides where it runs.
job = textwrap.dedent('''
import sys
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("from-code").getOrCreate()
sc = spark.sparkContext
print("master:", sc.master)
print("deploy mode:", sc.getConf().get("spark.submit.deployMode"))
print("app name:", sc.appName)
print("shuffle partitions:", spark.conf.get("spark.sql.shuffle.partitions"))
print("run date:", sys.argv[1])
orders = spark.createDataFrame(
[("North", 55000.0), ("West", 30000.0), ("North", 1500.0)],
"region STRING, amount DOUBLE")
for r in orders.groupBy("region").agg(F.sum("amount").alias("sales")).orderBy("region").collect():
print(r.region, r.sales)
spark.stop()
''')
workdir = tempfile.mkdtemp()
script = os.path.join(workdir, "daily_sales.py")
with open(script, "w") as f:
f.write(job)
# 2) Submit it. On a real cluster only --master / --deploy-mode / resources change.
cmd = [shutil.which("spark-submit"),
"--master", "local[2]", # yarn | k8s://https://host:443 | spark://host:7077
"--deploy-mode", "client", # cluster = driver runs inside the cluster
"--name", "from-cli", # loses to appName() set in code
"--conf", "spark.sql.shuffle.partitions=4",
"--conf", "spark.ui.enabled=false",
script, "2026-01-05"] # arguments after the script go to sys.argv
result = subprocess.run(cmd, capture_output=True, text=True)
print(result.stdout.strip())
print("exit code:", result.returncode)
# Output:
# master: local[2]
# deploy mode: client
# app name: from-code
# shuffle partitions: 4
# run date: 2026-01-05
# North 56500.0
# West 30000.0
# exit code: 0
Key points
- local[*] = one thread per CPU core; local[2] = two task slots in one JVM
- Client mode: driver on the submitting machine; cluster mode: driver inside the cluster
- Arguments after the script name arrive in sys.argv
- Settings in code (SparkConf/builder) beat spark-submit flags, which beat spark-defaults.conf
- --py-files ships your own .py/.zip modules; --packages pulls JVM libraries such as connectors
- Standalone mode does not support cluster deploy mode for Python apps