spark-submit, deploy modes & master URLs

Apache Spark · Spark Architecture

Theory

spark-submit is the command that launches a Spark application. You give it a master URL (where to run), a deploy mode (where the driver runs), resources (executor memory, cores, count), extra settings with --conf, dependencies with --py-files or --packages, then the script and its arguments. The master URL picks the cluster manager: local, local[4] or local[*] for threads on one machine, spark://host:7077 for Spark's Standalone manager, yarn for Hadoop YARN, and k8s://https://host:443 for Kubernetes.

In client mode (the default) the driver runs inside the spark-submit process on the machine you submit from, so its output appears in your terminal. In cluster mode the cluster manager starts the driver on a cluster node, so the job keeps running if your laptop disconnects. For example, spark-submit --master yarn --deploy-mode cluster --executor-memory 8g job.py 2026-01-05 runs job.py on YARN with the date passed to sys.argv.

Why we use it

Notebooks hide how Spark starts, but production jobs, schedulers (Airflow, cron) and CI pipelines all launch Spark through spark-submit. Knowing the flags lets you run the same script on a laptop and on a cluster just by changing the command, not the code.

When to use it

Use spark-submit for scheduled batch jobs and anything you deploy. Choose client mode for interactive work and debugging from a gateway node, cluster mode for production. Keep .master() out of job code so the launcher decides; notebooks and managed platforms create the session for you.

How it works

  1. Run spark-submit with master, deploy mode and resources
  2. Parse flags and merge with spark-defaults.conf
  3. Deploy mode is cluster? Yes: Driver starts on a cluster node. No: Driver runs in the spark-submit process.
  4. Driver requests executors from the cluster manager
  5. Script runs; actions become jobs on the executors
  6. spark.stop() releases executors; exit code reports success

Syntax

spark-submit \
  --master local[*] | yarn | k8s://https://host:443 | spark://host:7077 \
  --deploy-mode client | cluster \
  --name daily-sales \
  --num-executors 10 --executor-cores 4 --executor-memory 8g --driver-memory 4g \
  --conf spark.sql.shuffle.partitions=200 \
  --py-files deps.zip --packages group:artifact:version \
  job.py arg1 arg2

Example

import os, shutil, subprocess, tempfile, textwrap

# 1) A small job script. It sets NO .master(): spark-submit decides where it runs.
job = textwrap.dedent('''
    import sys
    from pyspark.sql import SparkSession, functions as F

    spark = SparkSession.builder.appName("from-code").getOrCreate()
    sc = spark.sparkContext
    print("master:", sc.master)
    print("deploy mode:", sc.getConf().get("spark.submit.deployMode"))
    print("app name:", sc.appName)
    print("shuffle partitions:", spark.conf.get("spark.sql.shuffle.partitions"))
    print("run date:", sys.argv[1])

    orders = spark.createDataFrame(
        [("North", 55000.0), ("West", 30000.0), ("North", 1500.0)],
        "region STRING, amount DOUBLE")
    for r in orders.groupBy("region").agg(F.sum("amount").alias("sales")).orderBy("region").collect():
        print(r.region, r.sales)
    spark.stop()
''')
workdir = tempfile.mkdtemp()
script = os.path.join(workdir, "daily_sales.py")
with open(script, "w") as f:
    f.write(job)

# 2) Submit it. On a real cluster only --master / --deploy-mode / resources change.
cmd = [shutil.which("spark-submit"),
       "--master", "local[2]",            # yarn | k8s://https://host:443 | spark://host:7077
       "--deploy-mode", "client",         # cluster = driver runs inside the cluster
       "--name", "from-cli",              # loses to appName() set in code
       "--conf", "spark.sql.shuffle.partitions=4",
       "--conf", "spark.ui.enabled=false",
       script, "2026-01-05"]              # arguments after the script go to sys.argv
result = subprocess.run(cmd, capture_output=True, text=True)
print(result.stdout.strip())
print("exit code:", result.returncode)

# Output:
# master: local[2]
# deploy mode: client
# app name: from-code
# shuffle partitions: 4
# run date: 2026-01-05
# North 56500.0
# West 30000.0
# exit code: 0

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic