Running Spark: Databricks, EMR, Dataproc & Kubernetes

Apache Spark · MLlib & Ecosystem

Theory

The same PySpark code can run on many platforms; what changes is who manages the cluster. Every deployment has a driver (runs your main program and plans the work) and executors (run tasks), plus a cluster manager that provides them: Spark standalone, YARN (used by Hadoop-style clusters such as Amazon EMR and Google Cloud Dataproc) or Kubernetes (driver and executors run as pods). You submit with spark-submit or the platform's job service.

Managed platforms hide most of that. Databricks offers managed clusters, serverless compute, notebooks, jobs and governance on AWS, Azure and Google Cloud. Amazon EMR runs open-source Spark on EC2, on EKS or as EMR Serverless. Google Cloud Dataproc runs Spark clusters and Dataproc Serverless batches. On Kubernetes you run Spark yourself with spark-submit or an operator. Good job code reads paths and settings from arguments or config and does not hard-code master, so it moves between platforms.

Why we use it

Interviews and real projects expect you to know where Spark runs and what changes between platforms: cluster managers, deploy modes, how code and dependencies are shipped. Writing portable jobs avoids rewrites when a company changes cloud or platform.

When to use it

Choose Databricks when you want a managed lakehouse platform with notebooks, jobs and governance; EMR or Dataproc when you are on AWS or Google Cloud and want managed open-source Spark; Kubernetes when your company already runs everything on Kubernetes and has the skills to operate it.

How it works

  1. Write the job as functions plus a main()
  2. Package code and dependencies (wheel, zip or container image)
  3. Managed platform or self-managed? Yes: Submit via Databricks Jobs, EMR steps or Dataproc batches. No: spark-submit to YARN or Kubernetes.
  4. Cluster manager starts the driver and executors
  5. Job runs; logs and the Spark UI are available on the platform
  6. Results written to cloud storage or tables

Syntax

spark-submit --master yarn --deploy-mode cluster \
  --conf spark.executor.memory=4g --py-files deps.zip job.py --date 2026-01-05
spark-submit --master k8s://https://<api-server>:443 --deploy-mode cluster \
  --conf spark.kubernetes.container.image=<image> local:///app/job.py
# never hard-code .master() in job code; let the platform set it

Example

# Submit the SAME file everywhere; only the launcher and its flags change:
#   Local    : spark-submit --master local[*] daily_sales.py 2026-01-05
#   YARN/EMR : spark-submit --master yarn --deploy-mode cluster daily_sales.py 2026-01-05
#   Dataproc : gcloud dataproc jobs submit pyspark daily_sales.py --cluster=c1 --region=<region> -- 2026-01-05
#   K8s      : spark-submit --master k8s://https://<api-server>:443 --deploy-mode cluster \
#                --conf spark.kubernetes.container.image=<image> local:///app/daily_sales.py 2026-01-05
#   Databricks: add the file as a Python script task in a Job (the platform creates `spark`)
import sys
from pyspark.sql import SparkSession, functions as F


def build_report(spark, run_date):
    """Pure transformation logic: identical on Databricks, EMR, Dataproc or Kubernetes."""
    orders = spark.createDataFrame([
        (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
        (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-05"),
        (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    ], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING")
    return (orders.filter(F.col("order_date") == run_date)
                  .groupBy("region").agg(F.sum("amount").alias("sales"))
                  .orderBy("region"))


def main(argv):
    run_date = argv[1] if len(argv) > 1 else "2026-01-05"
    # No .master() here: spark-submit / the platform decides where this runs
    spark = SparkSession.builder.appName(f"daily-sales-{run_date}").getOrCreate()
    sc = spark.sparkContext
    print("master:", sc.master, "| app:", sc.appName, "| default parallelism:", sc.defaultParallelism)
    report = build_report(spark, run_date)
    report.show()
    # report.write.mode("overwrite").parquet(f"s3://bucket/reports/date={run_date}")


if __name__ == "__main__":
    main(sys.argv)

# Output:
# master: local[*] | app: daily-sales-2026-01-05 | default parallelism: 2
# +------+-------+
# |region|  sales|
# +------+-------+
# | North|55000.0|
# |  West|30000.0|
# +------+-------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic