Spark Connect & Client Apps

Apache Spark · MLlib & Ecosystem

Theory

Spark Connect is a client-server architecture for Spark. In classic PySpark, your Python program starts the driver JVM itself and talks to it through Py4J, so the client must live next to the cluster's driver. With Spark Connect, the driver runs as a server, and your application uses a thin client library. The client turns DataFrame operations into unresolved logical plans encoded as protocol buffers, sends them over gRPC, and receives results back as Apache Arrow batches.

You connect with SparkSession.builder.remote("sc://host:15002").getOrCreate() instead of master(...); a server is started with sbin/start-connect-server.sh, and the lightweight pyspark-client package is enough on the client side. The differences: RDD APIs and SparkContext are not available, JVM internals such as df._jdf do not exist and the client cannot change static cluster configuration. Official clients exist for Python and Scala.

Why we use it

Decoupling the client from the driver lets you run Spark code from IDEs, notebooks, web apps and small containers without a local JVM cluster setup. Client and server can be upgraded separately, and a client crash or memory problem does not take down the shared driver.

When to use it

Use Spark Connect when developing in a local IDE against a remote cluster, building applications that send Spark queries, or using shared multi-user clusters. Stay with classic mode if your code relies on RDDs, SparkContext or direct JVM access, or rewrite those parts with DataFrame APIs first.

How it works

  1. Client app builds a DataFrame query
  2. Client encodes an unresolved logical plan (protobuf)
  3. Plan is sent over gRPC to the Spark Connect server
  4. Server analyses, optimises and runs it on the cluster
  5. Does the code use RDDs or SparkContext? Yes: Not supported: rewrite with DataFrame APIs. No: Results stream back as Arrow batches.
  6. Client receives the result

Syntax

# server side
./sbin/start-connect-server.sh
# client side (pip install pyspark-client)
spark = SparkSession.builder.remote("sc://spark-host:15002").getOrCreate()
# or: export SPARK_REMOTE="sc://spark-host:15002"; pyspark --remote "sc://localhost"

Example

from pyspark.sql import SparkSession, functions as F

# For a real server use .remote("sc://spark-host:15002").
# "local[*]" starts a local Spark Connect server for practice (needs the connect extras:
# pip install "pyspark[connect]").
spark = SparkSession.builder.remote("local[*]").appName("techtown-connect").getOrCreate()
print("session class:", type(spark).__module__ + "." + type(spark).__name__)

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0), (2, 102, "West", "Phone", 2, 30000.0),
    (3, 101, "North", "Mouse", 3, 1500.0),   (4, 103, "South", "Laptop", 1, 52000.0),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE")

# The same DataFrame API: the client sends a plan over gRPC, results come back as Arrow
orders.groupBy("region").agg(F.sum("amount").alias("sales")).orderBy("region").show()

try:
    orders.rdd.count()                        # RDDs are not part of the Connect protocol
except Exception as e:
    print("RDD API on Spark Connect ->", type(e).__name__)

spark.stop()

# Output:
# session class: pyspark.sql.connect.session.SparkSession
# +------+-------+
# |region|  sales|
# +------+-------+
# | North|56500.0|
# | South|52000.0|
# |  West|30000.0|
# +------+-------+
#
# RDD API on Spark Connect -> PySparkNotImplementedError

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic