Spark Connect & Client Apps
Apache Spark · MLlib & Ecosystem
Theory
Spark Connect is a client-server architecture for Spark. In classic PySpark, your Python program starts the driver JVM itself and talks to it through Py4J, so the client must live next to the cluster's driver. With Spark Connect, the driver runs as a server, and your application uses a thin client library. The client turns DataFrame operations into unresolved logical plans encoded as protocol buffers, sends them over gRPC, and receives results back as Apache Arrow batches.
You connect with SparkSession.builder.remote("sc://host:15002").getOrCreate() instead of master(...); a server is started with sbin/start-connect-server.sh, and the lightweight pyspark-client package is enough on the client side. The differences: RDD APIs and SparkContext are not available, JVM internals such as df._jdf do not exist and the client cannot change static cluster configuration. Official clients exist for Python and Scala.
Why we use it
Decoupling the client from the driver lets you run Spark code from IDEs, notebooks, web apps and small containers without a local JVM cluster setup. Client and server can be upgraded separately, and a client crash or memory problem does not take down the shared driver.
When to use it
Use Spark Connect when developing in a local IDE against a remote cluster, building applications that send Spark queries, or using shared multi-user clusters. Stay with classic mode if your code relies on RDDs, SparkContext or direct JVM access, or rewrite those parts with DataFrame APIs first.
How it works
- Client app builds a DataFrame query
- Client encodes an unresolved logical plan (protobuf)
- Plan is sent over gRPC to the Spark Connect server
- Server analyses, optimises and runs it on the cluster
- Does the code use RDDs or SparkContext? Yes: Not supported: rewrite with DataFrame APIs. No: Results stream back as Arrow batches.
- Client receives the result
Syntax
# server side
./sbin/start-connect-server.sh
# client side (pip install pyspark-client)
spark = SparkSession.builder.remote("sc://spark-host:15002").getOrCreate()
# or: export SPARK_REMOTE="sc://spark-host:15002"; pyspark --remote "sc://localhost"Example
from pyspark.sql import SparkSession, functions as F
# For a real server use .remote("sc://spark-host:15002").
# "local[*]" starts a local Spark Connect server for practice (needs the connect extras:
# pip install "pyspark[connect]").
spark = SparkSession.builder.remote("local[*]").appName("techtown-connect").getOrCreate()
print("session class:", type(spark).__module__ + "." + type(spark).__name__)
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0), (2, 102, "West", "Phone", 2, 30000.0),
(3, 101, "North", "Mouse", 3, 1500.0), (4, 103, "South", "Laptop", 1, 52000.0),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE")
# The same DataFrame API: the client sends a plan over gRPC, results come back as Arrow
orders.groupBy("region").agg(F.sum("amount").alias("sales")).orderBy("region").show()
try:
orders.rdd.count() # RDDs are not part of the Connect protocol
except Exception as e:
print("RDD API on Spark Connect ->", type(e).__name__)
spark.stop()
# Output:
# session class: pyspark.sql.connect.session.SparkSession
# +------+-------+
# |region| sales|
# +------+-------+
# | North|56500.0|
# | South|52000.0|
# | West|30000.0|
# +------+-------+
#
# RDD API on Spark Connect -> PySparkNotImplementedError
Key points
- Client sends unresolved logical plans over gRPC; results return as Arrow batches
- Connect with SparkSession.builder.remote('sc://host:15002')
- DataFrame and SQL code is unchanged; RDD and SparkContext APIs are not supported
- The thin pyspark-client package needs no local JVM
- Client and server are separate processes, so a client crash does not kill the driver