Databricks Connect & IDEs
Databricks · Notebooks & Development
Theory
Databricks Connect is a client library that lets you write Spark code in a local IDE such as VS Code, PyCharm or IntelliJ IDEA, or in your own app, and run it on Databricks compute. It is built on Spark Connect: your local Python process builds the DataFrame plan and sends it to a cluster or serverless compute, which runs it and sends back only the results. Unity Catalog permissions apply exactly as in a notebook.
You install the databricks-connect package (its version should match the compute's runtime or serverless environment), authenticate with databricks auth login, then create a session with DatabricksSession.builder.serverless().getOrCreate() or point it at a cluster ID. The Databricks extension for Visual Studio Code builds on this to run and debug files and to work with bundles. For example, Meera writes transforms.py locally, sets a breakpoint, and steps through code that reads main.sales.orders on serverless compute.
Why we use it
IDEs give you debugging, refactoring, linting, unit tests and AI coding tools that notebooks lack. Databricks Connect keeps that local workflow while the heavy data work still runs next to the data on Databricks.
When to use it
Use Databricks Connect for software-style development: packages, unit tests, debugging and apps that need Spark. Use notebooks for quick exploration and visual output. Code that needs RDDs or local Spark internals will not work over Spark Connect.
How it works
- Install databricks-connect in a local virtual environment
- databricks auth login to the workspace
- Create a DatabricksSession (serverless or cluster ID)
- Write DataFrame code in the IDE
- Plan is sent over Spark Connect to Databricks
- Results come back; debug locally with breakpoints
Syntax
pip install databricks-connect
databricks auth login --host <workspace-url>
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.serverless().getOrCreate()
spark = DatabricksSession.builder.clusterId("<cluster-id>").getOrCreate()Example
# transforms.py (runs in VS Code or PyCharm)
from databricks.connect import DatabricksSession
from pyspark.sql import functions as F
spark = DatabricksSession.builder.serverless().getOrCreate()
orders = spark.read.table("main.sales.orders")
summary = (orders.groupBy("region")
.agg(F.sum("amount").alias("total_amount"))
.orderBy(F.desc("total_amount")))
summary.show() # plan runs on Databricks, rows come back to your laptopKey points
- Built on Spark Connect: local client builds the plan, Databricks executes it
- Match the databricks-connect version to the runtime or serverless environment
- Authenticate with databricks auth login (OAuth) and a config profile
- Unity Catalog permissions are enforced on the remote compute
- The VS Code extension adds run/debug, bundle and sync features on top