Databricks Connect & IDEs

Databricks · Notebooks & Development

Theory

Databricks Connect is a client library that lets you write Spark code in a local IDE such as VS Code, PyCharm or IntelliJ IDEA, or in your own app, and run it on Databricks compute. It is built on Spark Connect: your local Python process builds the DataFrame plan and sends it to a cluster or serverless compute, which runs it and sends back only the results. Unity Catalog permissions apply exactly as in a notebook.

You install the databricks-connect package (its version should match the compute's runtime or serverless environment), authenticate with databricks auth login, then create a session with DatabricksSession.builder.serverless().getOrCreate() or point it at a cluster ID. The Databricks extension for Visual Studio Code builds on this to run and debug files and to work with bundles. For example, Meera writes transforms.py locally, sets a breakpoint, and steps through code that reads main.sales.orders on serverless compute.

Why we use it

IDEs give you debugging, refactoring, linting, unit tests and AI coding tools that notebooks lack. Databricks Connect keeps that local workflow while the heavy data work still runs next to the data on Databricks.

When to use it

Use Databricks Connect for software-style development: packages, unit tests, debugging and apps that need Spark. Use notebooks for quick exploration and visual output. Code that needs RDDs or local Spark internals will not work over Spark Connect.

How it works

  1. Install databricks-connect in a local virtual environment
  2. databricks auth login to the workspace
  3. Create a DatabricksSession (serverless or cluster ID)
  4. Write DataFrame code in the IDE
  5. Plan is sent over Spark Connect to Databricks
  6. Results come back; debug locally with breakpoints

Syntax

pip install databricks-connect
databricks auth login --host <workspace-url>
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.serverless().getOrCreate()
spark = DatabricksSession.builder.clusterId("<cluster-id>").getOrCreate()

Example

# transforms.py (runs in VS Code or PyCharm)
from databricks.connect import DatabricksSession
from pyspark.sql import functions as F

spark = DatabricksSession.builder.serverless().getOrCreate()

orders = spark.read.table("main.sales.orders")
summary = (orders.groupBy("region")
                 .agg(F.sum("amount").alias("total_amount"))
                 .orderBy(F.desc("total_amount")))

summary.show()   # plan runs on Databricks, rows come back to your laptop

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic