Notebooks (languages, magic commands %sql %md %pip %run)

Databricks · Notebooks & Development

Theory

A Databricks notebook is a web document made of cells that run on attached compute. Every notebook has a default language (Python, SQL, Scala or R) chosen in the language drop-down, and a cell can switch language with a magic command on its first line: %python, %sql, %scala or %r. Other magics: %md writes formatted Markdown text, %pip installs Python libraries for this notebook session, %run runs another notebook inline so its functions and variables become available, and %sh runs shell commands on the driver.

Cells share one Spark session, so a DataFrame registered as a temp view in a Python cell can be queried in a %sql cell. display() shows a sortable table with built-in charts. For example, cell 1 is %md with a heading, cell 2 is %run ./config to load the catalog name, cell 3 reads main.sales.orders in Python and cell 4 summarises it in %sql.

Why we use it

Real work mixes languages: Python for logic, SQL for quick analysis, Markdown for explanations. Magic commands let one notebook do all of this and share code across notebooks without packaging.

When to use it

Use notebooks for exploration, teaching, and as job tasks. Put shared helpers in a notebook you %run, or better in Python files you import. Use %pip at the top of the notebook. Move large reusable code to files or packages when it grows.

How it works

  1. Open a notebook and attach compute
  2. Read the first line of the next cell
  3. Does it start with a magic command such as %sql or %md? Yes: Run the cell in that language or tool. No: Run it in the notebook's default language.
  4. Show the result under the cell
  5. Move to the next cell (Shift+Enter)

Syntax

%python | %sql | %scala | %r   (switch cell language)
%md  Markdown text
%pip install <package>
%run ./path/to/notebook
%sh ls /Volumes/main/sales/raw_files
display(df)

Example

# Cell 1
%md
## Daily orders report

# Cell 2
%run ./config        # defines catalog = "main", schema = "sales"

# Cell 3 (Python)
df = spark.read.table(f"{catalog}.{schema}.orders")
df.createOrReplaceTempView("orders_v")

# Cell 4
%sql
SELECT region, SUM(amount) AS total
FROM orders_v
GROUP BY region
ORDER BY total DESC;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic