Notebooks (languages, magic commands %sql %md %pip %run)
Databricks · Notebooks & Development
Theory
A Databricks notebook is a web document made of cells that run on attached compute. Every notebook has a default language (Python, SQL, Scala or R) chosen in the language drop-down, and a cell can switch language with a magic command on its first line: %python, %sql, %scala or %r. Other magics: %md writes formatted Markdown text, %pip installs Python libraries for this notebook session, %run runs another notebook inline so its functions and variables become available, and %sh runs shell commands on the driver.
Cells share one Spark session, so a DataFrame registered as a temp view in a Python cell can be queried in a %sql cell. display() shows a sortable table with built-in charts. For example, cell 1 is %md with a heading, cell 2 is %run ./config to load the catalog name, cell 3 reads main.sales.orders in Python and cell 4 summarises it in %sql.
Why we use it
Real work mixes languages: Python for logic, SQL for quick analysis, Markdown for explanations. Magic commands let one notebook do all of this and share code across notebooks without packaging.
When to use it
Use notebooks for exploration, teaching, and as job tasks. Put shared helpers in a notebook you %run, or better in Python files you import. Use %pip at the top of the notebook. Move large reusable code to files or packages when it grows.
How it works
- Open a notebook and attach compute
- Read the first line of the next cell
- Does it start with a magic command such as %sql or %md? Yes: Run the cell in that language or tool. No: Run it in the notebook's default language.
- Show the result under the cell
- Move to the next cell (Shift+Enter)
Syntax
%python | %sql | %scala | %r (switch cell language)
%md Markdown text
%pip install <package>
%run ./path/to/notebook
%sh ls /Volumes/main/sales/raw_files
display(df)Example
# Cell 1
%md
## Daily orders report
# Cell 2
%run ./config # defines catalog = "main", schema = "sales"
# Cell 3 (Python)
df = spark.read.table(f"{catalog}.{schema}.orders")
df.createOrReplaceTempView("orders_v")
# Cell 4
%sql
SELECT region, SUM(amount) AS total
FROM orders_v
GROUP BY region
ORDER BY total DESC;Key points
- The magic command must be the first line of the cell
- %run must be alone in its cell and runs the other notebook in the same session
- %pip resets the Python state, so put it at the very top of the notebook
- All languages share one Spark session; temp views pass data between them
- display() gives a table with built-in charts; print() gives plain text