What is Databricks & the Lakehouse

Databricks · Databricks Fundamentals

Theory

Databricks is a cloud data and AI platform built by the creators of Apache Spark. It runs on AWS, Azure and Google Cloud and lets data engineers, analysts and data scientists work on the same data with notebooks, SQL, jobs, pipelines, dashboards and ML tools. Your data stays in cheap cloud object storage (S3, ADLS, GCS) as open table formats such as Delta Lake or Apache Iceberg.

The lakehouse idea is to combine the low-cost, open storage of a data lake with the reliability and speed of a data warehouse: ACID transactions, schemas, fine-grained security and fast SQL. For example, raw order files land in storage, a Spark job cleans them into a Delta table main.sales.orders, analysts query it from a SQL warehouse, and a data scientist trains a model on the same table, all governed by Unity Catalog.

Why we use it

Companies used to copy data between a lake (for ML and raw files) and a warehouse (for BI). That meant two copies, two security models and stale numbers. A lakehouse keeps one governed copy of the data that every workload reads.

When to use it

Use Databricks when you need large-scale data engineering, streaming, ML and SQL analytics on the same data in your own cloud storage. For a small team that only runs BI on modest data, a pure SQL warehouse product may be simpler.

How it works

  1. Raw files land in cloud object storage
  2. Spark or a pipeline ingests them into Delta tables
  3. Unity Catalog registers the tables and applies permissions
  4. Who needs the data? Yes: Analysts query it on a SQL warehouse. No: Engineers or ML users read it in notebooks and jobs.
  5. Everyone works on the same governed table

Syntax

Workspace URL: https://<workspace>.cloud.databricks.com
Three-level name: <catalog>.<schema>.<table>
spark.read.table("main.sales.orders")
SELECT ... FROM main.sales.orders

Example

# Python cell in a Databricks notebook
df = spark.read.table("main.sales.orders")

(df.groupBy("region")
   .sum("amount")
   .withColumnRenamed("sum(amount)", "total_amount")
   .orderBy("total_amount", ascending=False)
   .display())

-- Same question in SQL
SELECT region, SUM(amount) AS total_amount
FROM main.sales.orders
GROUP BY region
ORDER BY total_amount DESC;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic