What is Databricks & the Lakehouse
Databricks · Databricks Fundamentals
Theory
Databricks is a cloud data and AI platform built by the creators of Apache Spark. It runs on AWS, Azure and Google Cloud and lets data engineers, analysts and data scientists work on the same data with notebooks, SQL, jobs, pipelines, dashboards and ML tools. Your data stays in cheap cloud object storage (S3, ADLS, GCS) as open table formats such as Delta Lake or Apache Iceberg.
The lakehouse idea is to combine the low-cost, open storage of a data lake with the reliability and speed of a data warehouse: ACID transactions, schemas, fine-grained security and fast SQL. For example, raw order files land in storage, a Spark job cleans them into a Delta table main.sales.orders, analysts query it from a SQL warehouse, and a data scientist trains a model on the same table, all governed by Unity Catalog.
Why we use it
Companies used to copy data between a lake (for ML and raw files) and a warehouse (for BI). That meant two copies, two security models and stale numbers. A lakehouse keeps one governed copy of the data that every workload reads.
When to use it
Use Databricks when you need large-scale data engineering, streaming, ML and SQL analytics on the same data in your own cloud storage. For a small team that only runs BI on modest data, a pure SQL warehouse product may be simpler.
How it works
- Raw files land in cloud object storage
- Spark or a pipeline ingests them into Delta tables
- Unity Catalog registers the tables and applies permissions
- Who needs the data? Yes: Analysts query it on a SQL warehouse. No: Engineers or ML users read it in notebooks and jobs.
- Everyone works on the same governed table
Syntax
Workspace URL: https://<workspace>.cloud.databricks.com
Three-level name: <catalog>.<schema>.<table>
spark.read.table("main.sales.orders")
SELECT ... FROM main.sales.ordersExample
# Python cell in a Databricks notebook
df = spark.read.table("main.sales.orders")
(df.groupBy("region")
.sum("amount")
.withColumnRenamed("sum(amount)", "total_amount")
.orderBy("total_amount", ascending=False)
.display())
-- Same question in SQL
SELECT region, SUM(amount) AS total_amount
FROM main.sales.orders
GROUP BY region
ORDER BY total_amount DESC;Key points
- Data lives in your cloud object storage in open formats (Delta Lake, Apache Iceberg)
- Unity Catalog governs tables, files, models and functions in one place
- The same table serves ETL, SQL/BI and ML, so there is one copy of the truth
- Apache Spark and the Photon engine do the processing
- Databricks runs on AWS, Azure and Google Cloud