Metastore & three-level namespace
Databricks · Unity Catalog & Governance
Theory
Unity Catalog is Databricks' governance layer for data and AI. At the top is the metastore: one per cloud region, owned at the account level and attached to every workspace in that region, so permissions and objects are shared across those workspaces. Inside the metastore, objects are organized in a three-level namespace: catalog.schema.object. A catalog is the top grouping (often per environment or business unit, such as dev, prod or main); a schema (also called a database) groups related objects; and objects are tables, views, volumes, functions and models.
You refer to a table by its full name, main.sales.orders, or set defaults with USE CATALOG main and USE SCHEMA sales. Built-in catalogs include system (system tables) and samples, plus hive_metastore for legacy tables. For example, the same orders table can be read from the dev and prod workspaces in one region because both are attached to the same metastore.
Why we use it
Before Unity Catalog, each workspace had its own Hive metastore with separate permissions, so the same data was governed differently in each place. One metastore and a clear namespace give one set of permissions, lineage and audit for all workspaces.
When to use it
Always use three-level names in production code so it does not depend on the current catalog. Design catalogs around environments or domains (dev, prod, finance) and schemas around subject areas. Treat hive_metastore as legacy and migrate tables out of it.
How it works
- Query names main.sales.orders
- Resolve catalog main in the region's metastore
- Resolve schema sales inside main
- Resolve table orders inside sales
- Does the user have USE CATALOG, USE SCHEMA and SELECT? Yes: Read the table's files. No: Fail with a permission error.
- Return the rows
Syntax
<catalog>.<schema>.<table | view | volume | function | model>
CREATE CATALOG [IF NOT EXISTS] name
CREATE SCHEMA [IF NOT EXISTS] catalog.schema
USE CATALOG main; USE SCHEMA sales;
SHOW CATALOGS | SHOW SCHEMAS IN main | SHOW TABLES IN main.salesExample
CREATE CATALOG IF NOT EXISTS main;
CREATE SCHEMA IF NOT EXISTS main.sales
COMMENT 'Sales orders and customers';
USE CATALOG main;
USE SCHEMA sales;
SHOW TABLES IN main.sales;
-- Always-safe full name
SELECT c.city, SUM(o.amount) AS revenue
FROM main.sales.orders o
JOIN main.sales.customers c ON o.customer_id = c.customer_id
GROUP BY c.city;Key points
- One metastore per region, shared by all workspaces attached to it
- Namespace is catalog.schema.object; schema and database mean the same thing
- Objects include tables, views, volumes, functions and registered models
- system and samples are built-in catalogs; hive_metastore is the legacy one
- Use full three-level names in jobs so they do not depend on defaults