Storage credentials & external locations
Databricks · Unity Catalog & Governance
Theory
Unity Catalog needs permission to read and write your cloud storage. A storage credential wraps a cloud identity: an IAM role on AWS, an Azure managed identity (access connector) or a service principal, or a Google service account. An external location combines a storage credential with a cloud path, such as s3://techtown-landing/, and is the object you grant privileges on. Both are created by admins or users with the right metastore privileges.
Users never see the cloud keys. Instead they get privileges on the external location, such as READ FILES, WRITE FILES, CREATE EXTERNAL TABLE and CREATE EXTERNAL VOLUME, and Unity Catalog hands out short-lived credentials when a query touches that path. For example, the admin creates the credential landing_cred for an IAM role, then the external location landing_loc on s3://techtown-landing/ and grants READ FILES to the data-engineers group.
Why we use it
Before Unity Catalog, teams pasted access keys into notebooks or used workspace mounts that anyone could read. Storage credentials and external locations keep secrets out of code and control access to cloud paths with normal GRANT statements.
When to use it
Create an external location for every bucket or container path Databricks must reach: landing zones, external tables, external volumes and managed storage roots. Grant on external locations, not on storage credentials, for everyday users.
How it works
- Cloud admin creates an IAM role or managed identity
- Databricks admin creates a storage credential for it
- Create an external location: credential + cloud path
- Grant READ FILES or CREATE EXTERNAL TABLE to a group
- Does the user's query touch a path under that location? Yes: Unity Catalog checks privileges and vends a temporary token. No: Access is denied.
- Data is read or written securely
Syntax
CREATE STORAGE CREDENTIAL (usually via UI: Catalog → gear icon → Credentials)
CREATE EXTERNAL LOCATION name URL 's3://bucket/path' WITH (STORAGE CREDENTIAL cred)
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION name TO `group`
GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION name TO `group`Example
-- Admin: bind a path to an existing storage credential
CREATE EXTERNAL LOCATION IF NOT EXISTS landing_loc
URL 's3://techtown-landing/'
WITH (STORAGE CREDENTIAL landing_cred)
COMMENT 'Raw landing zone';
GRANT READ FILES, CREATE EXTERNAL TABLE
ON EXTERNAL LOCATION landing_loc TO `data-engineers`;
-- Engineer: read files directly from the path
SELECT * FROM read_files('s3://techtown-landing/orders/', format => 'json');
SHOW EXTERNAL LOCATIONS;Key points
- Storage credential = cloud identity; external location = credential + path
- Grant users privileges on external locations, not on storage credentials
- Privileges include READ FILES, WRITE FILES, CREATE EXTERNAL TABLE, CREATE EXTERNAL VOLUME
- Unity Catalog issues short-lived credentials, so no keys appear in notebooks
- External locations should not overlap each other