VACUUM & retention
Databricks · Delta Lake
Theory
When Delta rewrites data (UPDATE, MERGE, OPTIMIZE, overwrite), the old Parquet files are removed from the transaction log but stay in storage so that time travel and running readers still work. VACUUM permanently deletes files that are no longer referenced by the current version and are older than the retention threshold, which is 7 days (168 hours) by default. It frees storage and cost but ends time travel to versions that needed those files.
VACUUM t DRY RUN lists the files it would delete without deleting them. VACUUM t RETAIN 240 HOURS keeps more history; a value below 7 days is blocked by a safety check because it can break long-running queries and streams. The table property delta.deletedFileRetentionDuration changes the default. On Unity Catalog managed tables, predictive optimization runs VACUUM, OPTIMIZE and ANALYZE for you. For example, after a month of daily MERGE jobs, VACUUM removes the superseded files older than a week.
Why we use it
Without VACUUM, every rewrite leaves old files behind and storage grows forever. With careless VACUUM settings you lose the ability to roll back. Knowing retention lets you balance cost against recovery.
When to use it
Run VACUUM regularly on external tables or where predictive optimization is off, usually weekly with the default retention. Keep retention long enough for your time travel and streaming needs. Never lower it below 7 days without a clear reason.
How it works
- Old files pile up after MERGE and OPTIMIZE
- VACUUM ... DRY RUN to preview
- Are the files unreferenced and older than the retention period? Yes: Delete them from storage. No: Keep them for time travel.
- Storage cost goes down
- Time travel now reaches back only as far as the retention
Syntax
VACUUM t -- default 7-day retention
VACUUM t DRY RUN
VACUUM t RETAIN 240 HOURS
ALTER TABLE t SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 14 days')
ALTER SCHEMA s ENABLE PREDICTIVE OPTIMIZATIONExample
-- See what would be deleted
VACUUM main.sales.orders DRY RUN;
-- Delete unreferenced files older than 7 days (default)
VACUUM main.sales.orders;
-- Keep two weeks of time travel on this table
ALTER TABLE main.sales.orders SET TBLPROPERTIES (
'delta.deletedFileRetentionDuration' = 'interval 14 days');
-- Let Databricks do maintenance for managed tables in this schema
ALTER SCHEMA main.sales ENABLE PREDICTIVE OPTIMIZATION;Key points
- VACUUM deletes only files not referenced by the current version and older than retention
- Default retention is 7 days; going lower is blocked by a safety check
- After VACUUM you cannot time travel to versions that needed the deleted files
- Use DRY RUN first to see what will be removed
- Predictive optimization runs VACUUM automatically on Unity Catalog managed tables