Retries, Notifications & Repair Runs
Databricks · Lakeflow Jobs
Theory
Production jobs need to recover from hiccups and tell people when they cannot. Each task can have a retry policy (click Add next to Retries) with a number of retries and a wait between them, plus a Timeout so a stuck task is stopped. Notifications can be set on the job or per task for Start, Success, Failure, duration warning and streaming backlog, sent to email or to system destinations such as Slack, Microsoft Teams, PagerDuty or a webhook. When a multi-task run fails, Repair run reruns only the failed tasks and the tasks that depend on them, reusing the successful results instead of starting over.
Example: in orders_daily, build_gold fails because a source table was locked. After fixing it, Ravi opens the failed run, clicks Repair run, and only build_gold and refresh_dashboard run again; ingest_orders is not repeated.
Why we use it
Transient errors are common in distributed systems. Retries fix most of them automatically, alerts make sure humans see the rest quickly, and repair runs save the time and cost of rerunning tasks that already succeeded.
When to use it
Add retries to tasks that hit external systems or flaky sources, and avoid retrying non-idempotent steps that could double-write. Alert on failure at least; add duration warnings for SLAs. Use Repair run after fixing a failure; use a fresh Run now when inputs changed for all tasks.
How it works
- Task fails during a run
- Retries left for this task? Yes: Wait the retry interval and run the task again. No: Mark the task Failed.
- Downstream tasks are skipped (unless Run if says otherwise)
- Failure notification goes to email or Slack
- Engineer fixes the cause and clicks Repair run
- Only failed and dependent tasks rerun; the run finishes
Syntax
Task settings: Retries → Add (number, interval, retry on timeout) | Timeout | Notifications
Job details: Job notifications → Add notification (Start, Success, Failure, Duration warning)
Failed run page: Repair run → select tasks → Repair run (optionally change parameters)
API: POST /api/2.2/jobs/runs/repairExample
# Retry, timeout and notification settings in a bundle
resources:
jobs:
orders_daily:
name: orders_daily
email_notifications:
on_failure: [data-oncall@techtown.in]
tasks:
- task_key: build_gold
max_retries: 2
min_retry_interval_millis: 300000 # 5 minutes
retry_on_timeout: false
timeout_seconds: 3600
email_notifications:
on_duration_warning_threshold_exceeded: [ravi@techtown.in]
sql_task:
file: { path: ../src/build_gold.sql }
warehouse_id: ${var.warehouse_id}Key points
- Retries apply per task; each attempt is visible in the run details
- Make retried tasks idempotent (MERGE, COPY INTO, overwrite) to avoid duplicates
- Timeout stops a hung task; add duration warnings for SLA alerts
- Notifications can go to email or Slack/Teams/PagerDuty/webhook destinations
- Repair run reruns only failed and downstream tasks, keeping successful ones
- Fix the root cause before repairing, or the repair fails the same way