Retries, Notifications & Repair Runs

Databricks · Lakeflow Jobs

Theory

Production jobs need to recover from hiccups and tell people when they cannot. Each task can have a retry policy (click Add next to Retries) with a number of retries and a wait between them, plus a Timeout so a stuck task is stopped. Notifications can be set on the job or per task for Start, Success, Failure, duration warning and streaming backlog, sent to email or to system destinations such as Slack, Microsoft Teams, PagerDuty or a webhook. When a multi-task run fails, Repair run reruns only the failed tasks and the tasks that depend on them, reusing the successful results instead of starting over.

Example: in orders_daily, build_gold fails because a source table was locked. After fixing it, Ravi opens the failed run, clicks Repair run, and only build_gold and refresh_dashboard run again; ingest_orders is not repeated.

Why we use it

Transient errors are common in distributed systems. Retries fix most of them automatically, alerts make sure humans see the rest quickly, and repair runs save the time and cost of rerunning tasks that already succeeded.

When to use it

Add retries to tasks that hit external systems or flaky sources, and avoid retrying non-idempotent steps that could double-write. Alert on failure at least; add duration warnings for SLAs. Use Repair run after fixing a failure; use a fresh Run now when inputs changed for all tasks.

How it works

  1. Task fails during a run
  2. Retries left for this task? Yes: Wait the retry interval and run the task again. No: Mark the task Failed.
  3. Downstream tasks are skipped (unless Run if says otherwise)
  4. Failure notification goes to email or Slack
  5. Engineer fixes the cause and clicks Repair run
  6. Only failed and dependent tasks rerun; the run finishes

Syntax

Task settings: Retries → Add (number, interval, retry on timeout) | Timeout | Notifications
Job details: Job notifications → Add notification (Start, Success, Failure, Duration warning)
Failed run page: Repair run → select tasks → Repair run (optionally change parameters)
API: POST /api/2.2/jobs/runs/repair

Example

# Retry, timeout and notification settings in a bundle
resources:
  jobs:
    orders_daily:
      name: orders_daily
      email_notifications:
        on_failure: [data-oncall@techtown.in]
      tasks:
        - task_key: build_gold
          max_retries: 2
          min_retry_interval_millis: 300000    # 5 minutes
          retry_on_timeout: false
          timeout_seconds: 3600
          email_notifications:
            on_duration_warning_threshold_exceeded: [ravi@techtown.in]
          sql_task:
            file: { path: ../src/build_gold.sql }
            warehouse_id: ${var.warehouse_id}

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic