Monitoring Job Runs
Databricks · Lakeflow Jobs
Theory
Databricks gives you several places to watch jobs. A job's Runs tab shows recent runs as a matrix of tasks by run, coloured by status, with duration and start time, so slow or failing tasks stand out. Clicking a run opens its task graph and each task's output, logs and error message. The Runs page in the sidebar lists job and pipeline runs across the workspace with filters for status and time. For trends and reporting, the system tables in system.lakeflow (jobs, job_tasks, job_run_timeline, job_task_run_timeline) hold run history you can query with SQL or put on a dashboard.
Example: Meera queries system.lakeflow.job_run_timeline for the last 7 days, counts runs and failures per job, and finds orders_daily failed 1 of 3 runs, which she pins to an ops dashboard.
Why we use it
Without monitoring, failures and slowdowns are found by angry dashboard users. Run views show what broke and why, and system tables let you spot trends such as rising duration or repeated failures before they cause missed SLAs.
When to use it
Use the job Runs tab and run details for debugging one job. Use the Runs page for a workspace-wide view. Use system.lakeflow tables for SLAs, failure trends and cost joins with system.billing. Set notifications so you do not need to watch the UI all day.
How it works
- Something looks wrong (alert, late data)
- Open the job's Runs tab and find the red or slow run
- Click the run → open the failed task → read output and error
- Is it a one-off or a trend? Yes: Query system.lakeflow tables over several days. No: Fix and Repair run.
- Add a dashboard or alert so it is caught earlier next time
Syntax
UI: Jobs & Pipelines → <job> → Runs tab (matrix view) | Sidebar → Runs
Run page: task graph → task → Output / Logs / Error
SQL: system.lakeflow.jobs, system.lakeflow.job_run_timeline,
system.lakeflow.job_task_run_timeline (columns include job_id, run_id, period_start_time, result_state)Example
-- Runs and failures per job in the last 7 days
SELECT j.name AS job_name,
COUNT(DISTINCT r.run_id) AS runs,
COUNT(DISTINCT CASE WHEN r.result_state = 'FAILED' THEN r.run_id END) AS failed_runs
FROM system.lakeflow.job_run_timeline r
JOIN system.lakeflow.jobs j
ON r.workspace_id = j.workspace_id AND r.job_id = j.job_id
WHERE r.period_start_time >= current_date() - INTERVAL 7 DAYS
GROUP BY j.name
ORDER BY failed_runs DESC;Key points
- The Runs tab matrix shows each task's status across recent runs
- Open a failed task to see the notebook output and the error message
- The Runs page in the sidebar shows runs across all jobs and pipelines
- system.lakeflow tables keep run history for SQL analysis and dashboards
- A long run can appear in several timeline rows, so count DISTINCT run_id
- Join with system.billing.usage on job_id to see cost per job