Monitoring Job Runs

Databricks · Lakeflow Jobs

Theory

Databricks gives you several places to watch jobs. A job's Runs tab shows recent runs as a matrix of tasks by run, coloured by status, with duration and start time, so slow or failing tasks stand out. Clicking a run opens its task graph and each task's output, logs and error message. The Runs page in the sidebar lists job and pipeline runs across the workspace with filters for status and time. For trends and reporting, the system tables in system.lakeflow (jobs, job_tasks, job_run_timeline, job_task_run_timeline) hold run history you can query with SQL or put on a dashboard.

Example: Meera queries system.lakeflow.job_run_timeline for the last 7 days, counts runs and failures per job, and finds orders_daily failed 1 of 3 runs, which she pins to an ops dashboard.

Why we use it

Without monitoring, failures and slowdowns are found by angry dashboard users. Run views show what broke and why, and system tables let you spot trends such as rising duration or repeated failures before they cause missed SLAs.

When to use it

Use the job Runs tab and run details for debugging one job. Use the Runs page for a workspace-wide view. Use system.lakeflow tables for SLAs, failure trends and cost joins with system.billing. Set notifications so you do not need to watch the UI all day.

How it works

  1. Something looks wrong (alert, late data)
  2. Open the job's Runs tab and find the red or slow run
  3. Click the run → open the failed task → read output and error
  4. Is it a one-off or a trend? Yes: Query system.lakeflow tables over several days. No: Fix and Repair run.
  5. Add a dashboard or alert so it is caught earlier next time

Syntax

UI: Jobs & Pipelines → <job> → Runs tab (matrix view)  |  Sidebar → Runs
Run page: task graph → task → Output / Logs / Error
SQL: system.lakeflow.jobs, system.lakeflow.job_run_timeline,
     system.lakeflow.job_task_run_timeline (columns include job_id, run_id, period_start_time, result_state)

Example

-- Runs and failures per job in the last 7 days
SELECT j.name AS job_name,
       COUNT(DISTINCT r.run_id) AS runs,
       COUNT(DISTINCT CASE WHEN r.result_state = 'FAILED' THEN r.run_id END) AS failed_runs
FROM system.lakeflow.job_run_timeline r
JOIN system.lakeflow.jobs j
  ON r.workspace_id = j.workspace_id AND r.job_id = j.job_id
WHERE r.period_start_time >= current_date() - INTERVAL 7 DAYS
GROUP BY j.name
ORDER BY failed_runs DESC;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic