Common Anti-Patterns

Databricks · Architecture, Performance & Cost

Theory

Many slow or expensive Databricks setups share the same mistakes. Calling collect() or toPandas() on large DataFrames pulls all data to the driver and crashes it. Row-by-row Python UDFs where built-in functions exist block Photon and slow everything. Over-partitioning by high-cardinality columns such as customer_id creates millions of tiny files. Leaving all-purpose clusters running for scheduled jobs costs more than job or serverless compute. Writing small files constantly without compaction, skipping filters on clustered columns, using SELECT * on wide tables in dashboards, and full reloads where incremental loads would work all waste time and money. Hard-coded credentials in notebooks and working in the legacy hive_metastore instead of Unity Catalog are governance anti-patterns.

Example: a job running df.toPandas() on 200 million orders fails with driver out-of-memory; rewriting it as a Spark aggregation fixes it in minutes.

Why we use it

Interviewers and real projects both test whether you can spot these. Avoiding them is usually the cheapest performance and cost win, far more than buying bigger compute.

When to use it

Review this list when a job is slow, fails with memory errors, or costs more than expected, and during code reviews. Check the query profile or Spark UI first to confirm which problem you actually have before changing code.

How it works

  1. Job is slow, failing or costly
  2. Open the query profile or Spark UI
  3. Driver out of memory? Yes: Remove collect()/toPandas() on big data. No: Next check.
  4. Many tiny files or a skewed stage? Yes: Compact, cluster, fix skew. No: Check for UDFs and full reloads.
  5. Apply the fix and rerun
  6. Compare duration and DBUs before vs after

Syntax

Prefer: built-in functions over UDFs | spark aggregations over collect()/toPandas()
        liquid clustering over over-partitioning | Auto Loader/streaming tables over full reloads
        job/serverless compute for jobs | secrets: dbutils.secrets.get(scope, key)
Diagnose: Query profile, Spark UI (skew, spill, shuffle), system tables

Example

# Anti-pattern: pull everything to the driver
pdf = spark.table("main.sales.orders").toPandas()          # driver OOM on big data
totals = pdf.groupby("region")["amount"].sum()

# Better: let Spark aggregate, bring back only the small result
totals = (spark.table("main.sales.orders")
            .groupBy("region").sum("amount")
            .toPandas())

# Anti-pattern: Python UDF for something built in
from pyspark.sql.functions import udf, upper, col
to_upper = udf(lambda s: s.upper() if s else None)
df = spark.table("main.sales.customers").withColumn("name_uc", to_upper("name"))
# Better
df = spark.table("main.sales.customers").withColumn("name_uc", upper(col("name")))

# Anti-pattern: password in code. Better:
pwd = dbutils.secrets.get(scope="erp", key="db_password")

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic