Common Anti-Patterns
Databricks · Architecture, Performance & Cost
Theory
Many slow or expensive Databricks setups share the same mistakes. Calling collect() or toPandas() on large DataFrames pulls all data to the driver and crashes it. Row-by-row Python UDFs where built-in functions exist block Photon and slow everything. Over-partitioning by high-cardinality columns such as customer_id creates millions of tiny files. Leaving all-purpose clusters running for scheduled jobs costs more than job or serverless compute. Writing small files constantly without compaction, skipping filters on clustered columns, using SELECT * on wide tables in dashboards, and full reloads where incremental loads would work all waste time and money. Hard-coded credentials in notebooks and working in the legacy hive_metastore instead of Unity Catalog are governance anti-patterns.
Example: a job running df.toPandas() on 200 million orders fails with driver out-of-memory; rewriting it as a Spark aggregation fixes it in minutes.
Why we use it
Interviewers and real projects both test whether you can spot these. Avoiding them is usually the cheapest performance and cost win, far more than buying bigger compute.
When to use it
Review this list when a job is slow, fails with memory errors, or costs more than expected, and during code reviews. Check the query profile or Spark UI first to confirm which problem you actually have before changing code.
How it works
- Job is slow, failing or costly
- Open the query profile or Spark UI
- Driver out of memory? Yes: Remove collect()/toPandas() on big data. No: Next check.
- Many tiny files or a skewed stage? Yes: Compact, cluster, fix skew. No: Check for UDFs and full reloads.
- Apply the fix and rerun
- Compare duration and DBUs before vs after
Syntax
Prefer: built-in functions over UDFs | spark aggregations over collect()/toPandas()
liquid clustering over over-partitioning | Auto Loader/streaming tables over full reloads
job/serverless compute for jobs | secrets: dbutils.secrets.get(scope, key)
Diagnose: Query profile, Spark UI (skew, spill, shuffle), system tablesExample
# Anti-pattern: pull everything to the driver
pdf = spark.table("main.sales.orders").toPandas() # driver OOM on big data
totals = pdf.groupby("region")["amount"].sum()
# Better: let Spark aggregate, bring back only the small result
totals = (spark.table("main.sales.orders")
.groupBy("region").sum("amount")
.toPandas())
# Anti-pattern: Python UDF for something built in
from pyspark.sql.functions import udf, upper, col
to_upper = udf(lambda s: s.upper() if s else None)
df = spark.table("main.sales.customers").withColumn("name_uc", to_upper("name"))
# Better
df = spark.table("main.sales.customers").withColumn("name_uc", upper(col("name")))
# Anti-pattern: password in code. Better:
pwd = dbutils.secrets.get(scope="erp", key="db_password")Key points
- Never collect() or toPandas() large data; aggregate in Spark first
- Use built-in functions; Python UDFs block Photon and add serialisation cost
- Do not partition by high-cardinality columns; use liquid clustering
- Run scheduled work on job or serverless compute, not all-purpose clusters
- Load incrementally (Auto Loader, streaming tables) instead of full reloads
- Keep credentials in secrets and data in Unity Catalog, not hive_metastore