Feature Engineering in Unity Catalog

Databricks · Machine Learning & AI

Theory

A feature is an input column for a model, such as a customer's number of orders in the last 90 days. On Databricks, any Unity Catalog Delta table with a primary key can act as a feature table, and the Feature Engineering client (databricks-feature-engineering) adds tooling on top. You compute features with Spark or SQL, save them with create_table or write_table, then build a training set with FeatureLookup, which joins features to your labels by key (and by timestamp for point-in-time correctness, so training never sees future data). Models logged with fe.log_model remember which features they need, so batch scoring and serving look them up automatically. Online feature stores serve features at low latency for real-time endpoints.

Example: customer_features holds orders_90d and spend_90d per customer_id, computed nightly from main.sales.orders and reused by the churn and upsell models.

Why we use it

Teams often recompute the same features differently for training and production, causing silent errors (training/serving skew). Central feature tables give one definition, reuse across models, lineage, and automatic lookups at inference.

When to use it

Use feature tables when several models share features or when online serving needs the same features as training. For a one-off experiment, a plain DataFrame is fine. Add a timestamp key when features change over time and labels are historical.

How it works

  1. Raw data in main.sales.orders
  2. Compute features per key with Spark or SQL
  3. Save as feature table main.sales.customer_features (primary key customer_id)
  4. Build a training set with FeatureLookup on the labels
  5. Need real-time predictions? Yes: Publish features to an online store for serving. No: Use batch scoring with automatic lookups.
  6. Train, log with fe.log_model, register in Unity Catalog

Syntax

from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
fe = FeatureEngineeringClient()
fe.create_table(name="<cat.sch.table>", primary_keys=["id"], df=df, description="...")
fe.write_table(name=..., df=df, mode="merge")
ts = fe.create_training_set(df=labels, feature_lookups=[FeatureLookup(table_name=..., lookup_key="id")], label="y")
fe.log_model(model=m, name="model", flavor=mlflow.sklearn, training_set=ts, registered_model_name=...)
UI: Features (sidebar) / Catalog

Example

from pyspark.sql import functions as F
from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup

fe = FeatureEngineeringClient()

features = (spark.table("main.sales.orders")
  .where(F.col("order_date") >= F.date_sub(F.current_date(), 90))
  .groupBy("customer_id")
  .agg(F.count("*").alias("orders_90d"), F.sum("amount").alias("spend_90d")))

fe.create_table(name="main.sales.customer_features",
                primary_keys=["customer_id"], df=features,
                description="90-day order count and spend per customer")

labels = spark.table("main.sales.churn_labels")   # customer_id, churned
training_set = fe.create_training_set(
    df=labels,
    feature_lookups=[FeatureLookup(table_name="main.sales.customer_features",
                                   lookup_key="customer_id")],
    label="churned")
train_df = training_set.load_df()

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic