Feature Engineering in Unity Catalog
Databricks · Machine Learning & AI
Theory
A feature is an input column for a model, such as a customer's number of orders in the last 90 days. On Databricks, any Unity Catalog Delta table with a primary key can act as a feature table, and the Feature Engineering client (databricks-feature-engineering) adds tooling on top. You compute features with Spark or SQL, save them with create_table or write_table, then build a training set with FeatureLookup, which joins features to your labels by key (and by timestamp for point-in-time correctness, so training never sees future data). Models logged with fe.log_model remember which features they need, so batch scoring and serving look them up automatically. Online feature stores serve features at low latency for real-time endpoints.
Example: customer_features holds orders_90d and spend_90d per customer_id, computed nightly from main.sales.orders and reused by the churn and upsell models.
Why we use it
Teams often recompute the same features differently for training and production, causing silent errors (training/serving skew). Central feature tables give one definition, reuse across models, lineage, and automatic lookups at inference.
When to use it
Use feature tables when several models share features or when online serving needs the same features as training. For a one-off experiment, a plain DataFrame is fine. Add a timestamp key when features change over time and labels are historical.
How it works
- Raw data in main.sales.orders
- Compute features per key with Spark or SQL
- Save as feature table main.sales.customer_features (primary key customer_id)
- Build a training set with FeatureLookup on the labels
- Need real-time predictions? Yes: Publish features to an online store for serving. No: Use batch scoring with automatic lookups.
- Train, log with fe.log_model, register in Unity Catalog
Syntax
from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
fe = FeatureEngineeringClient()
fe.create_table(name="<cat.sch.table>", primary_keys=["id"], df=df, description="...")
fe.write_table(name=..., df=df, mode="merge")
ts = fe.create_training_set(df=labels, feature_lookups=[FeatureLookup(table_name=..., lookup_key="id")], label="y")
fe.log_model(model=m, name="model", flavor=mlflow.sklearn, training_set=ts, registered_model_name=...)
UI: Features (sidebar) / CatalogExample
from pyspark.sql import functions as F
from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
fe = FeatureEngineeringClient()
features = (spark.table("main.sales.orders")
.where(F.col("order_date") >= F.date_sub(F.current_date(), 90))
.groupBy("customer_id")
.agg(F.count("*").alias("orders_90d"), F.sum("amount").alias("spend_90d")))
fe.create_table(name="main.sales.customer_features",
primary_keys=["customer_id"], df=features,
description="90-day order count and spend per customer")
labels = spark.table("main.sales.churn_labels") # customer_id, churned
training_set = fe.create_training_set(
df=labels,
feature_lookups=[FeatureLookup(table_name="main.sales.customer_features",
lookup_key="customer_id")],
label="churned")
train_df = training_set.load_df()Key points
- Any UC Delta table with a primary key can be a feature table
- FeatureLookup joins features to labels by key, optionally point-in-time
- Models logged with fe.log_model look up their features automatically when scoring
- One feature definition for training and serving avoids training/serving skew
- Online feature stores serve features to real-time endpoints
- Schedule feature refreshes as a job or pipeline, like any other table