Model Serving
Databricks · Machine Learning & AI
Theory
Model Serving (often called Mosaic AI Model Serving in older material) deploys models as REST API endpoints on serverless compute that scales up with traffic and, optionally, down to zero when idle. You can serve your own custom models from Unity Catalog (any MLflow model, such as the churn classifier), foundation models hosted by Databricks through the Foundation Model APIs, and external models such as OpenAI or Anthropic through governed endpoints. One endpoint can serve several model versions with a traffic split for A/B tests. Requests can be logged to Unity Catalog inference tables for monitoring, and you can call an endpoint from apps over REST or from SQL with ai_query.
Example: Ravi creates endpoint churn-scorer serving main.sales.churn_model version 3. The CRM app posts a customer's recent order count and spend and gets a churn probability back in milliseconds.
Why we use it
Batch scoring is too slow when an app needs a prediction during a request. Model Serving turns a registered model into a secure, autoscaling API without you building containers, load balancers or GPU clusters.
When to use it
Use Model Serving for real-time or low-latency predictions and for LLM endpoints used by apps and agents. Use batch inference (spark_udf or ai_query over a table in a job) for scoring millions of rows on a schedule, which is cheaper than calling an endpoint row by row.
How it works
- Model registered in Unity Catalog
- Serving → Create serving endpoint, choose model and version
- Pick compute size and whether to scale to zero
- Endpoint becomes Ready
- Called by an app in real time? Yes: POST to /serving-endpoints/<name>/invocations. No: Use ai_query or batch scoring in a job.
- Monitor latency, errors and inference tables
Syntax
UI: Serving → Create serving endpoint → Name → Served entities: Entity (My models- Unity Catalog), Version, traffic %, compute size, Scale to zero → Create
REST: POST https://<workspace>/serving-endpoints/<name>/invocations {"dataframe_records": [...]}
SQL: ai_query('<endpoint>', <request>)Example
# Query the endpoint from Python with the Databricks SDK
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
resp = w.serving_endpoints.query(
name="churn-scorer",
dataframe_records=[{"orders_90d": 2, "spend_90d": 1800.0, "days_since_last": 41}])
print(resp.predictions)
# Or with plain REST
# curl -X POST https://<workspace>/serving-endpoints/churn-scorer/invocations \
# -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
# -d '{"dataframe_records": [{"orders_90d": 2, "spend_90d": 1800.0, "days_since_last": 41}]}'
-- Or from SQL over a table
SELECT customer_id,
ai_query('churn-scorer', named_struct('orders_90d', orders_90d,
'spend_90d', spend_90d, 'days_since_last', days_since_last)) AS churn_score
FROM main.sales.customer_features;Key points
- Endpoints are serverless and autoscale; scale to zero saves cost but adds cold starts
- Serve custom UC models, Databricks-hosted foundation models or external models
- Traffic split across versions supports A/B tests and safe rollouts
- Inference tables log requests and responses to Unity Catalog for monitoring
- Endpoint permissions: CAN QUERY, CAN VIEW, CAN MANAGE
- For large scheduled scoring, batch inference is cheaper than many REST calls