Lakeflow Connect Managed Connectors
Databricks · Data Ingestion
Theory
Lakeflow Connect is the ingestion part of Lakeflow. Its managed connectors pull data from SaaS apps and databases (for example Salesforce, Workday, ServiceNow, Google Analytics, SharePoint, SQL Server, MySQL and PostgreSQL) into Unity Catalog streaming tables with a guided UI and no code. Under the hood each connector creates a Lakeflow pipeline that loads incrementally, using change data capture or cursor columns where the source supports it, on serverless compute. Credentials are stored as a Unity Catalog connection.
Example: the sales ops team wants Salesforce Accounts and Opportunities in Databricks. Click Data Ingestion → Salesforce, choose the connection, select the objects, set destination main.salesforce, add a daily schedule and click Save and run pipeline. Tables refresh incrementally from then on.
Why we use it
Building and maintaining API or CDC code for every business system is slow and fragile. Managed connectors handle authentication, incremental loading, schema changes and retries, and land the data governed in Unity Catalog.
When to use it
Use Lakeflow Connect when a managed connector exists for your source. For files in storage use Auto Loader or COPY INTO. For sources without a connector, use Lakehouse Federation to query in place, or write custom JDBC or API code in a job. Check the docs for which connectors are GA.
How it works
- Sidebar → Data Ingestion
- Pick the source under Databricks connectors (e.g. Salesforce)
- Does a connection already exist? Yes: Select it. No: Create connection (admin privilege needed).
- Name the pipeline, select objects and the destination catalog/schema
- Add a schedule and failure notifications
- Save and run pipeline: streaming tables appear in main.salesforce
Syntax
UI: Data Ingestion → Add data → Databricks connectors → <source>
Connection → Ingestion setup → Source → Destination → Schedules and notifications → Save and run pipeline
Result: a managed ingestion pipeline writing streaming tables in <catalog>.<schema>Example
-- After the Salesforce connector pipeline has run
SELECT Name, Industry, BillingCity
FROM main.salesforce.account
WHERE BillingCity IN ('Jaipur', 'Pune', 'Delhi');
-- Join SaaS data with lakehouse data
SELECT c.name, c.city, COUNT(o.order_id) AS orders
FROM main.sales.customers c
JOIN main.salesforce.account a ON a.Name = c.name
LEFT JOIN main.sales.orders o ON o.customer_id = c.customer_id
GROUP BY c.name, c.city;Key points
- Managed connectors write Unity Catalog streaming tables via a Lakeflow pipeline
- Source credentials live in a Unity Catalog connection object
- Loads are incremental after the first full snapshot
- Set the refresh schedule on the Schedules and notifications step
- Database connectors (SQL Server, MySQL...) need a gateway or CDC setup on the source
- Connector list and status change often, so check the docs before you promise one