AI Search & Agents

Databricks · Machine Learning & AI

Theory

AI Search (renamed from Vector Search in 2026) stores embeddings of your text so you can find content by meaning, not exact keywords. You create an endpoint, then an index on a Delta table: a Delta Sync index keeps itself up to date as the source table changes and can compute embeddings for you with an embedding model endpoint. Queries return the most similar rows from Python, REST or SQL (vector_search()). This is the retrieval part of RAG (retrieval-augmented generation): an LLM answers using the retrieved passages. Agent Bricks is Databricks' platform for building, deploying and governing such agents in code, with MLflow Tracing and evaluation; for low-code agents Databricks now points to Genie Agents (Knowledge Assistant is legacy).

Example: a support agent searches an index of product FAQs and answers "How long is delivery to Jaipur?" citing the matching FAQ rows.

Why we use it

LLMs do not know your company's documents and can make things up. Retrieval from a governed index grounds answers in your own data, and an agent platform adds evaluation, tracing and governed deployment so the result is reliable enough for users.

When to use it

Use AI Search when users search or ask questions over large text collections (docs, tickets, product descriptions). Use Genie Agents for low-code Q&A, especially over structured tables, and code-first Agent Bricks agents for complex, multi-step workflows.

How it works

  1. Documents or FAQ rows in a Delta table
  2. Chunk text and create a Delta Sync index on an AI Search endpoint
  3. User asks a question; embed it and search the index
  4. Pass the top matches plus the question to an LLM
  5. Is the answer grounded and correct in evaluation? Yes: Deploy the agent (Model Serving / Agent Bricks). No: Improve chunks, prompts or retrieval and re-evaluate.
  6. Trace and monitor real conversations

Syntax

Python: from databricks.vector_search.client import VectorSearchClient
  vsc.create_delta_sync_index(endpoint_name, index_name, source_table_name, primary_key,
      pipeline_type="TRIGGERED", embedding_source_column, embedding_model_endpoint_name)
  index.similarity_search(query_text="...", columns=[...], num_results=5)
SQL: SELECT * FROM vector_search(index => '<cat.sch.index>', query_text => '...', num_results => 5)
UI: Catalog → source table → Create → AI Search index;  Agents (sidebar) for Agent Bricks

Example

from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()
index = vsc.create_delta_sync_index(
    endpoint_name="techtown-search",
    index_name="main.sales.faq_index",
    source_table_name="main.sales.faq",          # faq_id, question, answer (CDF enabled)
    primary_key="faq_id",
    pipeline_type="TRIGGERED",
    embedding_source_column="answer",
    embedding_model_endpoint_name="databricks-gte-large-en")

hits = index.similarity_search(query_text="delivery time to Jaipur",
                               columns=["faq_id", "question", "answer"], num_results=3)

-- Same search from SQL
SELECT faq_id, question
FROM vector_search(index => 'main.sales.faq_index',
                   query_text => 'delivery time to Jaipur', num_results => 3);

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic