Multimodal Models

Gen AI · Multimodal & Applied AI

Theory

Multimodal models understand, and sometimes generate, more than one kind of content (text, images, audio or video) within a single model, instead of chaining separate specialised models. Inputs such as an image are converted into an internal representation the model can reason over together with text, so you can send a photo plus a question in one request.

Practical uses include reading scanned invoices and receipts into structured fields, answering questions about a chart or screenshot, describing images for accessibility and natural voice conversations. Accuracy depends on input quality: blurry photos, tiny text and dense tables cause errors, and numbers read from images should be checked. For data teams, a common pattern is image in, validated JSON out, then load the result into a table.

Why we use it

Lots of business data is locked in images, scans, screenshots and audio. Multimodal models let you query and extract that content directly, without building separate OCR, vision and speech pipelines.

When to use it

Use them for documents, receipts, charts, screenshots, photos and voice. If the content is already clean text or a table, send it as text instead: it is cheaper and more reliable than an image.

How it works

  1. User sends an image plus a question
  2. Image is converted into a representation the model can read
  3. Text question is tokenised
  4. Model reasons over image and text together
  5. Are there fields to load? Yes: Validate them before saving. No: Show the answer to the user.
  6. Answer or structured data returned

Example

Input: [photo of a restaurant receipt] + "How much did I tip, and was it more than 18%?"

Model output: "Your tip was $8.50 on a $42.00 bill, which is about 20.2% — yes, that’s more than 18%."
(A text-only model could never do this — it needs to actually read the image.)

Key points

Gen AI topics

Foundations of Generative AI

Prompt Engineering

Retrieval & Embeddings

Customizing Model Behavior

AI Agents & Tool Use

Multimodal & Applied AI

Ethics, Cost & Careers

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Generative AI for the Data Stack

LLMs, prompt engineering, RAG, embeddings, agents and the tools shaping modern AI products — explained practically. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Section {{ sec.num }}

{{ sec.label }}

Gen AI  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

💡

{{ current.tldr }}

Explained

{{ para }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
See It In Action
{{ current.example }}
Key Takeaways — click to check off
{{ pt.mark }} {{ pt.text }}

Share this topic