Multimodal Models
Gen AI · Multimodal & Applied AI
Theory
Multimodal models understand, and sometimes generate, more than one kind of content (text, images, audio or video) within a single model, instead of chaining separate specialised models. Inputs such as an image are converted into an internal representation the model can reason over together with text, so you can send a photo plus a question in one request.
Practical uses include reading scanned invoices and receipts into structured fields, answering questions about a chart or screenshot, describing images for accessibility and natural voice conversations. Accuracy depends on input quality: blurry photos, tiny text and dense tables cause errors, and numbers read from images should be checked. For data teams, a common pattern is image in, validated JSON out, then load the result into a table.
Why we use it
Lots of business data is locked in images, scans, screenshots and audio. Multimodal models let you query and extract that content directly, without building separate OCR, vision and speech pipelines.
When to use it
Use them for documents, receipts, charts, screenshots, photos and voice. If the content is already clean text or a table, send it as text instead: it is cheaper and more reliable than an image.
How it works
- User sends an image plus a question
- Image is converted into a representation the model can read
- Text question is tokenised
- Model reasons over image and text together
- Are there fields to load? Yes: Validate them before saving. No: Show the answer to the user.
- Answer or structured data returned
Example
Input: [photo of a restaurant receipt] + "How much did I tip, and was it more than 18%?"
Model output: "Your tip was $8.50 on a $42.00 bill, which is about 20.2% — yes, that’s more than 18%."
(A text-only model could never do this — it needs to actually read the image.)Key points
- One model accepts mixed inputs such as image plus text
- Used for document and receipt understanding, visual Q&A and voice assistants
- Ask for structured JSON output when you plan to load results into tables
- Blurry images, tiny text and dense tables reduce accuracy
- Send clean text as text: images cost more and add error risk