Skip to content

multimodal model

A multimodal model is a machine learning model that takes in, and sometimes produces, more than one kind of data, such as text, images, audio, and video, within a single request. Models that pair images with text are often called vision-language models (VLMs).

Most multimodal models are built around a large language model. An encoder for each extra modality, or input type, turns raw input into vectors in the same embedding space as text tokens. The language model then reads image or audio tokens and words as one sequence:

Text, image, and audio pass through a tokenizer or encoders into one embedding sequence that a large language model reads to produce text.
Many Input Types, One Token Sequence

LLaVA, released in 2023, showed the simplest form of this design. It passes image features from a pretrained CLIP vision transformer through a single trainable projection into the language model’s word-embedding space. Google’s Gemini models were instead trained on interleaved text, images, audio, and video from the start.

In coding work, these models read screenshots of a user interface, error dialogs, architecture diagrams, and scanned documents. Images aren’t free, though. Each one is split into patches that count against the context window like any other tokens, and a full-screen screenshot can cost more than a thousand of them.

Small text, rotated images, counting, and precise spatial positions remain weak spots, and a model can hallucinate details that the image doesn’t contain.

Python MarkItDown: Convert Documents Into LLM-Ready Markdown

Tutorial

Python MarkItDown: Convert Documents Into LLM-Ready Markdown

Get started with Python MarkItDown to turn PDFs, Office files, images, and URLs into clean, LLM-ready Markdown in seconds.

intermediate ai tools

For additional information on related topics, take a look at the following resources:

Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.


By Martin Breuss • Updated Nov. 7, 2026