Vision-Language and Multimodal Models

A vision-language model (VLM) is any architecture that consumes both visual and textual input and produces an output — a similarity score, a caption, an answer, a bounding box — that depends jointly on the content of both modalities. The foundational design choice is how the two modalities communicate. In a dual-encoder (bi-encoder) architecture, an image encoder $f_\theta$ and a text encoder $g_\phi$ each independently map their input to a vector in $\mathbb{R}^d$, and the two towers only interact through a similarity function, almost always cosine similarity, computed after encoding. This is the CLIP design: expensive to make expressive (each modality must be reduced to a single vector that captures everything relevant), but extremely cheap at inference time because the entire image (or text) corpus can be pre-encoded and searched with a nearest-neighbor index. In a fusion architecture, the two modalities interact during encoding, typically through cross-attention layers in which text tokens attend to image patch tokens (or vice versa) at multiple depths of the network. Fusion models are strictly more expressive — they can represent fine-grained interactions like "the cat is to the left of the dog" that a single pooled vector struggles to capture — but they cannot be pre-computed independently, so they are unsuitable for large-scale retrieval and are instead used for tasks that require deep, compositional reasoning: captioning, VQA, visual grounding, and multimodal dialogue. The two paradigms are not mutually exclusive. Modern foundation systems (BLIP-2, Flamingo, LLaVA) use a dual-encoder-style vision backbone (frequently a frozen CLIP image encoder) to extract visual features cheaply, then attach a small trainable fusion module — a Q-Former, a perceiver resampler, or a simple linear projection — that translates these features into a form a pretrained autoregressive language model can consume via cross-attention or by concatenation into its input sequence. This design pattern, "frozen vision encoder + frozen LLM + lightweight trainable bridge," is the dominant recipe for building multimodal LLMs cheaply, because it reuses two enormously expensive pretraining runs (vision-language contrastive pretraining and language model pretraining) and trains only the small connective module from scratch.

Step 1 — Tokenize both modalities. Text is split into subword tokens via BPE or SentencePiece, exactly as in a standard LLM. Images are split into a grid of non-overlapping patches (e.g., $14 \times 14$ pixels), each linearly projected into a $d$-dimensional vector plus a learned positional embedding — the Vision Transformer recipe already established as a prerequisite for CLIP.

Step 2 — Encode. A Transformer encoder (ViT) processes the patch sequence with self-attention, producing a sequence of contextualized patch embeddings; a Transformer encoder or decoder processes the text tokens. For dual-encoder training, a special [CLS]/[EOS] token's final hidden state (or a mean-pooled representation) is projected to the shared embedding dimension and L2-normalized.

Step 3 — Align via contrastive loss (dual-encoder training). Given a batch of $N$ matched (image, text) pairs, compute the $N \times N$ matrix of cosine similarities scaled by a learned temperature $\tau$, and apply the symmetric InfoNCE loss so that each image's true caption scores highest among all $N$ candidate captions in the batch, and vice versa.

Step 4 — Bridge to a language model (fusion/generation training). Take the sequence of patch embeddings from a (usually frozen) vision encoder and pass them through a small trainable module — a handful of cross-attention or self-attention layers with a fixed number of learnable query tokens (a Q-Former/Perceiver Resampler) — that compresses potentially hundreds of patch tokens down to, say, 32 or 64 "visual tokens." These visual tokens are then either (a) prepended to the text token sequence and fed into a frozen decoder-only LLM, or (b) fed as keys/values into interleaved cross-attention layers inserted into an otherwise frozen LLM (the Flamingo design).

Step 5 — Train the language model head autoregressively. For captioning, VQA, and instruction-following, the model is trained with the standard next-token cross-entropy loss on the text portion of the sequence (visual tokens are never predicted, only conditioned upon):

where $\mathbf{v}_{1:M}$ are the $M$ visual tokens and $w_{1:T}$ are the target text tokens (a caption, or "question + answer").

Step 6 — Inference. For retrieval tasks, encode the query once and compare against a pre-built index of embeddings using approximate nearest neighbor search. For generative tasks, decode autoregressively (greedy, beam search, or nucleus sampling), re-attending to the fixed visual tokens at every step. ```pseudocode

# ---------- PHASE A: Dual-encoder contrastive pretraining (CLIP-style) ---------- function TRAIN_CONTRASTIVE(images, texts, batch_size N, temperature tau): initialize image_encoder f_theta, text_encoder g_phi initialize projection heads W_img, W_txt (map encoder outputs to shared dim d) repeat until converged: sample N matched pairs {(I_1,T_1), ..., (I_N,T_N)} for i in 1..N: v_i = normalize( W_img f_theta(I_i) ) # image embedding, ||v_i||=1 u_i = normalize( W_txt g_phi(T_i) ) # text embedding, ||u_i||=1

# similarity matrix, S[i][j] = cosine similarity of image i and text j for i in 1..N: for j in 1..N: S[i][j] = (v_i . u_j) / tau

# symmetric InfoNCE: each row and each column should peak on the diagonal loss_i2t = mean_i( -log( exp(S[i][i]) / sum_j exp(S[i][j]) ) ) loss_t2i = mean_j( -log( exp(S[j][j]) / sum_i exp(S[i][j]) ) ) loss = (loss_i2t + loss_t2i) / 2 backprop(loss); update(theta, phi, W_img, W_txt, tau) return f_theta, g_phi, W_img, W_txt

# ---------- PHASE B: Bridging a frozen vision encoder to a frozen LLM ---------- function TRAIN_MULTIMODAL_LLM(f_theta_FROZEN, LLM_FROZEN, data): initialize bridge B_psi # e.g. Q-Former: K learnable query tokens + cross-attn layers for (image, prompt_text, target_text) in data: patch_feats = f_theta_FROZEN(image) # [num_patches, d_vis] visual_tokens = B_psi(patch_feats) # [K, d_llm], K << num_patches input_seq = concat(visual_tokens, tokenize(prompt_text)) logits = LLM_FROZEN(input_seq, condition_on=visual_tokens) loss = cross_entropy(logits[len(input_seq):], tokenize(target_text)) backprop(loss) # gradients flow ONLY into B_psi (and optionally light LoRA adapters) update(psi) return B_psi

# ---------- INFERENCE: captioning / VQA via autoregressive decoding ---------- function GENERATE(image, question_or_prompt, f_theta, B_psi, LLM, max_len): visual_tokens = B_psi( f_theta(image) ) seq = concat(visual_tokens, tokenize(question_or_prompt)) output_tokens = [] for t in 1..max_len: next_token = argmax( LLM(seq)[-1] ) # greedy; beam/nucleus sampling in practice if next_token == EOS: break output_tokens.append(next_token) seq = concat(seq, next_token) return detokenize(output_tokens) ```

Contrastive pretraining cost. For a batch of size $N$, forming the similarity matrix costs $O(N^2 d)$ for the dot products plus $O(N \cdot C_{img} + N \cdot C_{txt})$ for the two encoder forward passes, where $C_{img}$ and $C_{txt}$ are the per-example encoder costs (each itself $O(L^2 d)$ for a Transformer of sequence length $L$ due to self-attention). The quadratic-in-batch-size term is the key scaling fact governing CLIP-style training: larger batches give a harder, more informative contrastive task (more negatives per positive), which empirically improves representation quality, but every doubling of $N$ doubles the memory and compute of the similarity matrix. In practice $N$ is pushed into the tens of thousands using memory-efficient tricks (chunked/sharded similarity computation across devices, as in OpenCLIP's implementation), because the effective difficulty of the InfoNCE objective scales with the number of in-batch negatives $N-1$.

Why contrastive alignment is a correct objective. The InfoNCE loss is a lower bound on (a monotonic transform of) the mutual information between the image and text encoders' outputs. Formally, for the positive pair $(I_i, T_i)$ and $N-1$ sampled negatives, minimizing

is equivalent to maximizing a bound $I(V;U) \geq \log N - \mathcal{L}_{i2t}$. As $N \to \infty$ and the encoders are expressive enough, the global optimum of this objective assigns high similarity to (and only to) semantically matched pairs, because any encoder that violates this (e.g., encoding two visually and semantically different images identically) fails to distinguish the correct caption from an incorrect one for at least one example in an arbitrarily large batch, incurring loss. This gives a clean (if idealized) argument for why contrastive pretraining recovers a semantically meaningful shared space rather than an arbitrary alignment.

Fusion/generation cost. For an image producing $P$ patch tokens compressed to $K \ll P$ visual tokens by the bridge module, and a text sequence of length $T$, each decoder layer costs $O((K+T)^2 d)$ for self-attention (or $O(T^2 d + T K d)$ if visual tokens are only attended to via dedicated cross-attention layers, as in Flamingo, rather than concatenated into the self-attention sequence). Compressing $P$ (often 256–1024 patches for a high-resolution image) down to $K$ (32–64) is what makes attaching a vision encoder to an LLM computationally tractable: without compression, a single image would cost as much context as a multi-page document. Autoregressive decoding for a caption of length $T$ costs $O(T)$ sequential steps, each $O((K+T)d^2 + (K+T)^2 d)$, giving total generation cost $O(T^2 (K+T) d + T(K+T)^2d)$ in the naive case, reduced in practice by KV-caching to $O(T \cdot (K+T) \cdot d)$ amortized per new token.

Correctness of autoregressive captioning. The chain rule of probability guarantees that the factorization

is exact — no approximation is introduced by decomposing sequence generation token-by-token, so any error in the resulting captions is a modeling or optimization error (an insufficiently expressive attention pattern, or a train/test mismatch such as exposure bias under greedy decoding), not a flaw in the factorization itself. This is the same invariant that underlies text-only autoregressive language modeling; the multimodal extension only changes what is being conditioned on.

Tradeoffs, stated honestly. Dual encoders are asymptotically much cheaper for retrieval — $O(1)$ per query against a pre-built approximate nearest-neighbor index of size $M$ items, versus $O(M)$ full forward passes for a fusion model that must jointly encode every candidate image against the query — but they cannot represent fine compositional relationships between objects because all information is squeezed through a single bottleneck vector before comparison. Fusion/cross-attention models fix this at the cost of losing pre-computability, and instruction-tuned MLLMs built on frozen backbones inherit whatever blind spots (spatial reasoning, counting, fine-grained attribute binding) already existed in the frozen vision encoder, since the bridge module cannot invent visual information the backbone failed to encode.

Consider a small illustrative training batch of $N=4$ image-text pairs (imagine real photographs; described here for concreteness):

After encoding, suppose the (illustrative) cosine similarity matrix $S$ (rows = images, columns = texts, before temperature scaling) is:

The diagonal (bold) is highest in every row and every column — the intended behavior — but note that off-diagonal entries $S[1][4]=0.61$ and $S[4][1]=0.58$ are also elevated, because images 1 and 4 are semantically similar (both golden retrievers at a beach) even though they are not the exact matched pair for each other's caption. This is the expected and correct behavior of a well-trained contrastive space: it clusters by meaning, not by exact identity, so a retrieval query for "dog on the beach" would correctly surface both I1 and I4, ranked by fine-grained similarity to the query embedding. Applying softmax with temperature $\tau=0.1$ to row 1 (i.e., $S[1][j]/\tau$): the logits become $(8.2, 1.0, 0.5, 6.1)$, and softmax gives approximately $(0.87, 0.0003, 0.0002, 0.13)$ — the model assigns 87% of its probability mass to the true caption and correctly reserves a nontrivial 13% for the confusable near-duplicate image, rather than confusing it with the cat or bicycle captions. The InfoNCE loss for this row is $-\log(0.87) \approx 0.14$ nats, small because the true pair is confidently ranked first; the harder row-4 case (competing against the very similar I1) will produce a proportionally larger loss, which is exactly the training signal that sharpens the model's ability to distinguish fine-grained visual differences between the two beach photos. Now trace VQA on image I1 with the question "what is the dog doing?" The frozen ViT produces, say, $P=256$ patch tokens; the Q-Former compresses these to $K=32$ visual tokens, emphasizing regions with motion blur and paw contact with sand (learned via its cross-attention weights during training). These 32 tokens are prepended to the tokenized question. The frozen LLM decodes autoregressively: $P(\text{"running"} \mid \text{visual tokens}, \text{"what is the dog doing?"})$ dominates over alternatives like "sleeping" because the cross-attention pattern in the bridge module has already encoded sand-spray and a mid-stride leg posture into the visual tokens the LLM conditions on — the answer is only as good as the information that survived the $P \to K$ compression bottleneck.

Vision-language retrieval powers e-commerce visual search ("find this dress"), stock-photo and asset-management search, and multimodal content moderation pipelines that must classify images and captions jointly. Captioning and VQA systems underlie accessibility tools (screen readers describing images for blind users), automated alt-text generation at social platform scale, and radiology report drafting assistants that must be paired with human oversight given the hallucination risks above. Instruction-tuned multimodal LLMs are now embedded in consumer assistants, robotics perception stacks (grounding natural-language commands like "pick up the blue mug" in camera input), and document-understanding pipelines that read charts, forms, and screenshots. These same architectures — a vision backbone feeding a shared representation — are also the substrate for identity-sensitive applications such as face search and verification, where the alignment properties and biases of the embedding space carry direct consequences, the topic taken up in Face Recognition and Ethics. The underlying representation-learning machinery connects tightly to Text-to-Image with CLIP and Latent Diffusion (the CLIP encoder is frequently reused verbatim as the vision tower) and, for 3D-aware multimodal grounding, to scene representations built on Neural Radiance Fields.

Ask the tutor: How would you redesign the bridge module and training objective to reduce hallucination while keeping inference cost low?