GPT-4V and Vision-Language Models

GPT-4V (GPT-4 with Vision) is a large multimodal model that extends the transformer-based language model paradigm to accept both image and text as input, generating text responses grounded in visual content. Vision-language models (VLMs) more broadly represent the class of architectures that learn joint representations over image and text modalities, enabling tasks like visual question answering, image captioning, and grounded reasoning. A counterintuitive fact: despite accepting raw pixels, GPT-4V does not train a vision encoder from scratch — it leverages a pretrained visual backbone (closely related to CLIP's ViT) whose weights are largely frozen, meaning the "vision understanding" is inherited rather than jointly learned end-to-end.

The core insight

The fundamental insight of modern VLMs is that vision and language can be unified by projecting image patches into the same embedding space as text tokens — once an image is "tokenized" into a sequence of visual embeddings, a standard autoregressive transformer can treat it like any other sequence of inputs. The "aha moment" is recognizing that the attention mechanism is modality-agnostic: it only cares about relationships between vectors in a shared space, not whether those vectors came from pixels or words. This means that a sufficiently powerful language model needs only a learned "bridge" (a projection layer or cross-attention module) to acquire multimodal reasoning capabilities. The hard problem is not the architecture but the alignment: teaching the model that the visual token for a red apple and the text token sequence "red apple" refer to the same concept.

Common trapCommon Misconception

The most common mistake practitioners make is assuming GPT-4V and similar VLMs perform pixel-level understanding by processing raw image arrays through the language model directly. In reality, the image is first encoded by a pretrained vision encoder (typically a Vision Transformer, or ViT) into a compact sequence of patch embeddings, and only those embeddings are passed to the language model — the LLM never sees raw pixels. The consequence of this misunderstanding is that practitioners expect fine-grained spatial reasoning (e.g., "what is at pixel coordinate 320, 240?") to work reliably, when in fact the bottleneck is the patch-level tokenization: a 224×224 image divided into 16×16 patches yields only 196 visual tokens, coarsely discretizing spatial information. The correct mental model is that the vision encoder is a lossy compression step, and the language model reasons over that compressed representation — spatial precision is fundamentally limited by patch resolution.

How It Works

Visual Tokenization via Vision Transformer

A Vision Transformer (ViT) splits an input image $I \in \mathbb{R}^{H \times W \times C}$ into a grid of $N$ non-overlapping patches, each of size $P \times P$ . Each patch is linearly projected to a $d$ -dimensional embedding.

\quad N = \frac{H \cdot W}{P^2}, \qquad \mathbf{z}_i = \mathbf{W}_p \cdot \text{vec}(\text{patch}_i) + \mathbf{b}_p

Explanation: The image is divided into N patches, and each flattened patch vector is linearly mapped to a d-dimensional visual token $\mathbf{z}_i$ via projection matrix $\mathbf{W}_p \in \mathbb{R}^{d \times (P^2 C)}$ . Here $H, W$ are image height and width, $C$ is the number of channels (3 for RGB), $P$ is the patch size (commonly 16 or 32), and $\text{vec}(\cdot)$ flattens the patch to a 1D vector of length $P^2 C$ .

Cross-Modal Projection (the "Bridge")

After the ViT encodes patches to visual embeddings $\mathbf{Z}_v \in \mathbb{R}^{N \times d_v}$ , a learned projection maps them into the language model's embedding space $\mathbb{R}^{d_{\text{lm}}}$ .

\quad \hat{\mathbf{Z}}_v = \mathbf{W}_{\text{proj}} \mathbf{Z}_v + \mathbf{b}_{\text{proj}}, \qquad \mathbf{W}_{\text{proj}} \in \mathbb{R}^{d_{\text{lm}} \times d_v}

Explanation: This linear projection bridges the vision encoder's output dimension $d_v$ and the language model's token dimension $d_{\text{lm}}$ , making visual tokens dimensionally compatible with text tokens. In LLaVA-style architectures $d_v = 1024$ (ViT-L) and $d_{\text{lm}} = 4096$ (LLaMA-class), so $\mathbf{W}_{\text{proj}} \in \mathbb{R}^{4096 \times 1024}$ .

Multimodal Input Sequence Construction

Visual tokens $\hat{\mathbf{Z}}_v$ and text tokens $\mathbf{T} \in \mathbb{R}^{L \times d_{\text{lm}}}$ are concatenated to form the full input sequence fed to the decoder.

\quad \mathbf{S} = [\hat{\mathbf{Z}}_v \; \| \; \mathbf{T}] \in \mathbb{R}^{(N + L) \times d_{\text{lm}}}

Explanation: The model processes a single unified sequence of length N+L, where the first N positions are visual and the remaining L positions are textual — causal attention then allows text positions to attend to all visual tokens. The language model's causal attention mask ensures text tokens can attend to all prior tokens (including image tokens), while the image tokens can only attend to each other, preserving the autoregressive generation structure.

CLIP Contrastive Pretraining Objective

The vision encoder used in most VLMs (including GPT-4V's lineage) is initialized from CLIP, trained with the InfoNCE contrastive loss over a batch of $B$ image-text pairs.

\quad \mathcal{L}_{\text{CLIP}} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(\mathbf{v}_i^\top \mathbf{t}_i / \tau)}{\sum_{j=1}^{B} \exp(\mathbf{v}_i^\top \mathbf{t}_j / \tau)}

Explanation: This loss maximizes the cosine similarity between the image embedding $\mathbf{v}_i$ and its paired text embedding $\mathbf{t}_i$ relative to all other text embeddings in the batch, with temperature $\tau$ controlling the sharpness of the distribution. Here $\mathbf{v}_i, \mathbf{t}_i \in \mathbb{R}^d$ are L2-normalized image and text embeddings, $\tau$ is a learned temperature scalar (initialized around 0.07), and the symmetric version also sums the text-to-image direction.

Autoregressive Generation Loss

Given multimodal sequence $\mathbf{S}$ , the language model is trained to predict text tokens autoregressively, with loss computed only over text positions (not image tokens).

\quad \mathcal{L}_{\text{LM}} = -\sum_{t=N+1}^{N+L} \log P_\theta(s_t \mid s_1, s_2, \ldots, s_{t-1})

Explanation: The cross-entropy loss is summed only over text token positions (indices $N+1$ through $N+L$ ), so the model learns to generate contextually appropriate text conditioned on the visual prefix. $\theta$ denotes all trainable parameters of the language model (and projection layer), $s_t$ is the ground-truth token at position $t$ , and conditioning includes all preceding visual and text tokens. --- ```python import torch import torch.nn as nn from transformers import CLIPVisionModel, AutoModelForCausalLM class SimpleLLaVABridge(nn.Module): """Minimal VLM bridge: ViT encoder → linear projection → LLM token space.""" def __init__(self, vision_dim: int = 1024, lm_dim: int = 4096): super().__init__()

# Learnable projection: only this is trained in stage-1 of LLaVA self.proj = nn.Linear(vision_dim, lm_dim) self.vision_encoder = CLIPVisionModel.from_pretrained( "openai/clip-vit-large-patch14" )

# Freeze vision encoder during projection alignment phase for param in self.vision_encoder.parameters(): param.requires_grad = False def encode_image(self, pixel_values: torch.Tensor) -> torch.Tensor: """ Args: pixel_values: (B, 3, 224, 224) normalized image tensor Returns: visual_tokens: (B, N, lm_dim) where N = 256 for patch14 """ with torch.no_grad():

# (B, N+1, vision_dim) — includes [CLS] token at index 0 vision_out = self.vision_encoder( pixel_values=pixel_values, output_hidden_states=True, ).last_hidden_state

# Drop [CLS] token, keep patch tokens: (B, N, vision_dim) patch_tokens = vision_out[:, 1:, :]

# Project to LLM space: (B, N, lm_dim) return self.proj(patch_tokens) def build_multimodal_sequence( self, visual_tokens: torch.Tensor, # (B, N, lm_dim) text_embeds: torch.Tensor, # (B, L, lm_dim) ) -> torch.Tensor: """Concatenate [IMG_TOKENS | TEXT_TOKENS] along sequence dim.""" return torch.cat([visual_tokens, text_embeds], dim=1) # (B, N+L, lm_dim) ```

researcher · Alec Radford | b. 1990 | Architect of CLIP and GPT series

Radford led the OpenAI team that developed both the GPT series and CLIP (Contrastive Language-Image Pretraining), the vision encoder that underpins virtually all modern VLMs including GPT-4V. His 2021 CLIP paper demonstrated that training on 400 million noisy image-text pairs from the internet could produce transferable visual representations that rival supervised ImageNet pretraining — a result that was widely considered implausibly strong before publication. Radford is notably private for someone whose work shapes daily AI usage at scale; he rarely appears at conferences and has given almost no public interviews.

researcher · Haotian Liu | b. 1996 | Creator of LLaVA

Liu, a PhD student at the University of Wisconsin-Madison (later Microsoft Research), developed LLaVA (Large Language and Vision Assistant) in 2023, the open-source framework that made VLM research accessible to the broader community by showing that a simple linear projection between CLIP and LLaMA suffices for strong multimodal performance. His insight that instruction-tuned multimodal data generated by GPT-4 (text-only) could be used to fine-tune open VLMs — a form of "language-bootstrapped" visual instruction tuning — dramatically lowered the data barrier for VLM research. LLaVA-1.5 achieved near GPT-4V-level performance on standard benchmarks using only 558K training samples, a 100× reduction from prior works.

Figure

VLM Architecture: Image → Tokens → Language Model

Figure

CLIP Contrastive Training: Image–Text Alignment Matrix