Contrastive Learning

A contrastive objective trains an encoder $f_\theta$ to map inputs into a vector space such that positive pairs — two views of the same underlying instance — have high similarity, while negative pairs — views of different instances — have low similarity. Positive pairs are typically manufactured by applying two independent random augmentations (crop, color jitter, blur, rotation, masking) to the same raw input $x$, producing views $x_i$ and $x_j$. Every other item in the training batch (and its own augmented view) serves as a negative for $x_i$. Crucially, no class label is ever used: the "label" is simply which sample this augmented view came from, information the pipeline already knows for free. The similarity function is almost always cosine similarity in a normalized embedding space,

reusing the geometric intuition from cosine similarity: vectors that point in the same direction are "the same," regardless of magnitude. The similarities are converted into a probability distribution over "which candidate is the true positive" using a temperature-scaled softmax, and the loss is the negative log-likelihood of the correct match. This loss is called InfoNCE (Information Noise Contrastive Estimation), and it is the near-universal choice in modern contrastive frameworks (SimCLR, MoCo, CLIP, and their descendants). Formally, for an anchor $z_i = f_\theta(x_i)$ with positive $z_j$ and a set of negatives $\{z_k\}_{k \neq i}$ drawn from the rest of the batch,

where $N$ is the batch size (so $2N$ views total across the two augmented copies), and $\tau > 0$ is a temperature hyperparameter that controls how peaked the softmax distribution is. Small $\tau$ makes the loss punish near-miss negatives harshly (hard-negative sensitivity); large $\tau$ smooths the distribution and weakens the gradient signal. The total batch loss averages $\mathcal{L}_{i,j}$ over all $2N$ anchors, each paired with its own positive.

The training pipeline has four moving parts: an augmentation module, a shared encoder, a projection head, and the InfoNCE loss with an optimizer step. The projection head — a small MLP appended after the encoder — is a subtlety worth naming explicitly: it is used only during pretraining to compute the contrastive loss, and discarded afterward, because the raw encoder features transfer better to downstream tasks than the loss-optimized projection output.

Step-by-step mechanism: 1. Sample a minibatch of $N$ raw examples $\{x_1, \dots, x_N\}$. 2. For each $x_k$, draw two independent random augmentations $t, t'$ from an augmentation family $\mathcal{T}$, producing two views $\tilde x_{2k-1} = t(x_k)$ and $\tilde x_{2k} = t'(x_k)$. This yields $2N$ views total, and $(\tilde x_{2k-1}, \tilde x_{2k})$ is a positive pair. 3. Pass every view through the shared encoder $f_\theta$ to get representations $h = f_\theta(\tilde x)$, then through a projection head $g_\theta$ to get embeddings $z = g_\theta(h)$, then L2-normalize $z$. 4. Compute the $2N \times 2N$ pairwise cosine-similarity matrix over all embeddings. 5. For each view, treat its augmented partner as the single positive class and every other view (except itself) as negatives; compute the InfoNCE cross-entropy loss. 6. Average the loss over all $2N$ anchors, backpropagate, and update $\theta$ (encoder + projection head) with an optimizer (commonly SGD with LARS, or Adam). 7. Repeat for many epochs; at the end, discard the projection head and keep $f_\theta$ as the pretrained backbone, ready for Transfer Learning via linear probing or fine-tuning. ```

# Contrastive pretraining with the InfoNCE / NT-Xent loss (SimCLR-style) function TRAIN_CONTRASTIVE(dataset, encoder f, projector g, augment_family T, batch_size N, temperature tau, epochs): for epoch in 1..epochs: for batch {x_1, ..., x_N} sampled from dataset: views = [] # will hold 2N augmented views for k in 1..N: t1, t2 = sample two augmentations from T views.append(t1(x_k)) # view index 2k-1 views.append(t2(x_k)) # view index 2k

# Encode + project + normalize Z = [] for v in views: h = f(v) # backbone representation z = g(h) # projection head output z = z / norm(z) # L2-normalize -> unit hypersphere Z.append(z)

# Similarity matrix, 2N x 2N S = matrix of size (2N, 2N) for i in 1..2N: for j in 1..2N: S[i][j] = dot(Z[i], Z[j]) / tau

# Mask out self-similarity (i == j) with -infinity for i in 1..2N: S[i][i] = -infinity

# InfoNCE loss: each view's positive is its augmentation partner loss = 0 for i in 1..2N: j = partner_index(i) # the other view of the same original x_k numerator = exp(S[i][j]) denominator = sum(exp(S[i][k]) for k in 1..2N if k != i) loss += -log(numerator / denominator) loss = loss / (2 * N) gradients = backprop(loss, wrt = parameters of f and g) update parameters of f, g using gradients # e.g. SGD+LARS, Adam return f # discard g; keep encoder f as the pretrained backbone ``` The mask in the similarity matrix (`S[i][i] = -infinity`) is not a stylistic choice — without it, the softmax denominator would include the trivial self-similarity term $\exp(1/\tau)$, which is always the maximum possible similarity, and the loss would degenerate into a near-constant regardless of the actual positive pair. This is the single most common off-by-one-style bug in re-implementations of the loss.

Time complexity. For a batch of $N$ original samples ($2N$ views), computing the full pairwise similarity matrix costs $O(N^2 d)$, where $d$ is the embedding dimension, since every pair of the $2N$ vectors requires an $O(d)$ dot product and there are $O(N^2)$ pairs. Encoder forward/backward passes cost $O(N \cdot C)$ where $C$ is the per-sample cost of the backbone (e.g., a ResNet or Vision Transformer forward pass), which typically dominates for reasonable $N$ but the quadratic term becomes non-trivial as $N$ grows into the thousands, which is precisely why memory-bank methods like MoCo decouple negative count from batch size (see below). Each training step is thus

Space complexity. Storing the similarity matrix requires $O(N^2)$ memory (dominated by $N^2$, not $N^2 d$, since only scalar similarities are kept after the dot products); storing activations for backpropagation through the encoder costs $O(N \cdot A)$ where $A$ is the per-sample activation memory of the backbone. Because gradients must flow back through both views of every sample, effective memory use during contrastive pretraining is often cited as roughly double that of a supervised run at the same batch size — a genuine practical cost, not a rounding error, and the reason large-batch contrastive methods require substantial accelerator memory or gradient checkpointing.

Why more negatives help (and why it has diminishing returns). The number of negatives $2N-2$ determines how hard the underlying $2N$-way classification problem is: with only one negative, the network merely needs a crude decision boundary; with thousands of negatives spanning diverse concepts, the network is forced to build a representation with genuinely fine-grained discriminative structure. This is why SimCLR benefits from very large batches (thousands of images) and why MoCo instead maintains a queue of past embeddings — a size-$K$ FIFO memory bank refreshed by a momentum-updated encoder — to get many negatives ($K \gg N$) without paying the $O(N^2)$ compute and memory cost of huge batches. In MoCo, memory cost for the queue is $O(K d)$ (much cheaper than recomputing $K$ forward passes every step), and the update rule for the momentum encoder parameters $\theta_k$ given the query encoder $\theta_q$ is

which keeps the negative embeddings consistent across steps even though they were produced by slightly-stale encoder weights — the key trick that makes decoupling negatives from batch size stable.

Correctness argument — why minimizing InfoNCE learns good representations. InfoNCE is not an ad-hoc heuristic; it is a variational lower bound on the mutual information between two views of the same instance. Concretely, for a positive pair $(x, x^+)$ drawn from the joint distribution $p(x, x^+)$ of co-occurring views, and $N-1$ negatives drawn independently from the marginal $p(x^-)$, one can show

so that minimizing InfoNCE is equivalent to maximizing a lower bound on the mutual information between the two augmented views. Intuitively: if the encoder throws away information that distinguishes the true positive from the negatives, the softmax classifier in the loss cannot pick the correct match, and the loss stays high; the only way to drive the loss down is to retain and sharpen exactly the information that is (a) shared between the two views (hence robust to the augmentation) and (b) distinctive relative to other instances in the batch (hence semantically informative). This is also why the choice of augmentations matters so much: augmentations define which information is considered "nuisance" (color jitter throws away exact hue, so the network stops encoding hue) versus "signal" (object identity survives cropping, so the network is forced to encode it).

Honest tradeoffs. The bound $\log N - \mathcal{L}_{\text{InfoNCE}}$ is tighter — and the negative sampling task harder — as $N$ grows, but the bound is also provably loose for small $N$: with few negatives, the mutual-information estimate saturates and the representation quality plateaus below what larger negative pools would achieve, which is exactly the empirical batch-size sensitivity seen in SimCLR ablations. There is no free correctness guarantee that the learned invariances match the task's true invariances — if the augmentation family is mismatched to the downstream task (e.g., color jitter thrown away but the downstream task needs to distinguish colors), the representation will systematically underfit that task, a limitation baked into the method's very mechanism, not fixable by more compute alone.

Consider a toy batch of $N=2$ raw images: a cat photo $x_1$ and a car photo $x_2$. Each produces two augmented views, giving four embeddings after normalization (illustrative 2-D vectors for hand-tracing; real systems use $d = 128$–$2048$):

Take temperature $\tau = 0.5$ and compute the loss for anchor $\tilde x_1$, whose positive is $\tilde x_2$ and whose negatives are $\tilde x_3, \tilde x_4$.

Step 1 — cosine similarities from $\tilde x_1$:

Step 2 — divide by $\tau = 0.5$ (multiply by 2):

Step 3 — exponentiate:

Step 4 — InfoNCE loss for anchor $\tilde x_1$:

A loss of $0.302$ nats is close to the minimum achievable given these similarities (the true positive already dominates the denominator), meaning the encoder is doing well for this anchor. By contrast, at initialization (random weights), a typical similarity profile might be nearly flat, e.g. all three similarities around $0.3$–$0.4$ after scaling, giving a loss close to $-\log(1/3) = 1.099$ nats — the value for uniform (chance-level) 3-way classification. Training drives the loss from this uninformative starting point down toward the well-separated value above, which is the concrete, checkable signature that the representation is learning to place the true positive closer than the negatives. Repeating this computation symmetrically for anchor $\tilde x_2$ (whose positive is $\tilde x_1$) and averaging all four anchor losses gives the batch's total InfoNCE loss, which is what gets backpropagated.

Contrastive learning's mathematical roots lie in noise-contrastive estimation (NCE), a technique for fitting unnormalized probabilistic models by turning density estimation into binary classification against noise samples — the "positive vs. negative" logic that later became the core of representation learning. This idea was extended to representation learning through metric-learning losses (triplet loss, N-pair loss) before crystallizing into the InfoNCE formulation used in modern self-supervised vision and language models.

Contrastive pretraining is now a default step in production vision pipelines: encoders pretrained with SimCLR- or MoCo-style objectives on unlabeled web-scale image corpora are fine-tuned for classification, detection, and segmentation with far fewer labeled examples than training from scratch. CLIP extends the same InfoNCE machinery across modalities, aligning images and their captions in a shared space, which now underlies text-to-image retrieval, zero-shot image classification, and the vision-language backbones inside many generative image systems. Recommendation systems use contrastive objectives to learn user/item embeddings from implicit interaction signals (clicks, co-purchases) treated as "positive pairs" without explicit relevance labels. Audio and speech systems apply the same recipe to time-shifted or noise-perturbed audio segments to learn representations for downstream speech recognition. Because the method requires only a notion of "two views that should agree," it generalizes far beyond images — to graphs (contrasting subgraph views), video (contrasting temporally nearby frames), and molecules (contrasting augmented molecular graphs) — making it one of the most portable ideas in modern Self-Supervised Learning, and a natural complement to architectures built around Attention Mechanism-based encoders that benefit from large-scale unlabeled pretraining before task-specific fine-tuning.

Ask the tutor: How would the InfoNCE loss and training procedure need to change to align two different modalities, like images and text, instead of two augmented views of the same image?