Self-Supervised Vision

A pretext task is a supervised-looking problem whose labels are generated automatically from unlabeled data, rather than by a human annotator. The two dominant pretext families for images are contrastive learning and masked image modeling. In contrastive learning, two random augmentations of the same image (a crop, a color jitter, a blur) are treated as a positive pair, while views from different images are negative pairs. The network is trained so that positive pairs map to nearby points in an embedding space and negative pairs map to far-apart points, typically via a temperature-scaled softmax loss called the InfoNCE loss. The implicit hypothesis is that if a representation is invariant to the augmentations (which preserve semantic content) while remaining discriminative across images (which have different content), it has captured something close to "what the image is about." In masked image modeling (MIM), a large fraction of image patches — commonly 40–75% — is hidden, and the network must predict the missing content, either as raw pixels (as in MAE, Masked Autoencoders) or as discrete visual tokens produced by a separate tokenizer (as in BEiT). This mirrors masked language modeling in NLP but requires care: images are far more redundant locally than text, so trivial interpolation from neighboring pixels can solve the task without learning semantics unless the masking ratio is aggressive and the architecture is chosen carefully. Both families share a deeper commonality: they are instances of predictive self-supervision, where part of the input is used to predict another part (a different view, or the hidden pixels), forcing the network to model statistical dependencies that reflect scene structure — occlusion, object continuity, and part-whole relationships — without ever needing a semantic label. The universal failure mode to guard against is collapse, where the network finds a degenerate shortcut (e.g., mapping every image to the same constant vector, trivially satisfying an invariance loss) instead of learning anything useful; much of the architectural cleverness in this area exists purely to prevent collapse.

Contrastive learning: SimCLR

SimCLR is the cleanest instantiation of instance discrimination. Given a batch of $N$ images, each image is augmented twice to produce $2N$ views. For each view, its only true positive is the other view of the same source image; all other $2N-2$ views in the batch serve as negatives. The encoder $f$ (a ResNet or ViT) maps a view to a representation, and a small projection head $g$ maps that representation into the space where the contrastive loss is applied — the projection head is discarded after pretraining, and only $f$'s output is used downstream. ```

# SimCLR pretraining, one step

# batch: N images x_1 ... x_N (no labels)

# f: encoder, g: projection head, tau: temperature for each image x_i in batch: v_i,a = augment(x_i) # random crop, color jitter, blur, flip v_i,b = augment(x_i) # a second, independent augmentation views = [v_1,a, v_1,b, v_2,a, v_2,b, ..., v_N,a, v_N,b] # 2N views for each view v_k in views: h_k = f(v_k) # backbone representation z_k = normalize(g(h_k)) # projected, L2-normalized embedding loss = 0 for each view v_k with positive partner v_pos(k): numerator = exp(sim(z_k, z_pos(k)) / tau) denominator = sum over all j != k of exp(sim(z_k, z_j) / tau) loss += -log(numerator / denominator) loss = loss / (2N) backpropagate(loss) update(f, g) ``` Here `sim(a, b) = dot(a, b)` on L2-normalized vectors, i.e. cosine similarity. This is the InfoNCE loss: it is precisely a $2N$-way classification loss where the correct class for each view is its positive partner. Larger batch size $N$ means more negatives per positive, which empirically improves the learned representation up to very large batch sizes (SimCLR used batches of 4096–8192), because more negatives give a sharper contrastive signal and reduce the variance of the softmax denominator.

Contrastive learning without giant batches: MoCo

Large batches require large accelerators. MoCo (Momentum Contrast) decouples the number of negatives from the batch size by maintaining a queue of negative embeddings computed in previous steps, plus a momentum encoder that updates slowly to keep the queue's embeddings consistent with the current query encoder. ```

# MoCo pretraining, one step

# f_q: query encoder (trained by gradient descent)

# f_k: key encoder (momentum-updated, no gradient)

# queue: FIFO buffer of K negative embeddings, K >> batch size

# m: momentum coefficient, e.g. 0.999 for each image x_i in batch: q_i = normalize(f_q(augment(x_i))) # query view with no_grad(): k_i = normalize(f_k(augment(x_i))) # key view, same source image logits_i = [ dot(q_i, k_i) ] + [ dot(q_i, n) for n in queue ] labels_i = 0 # the key k_i is always the correct (first) entry loss = cross_entropy(logits_i / tau, labels_i), averaged over batch backpropagate(loss) # gradient flows only through f_q update(f_q)

# momentum update (no gradient) f_k.weights = m f_k.weights + (1 - m) f_q.weights enqueue(k_i for all i); dequeue oldest entries to keep queue size K ``` The momentum update is the crux: if $f_k$ were updated by gradient descent as fast as $f_q$, the embeddings already sitting in the queue would become stale relative to the current $f_q$, and the contrastive comparison would be inconsistent. A slowly-moving momentum encoder keeps the queue's negatives "fresh enough" while decoupling the negative pool size $K$ (tens of thousands) from the GPU batch size (hundreds).

Avoiding collapse without negatives: BYOL and SimSiam intuition

A subtlety: if you remove negatives entirely and only train the network to make two augmented views agree, the trivial optimum is a constant function — map every image to the same vector, and the agreement loss is zero. BYOL and SimSiam prevent this collapse using an asymmetric architecture (a predictor head on one branch only, stop-gradient on the target branch, and for BYOL a momentum target encoder) rather than negatives. The stop-gradient breaks the trivial constant-solution equilibrium: because the target branch is never updated to make the loss easier from its own side, the only way for the online branch to reduce loss is to genuinely predict the target's (non-trivial, momentum-averaged) representation.

Masked image modeling: MAE

MAE (Masked Autoencoders) exploits the ViT's patch structure directly. A large fraction of patches (typically 75%) is dropped before the encoder ever sees them — not masked-with-a-special-token, but physically removed — which makes the encoder pass cheap, since it only processes 25% of the tokens. ```

# MAE pretraining, one step

# image split into P patches, patch_embed: linear projection + position embedding

# encoder: ViT operating only on visible tokens

# decoder: lightweight ViT that sees visible tokens + mask tokens patches = split_into_patches(image) # P patches total mask_ids, visible_ids = random_partition(P, keep_ratio=0.25) visible_tokens = [ patch_embed(patches[i]) for i in visible_ids ] encoded = encoder(visible_tokens) # only 25% of tokens processed

# reinsert mask tokens (a single shared learned vector) at masked positions,

# add positional embeddings so decoder knows WHERE each token belongs decoder_input = combine(encoded, mask_token, visible_ids, mask_ids, positions) decoded = decoder(decoder_input) # full-length sequence predicted_pixels = linear_head(decoded) # per-patch pixel prediction loss = 0 for i in mask_ids: loss += mean_squared_error(predicted_pixels[i], normalize(patches[i])) loss = loss / len(mask_ids) # loss computed ONLY on masked patches backpropagate(loss) update(encoder, decoder)

# after pretraining: discard decoder, keep encoder for downstream tasks ``` Two design choices matter enormously for correctness of the pretext task. First, computing the loss only on masked patches (not visible ones) forces the network to actually infer content rather than get credit for copying visible input. Second, the very high masking ratio (75%) removes the redundancy shortcut: with low masking, a masked patch can often be interpolated trivially from adjacent visible patches (since natural images are locally smooth), so the network never needs to learn global semantic structure. At 75% masking, patches are typically far from any visible patch, so successful reconstruction requires the encoder to represent object identity, shape, and layout, not just local texture continuity.

Masked image modeling with a tokenizer: BEiT

BEiT instead predicts discrete visual tokens rather than raw pixels. A separately pretrained discrete tokenizer (e.g., a VQ-VAE-style encoder) first converts the image into a grid of discrete codes; masked patches are then predicted as a classification problem over the token vocabulary, analogous to masked language modeling over a word vocabulary. This sidesteps a subtlety of pixel-level reconstruction (that pixel-space MSE loss can over-reward blurry, low-frequency-correct predictions) by making the target a genuinely discrete, semantically pooled quantity.

SimCLR / contrastive time and space. For a batch of $N$ images producing $2N$ views, computing all pairwise similarities is $O((2N)^2 d)$ where $d$ is the embedding dimension, since the InfoNCE denominator requires a similarity between every pair of views. This is negligible next to the cost of the encoder forward/backward pass, which is $O(N \cdot C)$ for per-image encoder cost $C$ (for a ResNet-50, $C$ is on the order of $4 \times 10^9$ FLOPs; for a ViT, $C$ scales roughly with the square of the number of patches due to self-attention). Memory is dominated by storing $2N$ activations for backpropagation, which is why large-batch contrastive training needs large accelerator memory or gradient checkpointing.

MoCo time and space. The queue holds $K$ negative embeddings of dimension $d$, costing $O(Kd)$ memory — cheap, since $K$ is tens of thousands and $d$ is a few hundred, versus recomputing $K$ forward passes every step. The per-step compute is $O(N \cdot C)$ for the batch (only the query encoder needs gradients; the momentum encoder's forward pass is gradient-free but still costs a forward pass, roughly doubling per-step forward compute versus a plain encoder but not adding any backward compute for $f_k$).

MAE time and space. Because only the visible fraction $\rho$ (e.g., $\rho = 0.25$) of $P$ patches is passed to the encoder, encoder compute scales as $O(\rho P \cdot \text{(ViT layer cost)})$ instead of $O(P \cdot \text{(ViT layer cost)})$. Since self-attention cost within a ViT layer is $O(P^2 d)$ for sequence length $P$, dropping to $\rho P$ tokens reduces this term to $O(\rho^2 P^2 d)$ — a quadratic saving, roughly a 16× reduction in attention FLOPs at $\rho=0.25$. This is why MAE pretraining is markedly cheaper per step than encoding full images, despite using very deep ViT backbones.

Why contrastive learning doesn't just collapse (informally). The InfoNCE loss is a lower bound related to mutual information between the two views' representations; minimizing it maximizes a bound on $I(z_A; z_B)$ subject to the representation staying discriminative across the negatives in the batch, because the softmax denominator explicitly penalizes any representation where negatives look similar to the anchor. A constant embedding function trivially satisfies "positives agree" but catastrophically fails "negatives disagree," since every negative similarity would then also be maximal, making the denominator huge and the loss large — collapse is therefore not a minimum of the InfoNCE loss, unlike a naive squared-distance-only objective. This is the formal reason negatives are structurally necessary in this family (or an asymmetric predictor/stop-gradient substitute is needed, as in BYOL/SimSiam, to break the same trivial optimum by other means).

Why MAE's masked prediction forces semantics (informally). With masking ratio $\rho_{mask} = 0.75$, the expected distance from a masked patch to its nearest visible neighbor is large relative to the spatial correlation length of natural images (local smoothness typically decays within a few patches), so low-level interpolation is insufficient. The reconstruction loss can then only be driven down by the encoder producing a representation that encodes higher-level regularities — object shape, symmetry, category-typical texture — that generalize past the immediate neighborhood, because that is the only remaining source of predictive signal once local smoothness is exhausted.

Correctness of transfer (what "correctness" even means here). There is no formal proof that self-supervised features are optimal for a downstream task; the field's evidence is empirical: linear probing (freeze the encoder, train only a linear classifier on top) and fine-tuning accuracy on ImageNet, and mAP on detection/segmentation benchmarks, consistently show that contrastive and MIM pretraining transfer competitively with, and sometimes exceed, supervised pretraining, especially when downstream labels are scarce or when the downstream distribution differs from the label taxonomy used in supervised pretraining. The honest tradeoff: contrastive methods tend to produce features that are strong for global, instance-level tasks (classification) but historically weaker at dense/local tasks without extra tricks, while MIM tends to produce features stronger for dense prediction (detection, segmentation) since the reconstruction task is inherently local-and-global, but MIM features are often weaker under a linear probe (they need fine-tuning to shine), because the pretext task never explicitly enforces linear separability of instances the way instance discrimination does.

Trace a single SimCLR-style step with a tiny toy batch of $N = 3$ images (a cat, a dog, and a car), embedding dimension $d = 2$ for visualization, temperature $\tau = 0.5$. Suppose after augmentation and the encoder+projection head, the six L2-normalized embeddings (two per image) are:

For anchor $z_{1a}$ (cat, view A), compute cosine similarity to all five other views (dot product, since vectors are already unit-normalized):

InfoNCE loss contribution for anchor $z_{1a}$:

This is a small loss (probability $\approx 0.63$ assigned to the correct positive out of five candidates), reflecting that the cat's two augmented views are indeed much more similar to each other (dot product $0.993$) than to any other image's views (all $\le 0.392$, and strongly negative for the car). The same computation is repeated for each of the other five anchors, and the six per-anchor losses are averaged to get the batch loss, whose gradient then pushes $z_{1a}$ and $z_{1b}$ marginally closer together and pushes them marginally further from the dog and car embeddings — exactly the intended contrastive update. Notice that the dog embeddings, despite being a different image, are noticeably closer to the cat embeddings ($\approx 0.3$–$0.4$) than the car embeddings ($\approx -0.9$); a well-trained representation is expected to reflect exactly this kind of coarse visual/semantic similarity structure among negatives, even though the loss only explicitly requires negatives to be pushed away, not ranked by similarity. Now trace one MAE reconstruction step on a $4\times4$-patch toy image (16 patches, so with $\rho_{mask}=0.75$, 12 patches are masked and 4 remain visible). The encoder receives only the 4 visible-patch embeddings plus their positions; suppose these are patches at grid positions $(0,0), (1,3), (2,1), (3,2)$ — scattered, not clustered, which is typical of MAE's uniform random masking and is itself intentional (block masking would leave large uninterpolatable holes but also correlated, easier local cues at the hole's boundary, whereas scattered masking forces genuinely non-local inference across most of the image). The decoder reinserts the shared mask token at the 12 masked positions, combined with position embeddings so it knows where each unknown patch is, and predicts pixel values for all 16 patches, but the MSE loss is computed only over the 12 masked positions — the 4 visible positions contribute zero loss, since the decoder trivially could just copy them and that offers no training signal about the encoder's semantic quality.

Self-supervised vision synthesizes decades of representation-learning ideas — autoencoders, word2vec-style contrastive prediction, and metric learning — applied specifically to the structure of images.

Self-supervised pretraining is now the default backbone-training stage in production and research vision pipelines wherever labels are scarce relative to available imagery: medical imaging (pretraining on large unlabeled scan archives before fine-tuning on small annotated cohorts), satellite and remote sensing (petabytes of unlabeled imagery with sparse ground-truth), and general-purpose vision foundation models such as DINOv2, which combine contrastive and masked-prediction objectives to produce features usable off-the-shelf for retrieval, depth estimation, and segmentation without any task-specific fine-tuning. Self-driving perception stacks use self-supervised pretraining on unlabeled fleet video to warm-start detectors before fine-tuning on the much smaller set of hand-labeled frames. Search and recommendation systems use contrastively trained image embeddings directly for visual similarity search, since the InfoNCE objective is itself explicitly optimizing an embedding space for nearest-neighbor retrieval. The technique also underlies multimodal systems: CLIP's contrastive image-text objective is a direct descendant of the instance-discrimination idea, generalized to align two different modalities instead of two augmented views, and CLIP-style pretraining is now foundational to text-to-image generation and open-vocabulary detection systems.

Ask the tutor: How does DINO's self-distillation objective relate to both the contrastive and masked-image-modeling families described here?