Self-Supervised Learning

Formally, let $x$ be a raw input (an image, a sentence, an audio clip) drawn from an unlabeled distribution $\mathcal{D}$. A pretext task is a function $T$ that derives an input–target pair $(\tilde{x}, y)$ from $x$ alone, with no human annotation: for example $\tilde{x}$ could be $x$ with a random patch masked out and $y$ the missing patch, or $\tilde{x}$ could be a random augmentation of $x$ and $y$ the identity of $x$ among a batch of distractors. A network is decomposed into an encoder $f_\theta$, mapping raw input to a representation $z = f_\theta(\tilde{x})$, and a pretext head $g_\phi$, mapping $z$ to a prediction of $y$. Training minimizes a pretext loss $\mathcal{L}_{\text{pretext}}(g_\phi(f_\theta(\tilde{x})), y)$ over the unlabeled corpus. After pretraining, the pretext head $g_\phi$ is discarded (or repurposed) and the encoder $f_\theta$ is transferred: either its output $z$ is fed into a new, small classifier trained with labeled data (linear probing, or more generally feature extraction), or the entire network $f_\theta$ is further trained end-to-end on the labeled task with a small learning rate (fine-tuning). The central empirical and theoretical claim of the field is that representations $z$ optimized to solve a well-designed pretext task carry semantic structure — object identity, syntax, speaker identity — that is far more sample-efficient to adapt to a new labeled task than learning from raw pixels or raw tokens directly. Self-supervised methods fall into two broad families. Generative / reconstructive methods ask the network to reconstruct missing or corrupted parts of the input — this includes denoising autoencoders, masked language modeling (BERT), masked image modeling (MAE), and autoregressive next-token prediction (GPT); these methods build directly on the encoder–decoder machinery of Variational Autoencoders but typically drop the explicit latent-variable prior in favor of a deterministic bottleneck. Discriminative / contrastive methods instead ask the network to distinguish related pairs of inputs (two augmented views of the same image) from unrelated pairs, without ever reconstructing pixels; this family — instance discrimination, SimCLR, MoCo, BYOL — is the subject of Contrastive Learning and is only sketched here as the natural next step after understanding pretext-task design.

The generic self-supervised pipeline has four stages, regardless of which family of pretext task is used.

Stage 1 — Design the pretext task. Choose a corruption or view-generation process $\tau$ that transforms $x$ into $(\tilde x, y)$ such that $y$ is recoverable from $\tilde x$ only if the encoder has learned meaningful structure, not trivial shortcuts (e.g., masking must hide enough information that copying pixels is impossible; augmentations must be strong enough that matching low-level statistics is insufficient).

Stage 2 — Pretrain. Run stochastic gradient descent on the pretext loss over the (large, unlabeled) dataset.

Stage 3 — Transfer. Freeze or partially unfreeze the encoder; attach a small task-specific head; train on the (small, labeled) downstream dataset.

Stage 4 — Evaluate. Measure downstream accuracy, typically comparing against a fully-supervised baseline trained from scratch on the same small labeled set, and against a "linear-probe" baseline to isolate representation quality from fine-tuning capacity. Below is pseudocode for a generic masked-reconstruction pretext task (the pattern behind denoising autoencoders, BERT-style masked language modeling, and MAE-style masked image modeling), followed by the transfer step. ```

# ---------- Stage 1+2: Self-supervised pretraining ----------

# D_unlabeled: large unlabeled dataset of raw inputs x

# f_theta: encoder network (produces representation z from a masked input)

# g_phi: pretext head (predicts the masked content from z)

# mask_rate: fraction of input tokens/patches to hide function PRETRAIN(D_unlabeled, f_theta, g_phi, mask_rate, num_steps, lr): for step in 1 .. num_steps: x <- sample_minibatch(D_unlabeled) # no labels used M <- sample_random_mask(x, mask_rate) # boolean mask over tokens/patches x_visible <- apply_mask(x, M) # hide masked positions z <- f_theta(x_visible) # encode the corrupted input y_hat <- g_phi(z, M) # predict content only at masked positions y_true <- extract_masked_content(x, M) # ground truth is x itself: free label loss <- reconstruction_loss(y_hat, y_true) # e.g. cross-entropy (tokens) or MSE (pixels) (theta, phi) <- gradient_step(loss, theta, phi, lr) return f_theta # keep the encoder, discard g_phi

# ---------- Stage 3: Transfer to a downstream task ----------

# D_labeled: small labeled dataset {(x_i, y_i)}

# mode: "linear_probe" (freeze f_theta) or "finetune" (adapt f_theta too) function TRANSFER(f_theta, D_labeled, mode, num_steps, lr_head, lr_backbone): h_psi <- initialize_small_head() # e.g. one linear layer if mode == "linear_probe": freeze(f_theta) for step in 1 .. num_steps: (x, y) <- sample_minibatch(D_labeled) z <- f_theta(x) # no masking at transfer time y_hat <- h_psi(z) loss <- supervised_loss(y_hat, y) # e.g. cross-entropy if mode == "linear_probe": psi <- gradient_step(loss, psi, lr_head) else: # finetune (theta, psi) <- gradient_step(loss, {theta, psi}, {lr_backbone, lr_head}) return f_theta, h_psi ``` The masking pattern generalizes immediately to other pretext tasks by changing only the `sample_random_mask` / `apply_mask` / `reconstruction_loss` steps: for autoregressive pretext tasks, the "mask" is a causal one (each position sees only earlier positions) and the loss is next-token cross-entropy; for context prediction (Word2Vec-style, or image jigsaw puzzles), $\tilde x$ is a fixed transformation (e.g. patch permutation) and $y$ is the transformation parameters; for contrastive instance discrimination, the pipeline instead produces two augmented views per input and the loss compares representations across a batch (this is developed fully under Contrastive Learning). The diagram below shows the architecture split into the reusable encoder and the disposable pretext head, and how the same encoder is reused at transfer time with a new, small head.

Time and space complexity. Pretraining cost is dominated by the forward/backward pass through the encoder $f_\theta$ and head $g_\phi$ over $N$ unlabeled examples for $E$ epochs: if the encoder has $P$ parameters and processing one example costs $O(P)$ FLOPs (true for standard feedforward/transformer/CNN layers where compute scales with parameter count times sequence length or spatial size), total pretraining compute is

which is identical in form to ordinary supervised training — self-supervision changes what the loss compares, not the asymptotic cost per example. The practical cost difference is that $N$ (unlabeled) is typically $10$–$1000\times$ larger than the labeled set available for direct supervised training, so total wall-clock pretraining cost is large, but it is a one-time cost amortized across many downstream tasks. Memory is $O(P)$ for parameters plus $O(B \cdot P)$-ish activation memory for a batch of size $B$ (exact constant depends on architecture), same as supervised training. Transfer (linear probing) is cheap: with the encoder frozen, only the head's parameters (typically $O(d \cdot C)$ for representation dimension $d$ and $C$ classes) are trained, so transfer cost is $O(M \cdot E' \cdot d \cdot C)$ for $M$ labeled examples over $E'$ epochs, dominated by the (unavoidable) forward pass through the frozen encoder if it isn't cached, or negligible if features are precomputed once.

Correctness argument. There is no single "correctness proof" for self-supervised learning the way there is for sorting, because success is an empirical, statistical claim rather than an exact guarantee; but the claim has a clean information-theoretic justification. Suppose the pretext task is designed so that solving it requires the encoder to preserve information about a latent semantic factor $s$ (object category, syntax role) that also determines the downstream label $y$. Formally, if the mutual information $I(z; s)$ between the representation $z = f_\theta(\tilde x)$ and the latent factor $s$ is high — because the pretext objective's Bayes-optimal solution provably needs $s$ to predict $y_{\text{pretext}}$ — then by the data-processing inequality, any downstream classifier can achieve at least as good a Bayes error using $z$ as using a representation with no information about $s$. Concretely: 1. Masking must destroy exactly the information you want the network to infer, and no more. If mask holes are too small, the network can interpolate from local texture/context without learning global structure (a shortcut) — the representation may end up strong on low-level statistics but weak on semantics. If mask holes are too large (e.g. removing the only instance of an object), the task becomes unsolvable and gradients are uninformative noise. Well-tuned masking rates (BERT: 15%, MAE: 75% of image patches) sit at the point where the task is solvable only via semantic inference — MAE removing 75% of patches works because natural images are highly redundant locally but globally structured, forcing reliance on object-level shape rather than pixel copying. 2. The pretext loss is a proxy objective; transfer quality is bounded, not guaranteed, by pretext performance. A network can achieve near-zero pretext loss by memorizing dataset-specific artifacts (a shortcut) uncorrelated with the downstream task — this is the well-documented "shortcut learning" failure mode, and it is why practitioners always empirically validate transfer rather than trusting pretext loss alone. 3. Linear probing isolates representation quality from optimization capacity. If a linear classifier on frozen features already achieves strong downstream accuracy, this is direct evidence that the relevant semantic factors are linearly decodable from $z$ — a strong, checkable notion of "the representation is good," rather than "the whole system, given enough capacity to compensate, can fit the labels."

Honest tradeoffs. SSL pretraining is compute-expensive up front (large models, large unlabeled corpora, long training) and its benefit is only realized when downstream labeled data is scarce relative to pretraining data; when labeled data is already abundant, self-supervised pretraining often yields diminishing or negligible returns over training from scratch. Pretext-task design is an art with no closed-form recipe — a bad pretext task (too easy, too hard, or exploitable via shortcuts) can produce representations that look fine by pretext loss but transfer poorly, so pretext task quality must always be validated by downstream evaluation, never assumed.

Consider pretraining a small masked-autoencoder-style encoder on a toy dataset of $28\times 28$ grayscale digit images (structurally analogous to MNIST, no labels used), then transferring to 10-class digit classification with only 100 labeled examples.

Setup. Patchify each image into 16 patches of $7\times 7$ pixels. Randomly mask 9 of the 16 patches (≈56%, a moderate masking rate for a small toy image). Encoder: a 4-layer transformer over the 7 visible patch embeddings ($d=64$). Pretext head: a linear layer that reconstructs the pixel values of the 9 masked patches from the encoder's masked-position tokens (mean squared error loss).

The gap between step 9 and step 10 (91.4% vs 68.2%) is exactly the value added by self-supervised pretraining: the encoder learned, from thousands of unlabeled digit images, that a handwritten stroke pattern is locally predictable from its neighboring strokes, which happens to require encoding stroke topology — precisely the feature that discriminates digit classes. No label ever mentioned "seven"; the label-free reconstruction objective nonetheless induced digit-relevant structure because reconstructing occluded strokes and discriminating digit identity share the same underlying latent factor (stroke topology).

A second trace — why masking rate matters. Repeat the same experiment with only 1 of 16 patches masked (≈6%). Pretext MSE drops even faster (to near $0.001$) because each masked patch is almost fully predictable from its immediate neighbors via simple texture interpolation — the network never needs to reason about global stroke topology. Linear-probe accuracy on the same 100 labeled examples: $73.8\%$, barely better than the from-scratch baseline. This demonstrates concretely that low pretext loss does not imply good representations — the 6%-masking encoder solves its pretext task better (lower MSE) than the 56%-masking encoder, yet transfers far worse, because it learned local interpolation rather than global structure.

Self-supervised learning as a named paradigm crystallized from decades of work on unsupervised pretraining, denoising autoencoders, and word embeddings, and was substantially systematized and popularized by researchers working on deep representation learning, including Yann LeCun, who has repeatedly argued self-supervised learning is the primary lever for building general-purpose perception systems, and by teams behind BERT, word2vec, and masked autoencoders who turned the idea into standard industrial practice.

Self-supervised pretraining is the backbone of essentially every modern foundation model. Large language models (GPT-family, LLaMA, PaLM) are pretrained with autoregressive next-token prediction over web-scale text corpora before any instruction-tuning or reinforcement learning from human feedback is applied. Masked language modeling (BERT and its descendants) pretrains bidirectional text encoders used across search ranking, retrieval, and classification pipelines in production systems at major search and recommendation companies. In computer vision, masked image modeling (MAE) and joint-embedding methods (DINO, I-JEPA) pretrain vision backbones later fine-tuned for detection, segmentation, and medical imaging tasks where labeled data is expensive to obtain — a radiology lab with a handful of expert-annotated scans can fine-tune an encoder pretrained self-supervised on millions of unlabeled scans from routine clinical archives. Speech recognition systems (wav2vec 2.0, HuBERT) use masked prediction over audio codes to pretrain on unlabeled speech before fine-tuning on small transcribed datasets, enabling accurate transcription for low-resource languages. Self-supervised pretraining also underlies protein-structure and genomics foundation models (masked residue prediction), and drives self-driving perception stacks that pretrain on vast unlabeled driving footage before fine-tuning on the smaller labeled set of annotated hazards. In every one of these systems, the practical motivation is identical: unlabeled data is orders of magnitude cheaper and more abundant than labeled data, and self-supervision converts that abundance directly into representation quality.

Ask the tutor: How does the InfoNCE loss used in contrastive learning relate to the mutual-information argument for why self-supervised representations transfer well?