Semantic Segmentation

Semantic segmentation is a dense prediction task in computer vision where every pixel in an image is assigned a class label, producing a full-resolution semantic map rather than coarse bounding boxes or a single image-level label. It is foundational to autonomous driving, medical imaging, and augmented reality, where understanding where objects exist at pixel granularity is non-negotiable. A counterintuitive fact: early fully convolutional networks achieved state-of-the-art segmentation by discarding the fully connected layers of classification networks — the very layers responsible for "understanding" what was in the image — and replacing them with convolutions, which paradoxically improved spatial reasoning.

The core insight

The central insight of semantic segmentation is that classification and localization can be unified into a single end-to-end differentiable function by treating the output as a spatial map of class probability distributions rather than a single vector. A network that can classify an image patch can classify every patch simultaneously if its fully connected layers are replaced with convolutions — this is the "fully convolutional" epiphany. The challenge then becomes recovering the spatial resolution destroyed by pooling and strided convolutions, which gave rise to the encoder–decoder architectures that dominate the field today. Every pixel is its own multi-class classification problem, and the loss is simply the average cross-entropy over all pixels.

How It Works

Encoder–Decoder Architecture

The canonical pipeline compresses the input through an encoder (a backbone CNN or ViT that extracts hierarchical features) and then expands back to full resolution through a decoder, producing a `H × W × C` logit tensor where `C` is the number of classes.

Per-pixel cross-entropy loss is the standard training objective. Given ground-truth label yij∈{0,…,C−1}y_{ij} \in \{0, \ldots, C-1\} at pixel (i,j)(i,j) and the network's softmax output p^ij\hat{p}_{ij}:

1
LCE=−1HW∑i=1H∑j=1Wlog⁡p^ij, yij\quad \mathcal{L}_{CE} = -\frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} \log \hat{p}_{ij,\, y_{ij}}

Explanation: we average the negative log-likelihood of the correct class prediction over every pixel — identical to cross-entropy for image classification but summed spatially. Symbols: H,WH, W — spatial height and width; p^ij,c\hat{p}_{ij,c} — predicted probability for class cc at pixel (i,j)(i,j); yijy_{ij} — ground-truth class index at that pixel.

Handling Class Imbalance with Weighted Loss

Real-world datasets have severe class imbalance (background dominates foreground). A common remedy is per-class frequency weighting:

2
LWCE=−1HW∑i,jwyijlog⁡p^ij, yij,wc=median_freqfreq(c)\quad \mathcal{L}_{WCE} = -\frac{1}{HW} \sum_{i,j} w_{y_{ij}} \log \hat{p}_{ij,\, y_{ij}}, \quad w_c = \frac{\text{median\_freq}}{\text{freq}(c)}

Explanation: rare classes receive higher loss weight inversely proportional to their pixel frequency, preventing the network from ignoring thin structures like poles or cyclists. Symbols: wcw_c — scalar weight for class cc; freq(c)\text{freq}(c) — fraction of pixels belonging to class cc in the training set; median_freq\text{median\_freq} — the median of all class frequencies.

The Intersection over Union (IoU) Metric

The dominant evaluation metric is mean IoU (mIoU), averaged across classes:

3
IoUc=TPcTPc+FPc+FNc,mIoU=1C∑c=1CIoUc\quad \text{IoU}_c = \frac{TP_c}{TP_c + FP_c + FN_c}, \qquad \text{mIoU} = \frac{1}{C}\sum_{c=1}^{C} \text{IoU}_c

Explanation: IoU penalises both false positives (predicting a class where it isn't) and false negatives (missing pixels of that class), making it far stricter than pixel accuracy. Symbols: TPcTP_c — true positive pixels for class cc; FPcFP_c — false positive pixels; FNcFN_c — false negative pixels; CC — total number of classes.

Transposed Convolution for Upsampling

Decoders must upsample feature maps back to input resolution. The learnable transposed convolution computes:

4
y[m]=∑kw[k] x ⁣[⌊m−ks⌋]⋅1 ⁣[(m−k)  s=0]\quad y[m] = \sum_k w[k]\, x\!\left[\left\lfloor \tfrac{m-k}{s} \right\rfloor\right] \cdot \mathbf{1}\!\left[(m-k) \bmod s = 0\right]

Explanation: a transposed convolution inserts stride-induced zeros between input elements before convolving, effectively learning how to "spread" coarse feature maps into fine spatial detail. Symbols: xx — input feature map; ww — learned kernel; ss — stride (equals the upsampling factor); 1[⋅]\mathbf{1}[\cdot] — indicator function that enforces the stride alignment condition.

Dilated (Atrous) Convolution

DeepLab-style architectures avoid aggressive downsampling by replacing standard convolutions with atrous convolutions that expand the receptive field without reducing resolution:

5
(F∗rk)[i]=∑mF[i+r⋅m] k[m]\quad (F *_r k)[i] = \sum_{m} F[i + r \cdot m]\, k[m]

Explanation: inserting dilation rate rr between kernel taps exponentially enlarges the effective receptive field — combining rates {6,12,18}\{6,12,18\} in ASPP captures multi-scale context at a single resolution. Symbols: FF — input feature map; kk — convolution kernel; rr — dilation rate (r=1 recovers standard convolution); ii — output spatial index. --- ```python import torch import torch.nn as nn import torch.nn.functional as F class DoubleConv(nn.Module): """Two consecutive 3×3 conv + BN + ReLU blocks — the U-Net building block.""" def __init__(self, in_ch: int, out_ch: int): super().__init__() self.block = nn.Sequential( nn.Conv2d(in_ch, out_ch, 3, padding=1, bias=False), nn.BatchNorm2d(out_ch), nn.ReLU(inplace=True), nn.Conv2d(out_ch, out_ch, 3, padding=1, bias=False), nn.BatchNorm2d(out_ch), nn.ReLU(inplace=True), ) def forward(self, x): return self.block(x)

class MiniUNet(nn.Module): """Minimal U-Net illustrating skip connections and transposed-conv upsampling.""" def __init__(self, in_ch: int = 3, num_classes: int = 21): super().__init__()

# Encoder self.enc1 = DoubleConv(in_ch, 64) self.enc2 = DoubleConv(64, 128) self.pool = nn.MaxPool2d(2)

# Bottleneck self.bottleneck = DoubleConv(128, 256)

# Decoder self.up2 = nn.ConvTranspose2d(256, 128, kernel_size=2, stride=2) self.dec2 = DoubleConv(256, 128) # 128 (up) + 128 (skip) self.up1 = nn.ConvTranspose2d(128, 64, kernel_size=2, stride=2) self.dec1 = DoubleConv(128, 64) # 64 (up) + 64 (skip)

# Head self.head = nn.Conv2d(64, num_classes, kernel_size=1) def forward(self, x: torch.Tensor) -> torch.Tensor: s1 = self.enc1(x) # (B, 64, H, W) s2 = self.enc2(self.pool(s1)) # (B,128, H/2, W/2) b = self.bottleneck(self.pool(s2)) # (B,256, H/4, W/4) d2 = self.dec2(torch.cat([self.up2(b), s2], dim=1)) d1 = self.dec1(torch.cat([self.up1(d2), s1], dim=1)) return self.head(d1) # (B, num_classes, H, W)

# --- Training step --- model = MiniUNet(num_classes=21).cuda() optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-4) criterion = nn.CrossEntropyLoss(ignore_index=255) # 255 = unlabelled void def train_step(imgs: torch.Tensor, masks: torch.Tensor) -> float: """imgs: (B,3,H,W) float32; masks: (B,H,W) long""" model.train() logits = model(imgs.cuda()) # (B,21,H,W) loss = criterion(logits, masks.cuda()) optimizer.zero_grad(set_to_none=True) loss.backward() optimizer.step() return loss.item() ```

researcher · Jonathan Long | b. 1986 | Fully Convolutional Networks for Semantic Segmentation

Jonathan Long completed his PhD at UC Berkeley under Trevor Darrell and co-authored the landmark FCN paper (2015) with Evan Shelhamer and Darrell, which established the blueprint for end-to-end pixel-labelling by replacing dense layers with 1×1 convolutions and introducing skip connections from intermediate feature maps. The FCN paper won the CVPR 2015 Best Paper award and is one of the most-cited papers in computer vision history. Surprisingly, the key insight — that AlexNet's fc6 and fc7 layers are equivalent to convolutions with large kernels — required virtually no architectural change, just a reinterpretation of existing weights.

researcher · Olaf Ronneberger | b. 1973 | U-Net Architecture

Olaf Ronneberger is a computer science professor at the University of Freiburg (later Max Planck Institute) who designed U-Net in 2015 specifically for biomedical image segmentation where very few labelled images were available. His symmetric encoder–decoder with dense skip connections at every resolution level allowed the network to combine precise localisation from shallow layers with semantic context from deep layers — an architecture so effective it is still the dominant backbone for medical imaging seven years later. The original U-Net paper was written for the ISBI 2015 cell-tracking challenge, which the method won by a large margin with only 30 training images.

researcher · Liang-Chieh Chen | b. 1983 | DeepLab Series and Atrous Convolution

Liang-Chieh Chen is a research scientist at Google Brain who led the development of the DeepLab family of segmentation models (v1 through v3+), introducing atrous convolutions and the Atrous Spatial Pyramid Pooling (ASPP) module as principled replacements for resolution-destroying pooling. DeepLabv3+ (2018) remains a top-performing architecture on Cityscapes and PASCAL VOC and its design philosophy of maintaining high-resolution feature maps via dilation directly influenced subsequent transformer-based segmenters. Chen's insight that dilated convolutions and CRF post-processing could be unified into an end-to-end system removed a cumbersome hand-crafted step that the entire field had accepted as necessary.

Figure
Encoder–Decoder Architecture with Skip Connections
Semantic Segmentation: Encoder–Decoder with Skip Connections Input 3×512×512 RGB Conv×2 64 ch 512×512 skip s1 ↓MaxPool Conv×2 128 ch 256×256 ↓MaxPool Conv×2 256 ch 128×128 ↓MaxPool Bottleneck 512 ch 64×64 ↑ConvT 256 ch 128×128 +skip ↑ConvT 128 ch 256×256 +skip ↑ConvT 64 ch 512×512 +skip 1×1 Conv C classes 512×512 Logits skip connections ENCODER BOTTLENECK DECODER
Figure
Dilated Convolution Receptive Field Expansion

Atrous Spatial Pyramid Pooling (ASPP) Feature Map H×W×C 1×1 Conv r=1 3×3 Atrous rate=6 3×3 Atrous rate=12 3×3 Atrous rate=18 Global Avg Pool + 1×1