Image Classification with Deep Networks

Whole-image classification is the supervised learning task of assigning an image $x$ to exactly one label $y \in \{1, \dots, K\}$ from a predefined, mutually exclusive set of categories. This differs from object detection (which localizes multiple objects with bounding boxes) and from segmentation (which labels every pixel); classification produces a single global judgment about the entire image. The standard architecture has two conceptual parts. First, a convolutional backbone — built from the stacked convolution, nonlinearity, and pooling operations covered in CNNs for Images Revisited — transforms the raw pixel array into a compact feature vector, progressively trading spatial resolution for semantic abstraction. Second, a classification head takes that feature vector and produces a probability distribution over the $K$ classes. The classification head is almost always a single linear (fully connected) layer producing $K$ real-valued logits $z_1, \dots, z_K$, followed by the softmax function:

Softmax converts arbitrary real-valued scores into a valid probability distribution: every $p_k \in (0,1)$ and $\sum_k p_k = 1$. It is the multi-class generalization of the logistic sigmoid, and it is not an arbitrary choice — it falls out of assuming the logits are proportional to log class-conditional likelihoods under an exponential-family model, making the softmax classifier the maximum-entropy distribution consistent with those scores. Training minimizes the cross-entropy loss between the predicted distribution $p$ and the one-hot true label $y$:

Minimizing cross-entropy over the training set is exactly maximum-likelihood estimation of the network's parameters under the assumption that labels are drawn from the softmax-induced categorical distribution — this is the statistical justification for why "train with cross-entropy" is the correct objective rather than a heuristic. The architectural history of the field — AlexNet, VGG, ResNet — is the story of how to build the backbone that feeds this head so that the resulting features are linearly separable by class.

The training pipeline has three moving parts: the forward pass through the backbone and head, the loss computation, and backpropagation through both to update every weight. Architecturally, the three canonical networks differ mainly in how the backbone is built, while the head and training loop stay essentially fixed.

AlexNet (8 weight layers: 5 convolutional, 3 fully connected) established the template: convolution → ReLU → (occasional) local response normalization → max-pooling, repeated with increasing channel depth and decreasing spatial resolution, followed by two large fully connected layers, dropout for regularization, and a final $K$-way softmax. Its key engineering contributions were using ReLU instead of saturating sigmoids (faster convergence), training on two GPUs in parallel, using dropout to fight overfitting in the huge fully-connected layers, and using aggressive data augmentation (foreshadowed further in Data Augmentation for Vision).

VGG simplified the design space: use only $3\times3$ convolutions with stride 1 and padding 1 (which preserve spatial size), only $2\times2$ max-pooling with stride 2 (which halves spatial size), and double the channel count after every pooling stage. Two stacked $3\times3$ convolutions have the same receptive field as one $5\times5$ convolution but with fewer parameters and an extra nonlinearity, and three stacked $3\times3$ convolutions match a $7\times7$ receptive field while using roughly half the parameters and adding two extra ReLUs. VGG-16 and VGG-19 (16 and 19 weight layers) demonstrated that with this simple recipe, accuracy improves monotonically with depth, up to a point.

ResNet solved the point where it stops improving. Beyond roughly 20 layers, plain stacked networks suffer from the degradation problem: training accuracy — not just test accuracy — gets worse as depth increases, which rules out overfitting as the explanation and points instead to an optimization difficulty. ResNet's fix is the residual block: instead of asking a stack of layers to learn a full mapping $H(x)$, it asks them to learn a residual $F(x) = H(x) - x$, and adds the input back via a shortcut connection:

If the optimal mapping for a block is close to the identity, it is trivial to drive $F(x) \to 0$ (push weights toward zero), whereas forcing a plain stack of nonlinear layers to approximate the identity exactly is empirically hard. The shortcut also gives gradients a direct, unimpeded path back to earlier layers during backpropagation, mitigating vanishing gradients in very deep stacks. ```

# Generic image-classifier training loop (applies to AlexNet, VGG, ResNet) function TRAIN_CLASSIFIER(model, train_data, epochs, lr): initialize model.weights # e.g. He/Kaiming init for ReLU nets for epoch in 1..epochs: for (batch_images, batch_labels) in MINIBATCHES(train_data):

# ---- forward pass ---- features = model.backbone(batch_images) # conv stack -> feature map pooled = GLOBAL_OR_FLATTEN_POOL(features) # -> fixed-length vector logits = model.head(pooled) # linear layer -> K logits probs = SOFTMAX(logits) # per-example distribution loss = CROSS_ENTROPY(probs, batch_labels) loss += weight_decay * L2_NORM(model.weights) # regularization

# ---- backward pass ---- grads = BACKPROP(loss, model.weights) # chain rule through all layers model.weights = SGD_UPDATE(model.weights, grads, lr) # or Adam, momentum, etc. lr = DECAY_SCHEDULE(lr, epoch) # step / cosine decay return model function SOFTMAX(logits): # logits: length-K vector m = MAX(logits) # subtract max for numerical stability exps = [EXP(z - m) for z in logits] s = SUM(exps) return [e / s for e in exps] function CROSS_ENTROPY(probs, label_index): return -LOG(probs[label_index] + epsilon) # epsilon avoids log(0) function RESIDUAL_BLOCK(x, weights1, weights2): h = RELU(CONV(x, weights1)) h = CONV(h, weights2) if SHAPE(x) != SHAPE(h): x = CONV_1x1_PROJECTION(x) # match dims/stride when block downsamples return RELU(h + x) # the skip connection ``` The `RESIDUAL_BLOCK` function is the one structural addition ResNet makes to the plain VGG-style stack; everything else — convolution, ReLU, pooling, softmax head, cross-entropy training — is shared machinery.

Forward-pass time complexity. For a convolutional layer with input size $H \times W \times C_{in}$, kernel size $k \times k$, and $C_{out}$ output channels applied with stride 1, the cost is

multiply-accumulate operations, because each of the $H \cdot W$ output positions per output channel requires a $k \times k \times C_{in}$ dot product, repeated for $C_{out}$ channels. A full backbone's cost is the sum of this quantity over all layers. VGG's design choice of two stacked $3\times3$ convolutions in place of one $5\times5$ is not just about receptive field: parameter count drops from $25 C^2$ to $2 \cdot 9 C^2 = 18 C^2$ for equal channel width $C$, and three $3\times3$ layers replacing one $7\times7$ drop parameters from $49C^2$ to $27C^2$ — roughly 45% fewer weights for the same receptive field, plus two additional ReLU nonlinearities that increase representational capacity.

Space complexity. Training must store activations from the forward pass for every layer to compute gradients during backpropagation (reverse-mode automatic differentiation), giving memory cost linear in the number of layers times the feature map size at each layer — this is why very deep networks are memory-hungry at training time even though inference-only memory is much smaller (activations can be discarded layer by layer). AlexNet's split across two GPUs in the original paper was driven precisely by this memory constraint on 2012-era hardware.

Why softmax + cross-entropy is the correct training objective. Treat the network as producing $K$ logits $z(x;\theta)$ as a function of image $x$ and parameters $\theta$. Define the likelihood of the training set under the categorical model $P(y \mid x;\theta) = \text{softmax}(z(x;\theta))_y$. The log-likelihood of the whole training set, assuming i.i.d. samples, is

So minimizing the sum of per-example cross-entropy losses is exactly maximizing the likelihood of the observed labels under this model — cross-entropy is not a heuristic surrogate but the negative log-likelihood itself. A useful correctness check on the gradient: differentiating the loss with respect to logit $z_k$ gives the clean expression

meaning the backpropagated error signal at the output is simply "predicted probability minus 1 for the true class, minus 0 otherwise" — an intuitively correct and numerically well-behaved gradient with no vanishing or exploding terms at the classification head itself.

Why residual connections fix degradation. Consider a plain block computing $H(x)$ via stacked nonlinear layers versus a residual block computing $F(x) + x$ where $F$ is the same stack. If the identity mapping is (near-)optimal for a given block, the residual formulation only needs to drive the weights of $F$ toward zero — trivially achievable via gradient descent with small weight initialization — whereas the plain formulation must fit a stack of nonlinear (ReLU) layers to approximate identity exactly, which is a harder function-approximation target for generic nonlinear layers with random initialization. Empirically, this explains why plain networks past ~20 layers show rising training error (an optimization failure, not overfitting), while ResNets with skip connections train stably to 100+ and even 1000+ layers. During backpropagation, the identity shortcut in $y = F(x) + x$ contributes a $+1$ term to $\partial y/\partial x$, guaranteeing that gradient magnitude does not shrink to zero purely due to depth, which mitigates (though does not entirely eliminate) vanishing gradients.

Tradeoffs to be honest about. Softmax cross-entropy assumes classes are mutually exclusive and exhaustive; it is the wrong loss for multi-label problems (an image containing both "dog" and "frisbee") where independent per-class sigmoids are correct instead. Deeper is not always better in practice — depth increases inference latency and memory, and beyond a point accuracy gains saturate while cost keeps rising, so real deployments trade off ResNet-50 against ResNet-152 based on latency budgets. VGG's huge fully connected layers (over 100 million parameters in the first FC layer alone) make it far more parameter-heavy and slower to train than ResNet-50 despite similar or worse accuracy, which is why ResNet variants dominate as backbones today.

Trace a single training example through a small 3-class classifier to make the arithmetic concrete. Suppose after the backbone and pooling, the feature vector is $\phi$, and the final linear layer produces logits for classes {cat, dog, bird}: $z = [2.0,\ 1.0,\ 0.1]$. The true label is "dog" (index 2).

Step 1 — softmax. Subtract the max for stability ($m = 2.0$):

Sum $= 1.0000 + 0.3679 + 0.1496 = 1.5175$.

Check: $0.659 + 0.242 + 0.099 = 1.000$ ✓.

Step 2 — cross-entropy loss. True label is "dog", so

Because the model's top prediction ("cat", 65.9%) is wrong, the loss is substantial — a correct, confident prediction would give a loss near zero (e.g., $-\log(0.95) = 0.051$).

Step 3 — gradient at the logits. Using $\partial \mathcal{L}/\partial z_k = p_k - \mathbb{1}[k=y]$:

The gradient pushes the "cat" and "bird" logits down (positive gradient means increasing $z$ increases loss, so gradient descent decreases them) and pushes the "dog" logit up — exactly the correction the network needs, with magnitude proportional to how wrong the current probability is. This 3-vector then flows backward through the fully connected layer and the entire convolutional stack via the chain rule, updating every convolution filter by a tiny amount in the direction that would have made "dog" more likely for this particular image.

Step 4 — architecture comparison on the same input. If this feature vector $\phi$ came from three different backbones processing a $224\times224\times3$ cat photo, the backbones differ enormously upstream even though the head computation above is identical:

Notice ResNet-152 has fewer parameters than AlexNet despite being nearly 20× deeper — depth from residual blocks with bottleneck 1×1 convolutions is far more parameter-efficient than width from giant fully connected layers, and it achieves substantially lower error.

Convolutional classifiers descended from this lineage are the backbone of nearly every modern vision system. Object detectors (Faster R-CNN, YOLO, SSD) and segmentation networks reuse a pretrained classification backbone — typically a ResNet variant — as a feature extractor, a strategy formalized in Transfer Learning for Vision. Photo organization apps use classification-style features for search and duplicate detection. Medical imaging pipelines use fine-tuned classifiers to flag likely-malignant regions for radiologist review. Content moderation systems at social platforms run classification models continuously on uploaded images and video frames. Retail and e-commerce visual-search products embed images using classifier-derived features to find visually similar products. The benchmark that drove this progress, and the datasets and evaluation protocols that made comparison possible across labs, are covered in ImageNet and Benchmark Datasets; the training tricks (random crops, flips, color jitter) that let these networks generalize from finite training sets are covered in Data Augmentation for Vision.

Ask the tutor: Can you show why the residual formulation's gradient never fully vanishes, using the explicit chain rule across several stacked residual blocks?