AlexNet and Deep CNNs

AlexNet is a convolutional neural network with eight learned layers — five convolutional layers followed by three fully connected layers — trained end-to-end with stochastic gradient descent to classify 224×224 RGB images into 1,000 ImageNet categories. It follows the same conceptual blueprint as LeNet — alternating convolution, nonlinearity, and pooling, followed by dense layers and a softmax — but at a scale roughly a thousand times larger in parameters and depth of receptive field. Three ideas define what makes AlexNet more than "a bigger LeNet." First, the rectified linear unit (ReLU), $f(x) = \max(0, x)$, replaces saturating nonlinearities like sigmoid and tanh as the activation function throughout the network. Because ReLU's gradient is exactly 1 for positive inputs and 0 otherwise, it does not shrink gradients toward zero the way saturating functions do for large $|x|$, which lets gradients propagate through many layers without vanishing as severely. Second, dropout is applied in the fully connected layers: during training, each unit's activation is independently zeroed with probability $p$ (AlexNet uses $p=0.5$), forcing the network to learn redundant, distributed representations instead of co-adapting a small set of "expert" neurons — a crude but effective approximation to training an ensemble of exponentially many thinned networks. Third, the network is trained across two GPUs in parallel, with the convolutional layers partitioned so that each GPU holds half the feature maps, communicating only at specific layers — a design forced by the 3 GB memory ceiling of the GTX 580 cards available in 2012, but which also acts as a form of structured sparsity between the two halves of the network. Supporting these three pillars are secondary but important techniques: overlapping max pooling (pooling windows of size 3 with stride 2, rather than non-overlapping stride-2 windows of size 2), which slightly reduces overfitting and improves top-1/top-5 error by a fraction of a percent; local response normalization (LRN), a lateral inhibition scheme across feature-map channels inspired loosely by real neurons, since abandoned in favor of batch normalization but historically part of the recipe; and data augmentation — random crops, horizontal flips, and PCA-based color jittering — which multiplies the effective size of the training set without collecting new labels.

The network processes a 224×224×3 image (in practice, 227×227×3 given the stated stride and padding math) through five convolutional stages and three fully connected stages, ending in a 1000-way softmax.

Layer-by-layer structure (single-GPU logical view; the original splits channels across two GPUs): 1. Conv1: 96 filters, 11×11, stride 4, no padding → ReLU → local response normalization → max pool (3×3, stride 2). 2. Conv2: 256 filters, 5×5, stride 1, padding 2 → ReLU → LRN → max pool (3×3, stride 2). 3. Conv3: 384 filters, 3×3, stride 1, padding 1 → ReLU (no pooling). 4. Conv4: 384 filters, 3×3, stride 1, padding 1 → ReLU (no pooling). 5. Conv5: 256 filters, 3×3, stride 1, padding 1 → ReLU → max pool (3×3, stride 2). 6. FC6: 4096 units → ReLU → dropout (p=0.5). 7. FC7: 4096 units → ReLU → dropout (p=0.5). 8. FC8: 1000 units → softmax. Training uses mini-batch SGD with momentum 0.9, weight decay 0.0005, and a hand-tuned learning rate schedule that divides the learning rate by 10 when validation error plateaus. Weights are initialized from a zero-mean Gaussian; biases in certain layers (conv2, conv4, conv5, and the FC layers) are initialized to 1 to give ReLU units a positive input early in training, ensuring they start "on" rather than dead. ``` function ALEXNET_FORWARD(image): # image: 227 x 227 x 3 x = image

# Stage 1 x = Conv2D(x, filters=96, kernel=11, stride=4, pad=0) x = ReLU(x) x = LocalResponseNorm(x) x = MaxPool(x, size=3, stride=2) # overlapping pooling

# Stage 2 x = Conv2D(x, filters=256, kernel=5, stride=1, pad=2) x = ReLU(x) x = LocalResponseNorm(x) x = MaxPool(x, size=3, stride=2)

# Stage 3 (no pooling, deeper representation before spatial reduction) x = Conv2D(x, filters=384, kernel=3, stride=1, pad=1) x = ReLU(x)

# Stage 4 x = Conv2D(x, filters=384, kernel=3, stride=1, pad=1) x = ReLU(x)

# Stage 5 x = Conv2D(x, filters=256, kernel=3, stride=1, pad=1) x = ReLU(x) x = MaxPool(x, size=3, stride=2) x = Flatten(x) # -> 66256 = 9216

# Fully connected head x = Dense(x, units=4096) x = ReLU(x) x = Dropout(x, p=0.5) # only active at train time x = Dense(x, units=4096) x = ReLU(x) x = Dropout(x, p=0.5) logits = Dense(x, units=1000) probs = Softmax(logits) return probs

function TRAIN_STEP(batch_images, batch_labels, weights, velocity, lr): probs = ALEXNET_FORWARD(batch_images) loss = CrossEntropy(probs, batch_labels) grads = Backprop(loss, weights) for each weight w, gradient g, velocity v in zip(weights, grads, velocity): v = 0.9 v - lr g - lr 0.0005 w # momentum + weight decay w = w + v return loss ```

Parameter count. The bulk of AlexNet's ~60 million parameters lives in the fully connected layers: FC6 alone contributes $9216 \times 4096 \approx 37.7$ million weights, FC7 contributes $4096 \times 4096 \approx 16.8$ million, and FC8 contributes $4096 \times 1000 \approx 4.1$ million — over 95% of all parameters concentrated in three layers. The convolutional layers, by contrast, contribute only a few million parameters combined, since a convolutional filter's parameter count depends on kernel size and channel depth, not on spatial extent — this parameter-sharing is exactly why CNNs scale to large images at all, a principle inherited directly from LeNet.

Time complexity of one forward pass. For a convolutional layer with $C_{in}$ input channels, $C_{out}$ output filters of size $k \times k$, applied over an output feature map of size $H' \times W'$, the number of multiply-accumulate operations is

Summed over all five convolutional layers this dominates the FLOPs of the network (convolutions are compute-heavy but parameter-light), while the dense layers dominate the parameter count (dense layers are parameter-heavy but comparatively compute-light per layer, since there is no spatial replication of the weight matrix). A single forward pass through AlexNet costs on the order of $7\times 10^8$ multiply-adds, which was substantial in 2012 but is trivial by later standards.

Space complexity. Peak memory during training must hold: the parameters ($\approx 60$M floats), their gradients (same size), the momentum buffers (same size again), and the activations of every layer for every example in the mini-batch (needed for the backward pass). Activation memory dominates for large batch sizes and is the reason AlexNet had to be split across two GPUs — the feature maps at Conv1 and Conv2 are large ($55\times55\times96$ and $27\times27\times256$) and, multiplied by a batch of 128, exceeded a single card's 3GB budget when combined with parameters and gradients.

Why ReLU helps (a gradient argument). For a saturating activation like $\sigma(x) = 1/(1+e^{-x})$, the derivative $\sigma'(x) = \sigma(x)(1-\sigma(x))$ is at most $0.25$ and approaches $0$ as $|x|$ grows. In backpropagation through $L$ layers, the gradient with respect to an early layer's weights is a product of $L$ such Jacobian factors (chain rule), so a stack of saturating units can shrink gradients geometrically, a case of the vanishing-gradient problem. ReLU's derivative is exactly $1$ on the positive branch, so as long as a unit stays in its active region across the layers it participates in, no multiplicative shrinkage occurs from the nonlinearity itself — gradients pass through unattenuated by this specific mechanism, which is the mathematical reason ReLU networks in the original paper trained several times faster than an equivalent tanh network to reach the same training error.

Why dropout controls overfitting (a variance argument). Dropout can be viewed as training an ensemble of $2^n$ "thinned" sub-networks (where $n$ is the number of droppable units) that share weights, with the full network at test time approximating the geometric mean of this ensemble's predictions via weight scaling. Since ensembles reduce variance by averaging over many partially-independent models, and the shared-weight training procedure forces each unit to be useful in many different sub-network contexts rather than co-adapting with specific partners, dropout reduces the effective capacity available to memorize training-set idiosyncrasies without proportionally reducing the network's representational power. This is what allows a 60-million-parameter model to generalize from "only" 1.2 million labeled images.

Honest tradeoffs. LRN was later found empirically unnecessary once batch normalization arrived; the two-GPU split was a hardware artifact, not a fundamental algorithmic contribution, though it foreshadowed model-parallel training. Dropout, while effective, roughly doubles the number of epochs needed to converge (each thinned sub-network sees less signal per step), and the dense layers' huge parameter count is itself an inefficiency later architectures (e.g., using global average pooling) would remove.

Trace the spatial dimensions through the convolutional stack for a single 227×227×3 input image, using the standard output-size formula for a convolution:

After flattening, the 9216-dimensional vector passes through FC6 (9216→4096), FC7 (4096→4096), and FC8 (4096→1000), then softmax turns the 1000 logits into a probability distribution. Suppose, for a photograph of a Siberian husky, the raw logits at FC8 include unusually high values at indices corresponding to "Siberian husky" (index 250, logit 8.7) and "Eskimo dog" (index 248, logit 7.9), with most other logits near 0–2. Applying softmax,

concentrates probability mass on those two visually similar breeds — exactly the kind of fine-grained confusion ImageNet's 1,000-way task is known for, and exactly the kind of error the top-5 metric (checking whether the true label is among the five highest-probability classes) was designed to tolerate.

Second trace — why overlapping pooling matters. Compare Pool1 with stride 2, size 3 (overlapping) against a non-overlapping alternative of stride 2, size 2. Both reduce a 55×55 map by roughly half spatially (27×27 vs 27×27, since $(55-2)/2+1 = 27.5 \to 27$ with floor, coincidentally similar here), but the overlapping version lets each output unit see input pixels shared with its horizontal and vertical neighbors, so nearby pooled features are correlated rather than computed from disjoint pixel blocks. The original paper reports this reduces top-1 error by about 0.4% and top-5 error by about 0.3% compared to non-overlapping pooling, a small but measured and reproducible gain — illustrating that not every AlexNet design choice was a large leap; several were incremental refinements validated by ablation.

AlexNet is named for its lead author, Alex Krizhevsky, and was developed with Ilya Sutskever and Geoffrey Hinton at the University of Toronto. Hinton had spent decades advocating for neural networks through periods when the field considered them a dead end, championing backpropagation, unsupervised pretraining, and later deep architectures. The 2012 result validated that advocacy at a scale the community could not ignore.

AlexNet's direct architectural descendants — VGG, GoogLeNet/Inception, and eventually ResNet — power image classification, object detection, and segmentation systems across industry: content moderation pipelines, medical image triage, autonomous vehicle perception stacks, and satellite imagery analysis all trace their backbone designs to the conv-ReLU-pool pattern AlexNet proved out at scale. The GPU-parallel training pattern AlexNet pioneered became the default assumption for deep learning infrastructure; every major deep learning framework (PyTorch, TensorFlow, JAX) is designed around the premise that training happens on GPUs or GPU-like accelerators. Dropout, though partly superseded by other regularizers in certain settings, remains a standard tool across NLP and vision models. AlexNet's success is also credited with accelerating industrial investment in deep learning research, directly influencing the founding and growth of large-scale AI labs and cloud ML services. Its limitations — an inefficient, parameter-heavy fully connected head and a depth ceiling before training instability sets in — motivated the search for better-conditioned deep networks, a search that culminates in the skip-connection idea behind Residual Connections and ResNet.

Ask the tutor: How exactly does the two-GPU channel split in AlexNet affect which layers can "see" which feature maps, and why did the authors choose to let only some layers cross-communicate?