AlexNet
AlexNet is a deep convolutional neural network designed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton that won the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) by a stunning margin — cutting the top-5 error rate nearly in half compared to the runner-up. It reignited global interest in deep learning and established convolutional neural networks as the dominant paradigm for computer vision. The counterintuitive fact: AlexNet's most important innovations were not architectural — they were engineering tricks like dropout and GPU training that made large networks trainable for the first time.
Before AlexNet, people believed that handcrafted features (like SIFT or HOG) were necessary for image recognition — you needed domain experts to tell the model what to look for. AlexNet's "aha moment" is that if you stack enough learned filters, train on enough data, and use GPUs to make it feasible, the network discovers its own features automatically — from simple edges in layer 1 all the way to complex object parts in deeper layers. The model doesn't need to be told what a "cat ear" looks like; it figures that out on its own.
How It Works
AlexNet takes a 224×224 RGB image and passes it through a sequence of five convolutional layers followed by three fully connected layers, ultimately outputting probabilities over 1000 ImageNet classes. The fundamental operation in each convolutional layer is a convolution: a small filter (kernel) slides across the image, computing a dot product at every position to produce a feature map.
Explanation: Each output value z is the sum of elementwise products between the filter weights W and the corresponding image patch x, plus a bias term b.
- W — the learned filter weights (e.g., an 11×11×3 block of numbers)
- x — the local image patch the filter is currently sitting on
- b — a scalar bias, one per filter
- i, j — spatial positions within the filter
- c — color channel (R, G, or B in the first layer)
After every convolution, AlexNet applies the ReLU activation function:
Explanation: ReLU simply clamps all negative values to zero, keeping positive values unchanged — this introduces nonlinearity while being extremely cheap to compute.
- z — the pre-activation value from equation [1]
- f(z) — the activated output passed to the next layer
This sounds almost too simple, but replacing sigmoid (which squashes values into 0–1) with ReLU was one of AlexNet's key contributions. Gradients flow much more freely through ReLU, meaning deeper networks can actually learn. Here is a minimal Python example showing a single ReLU activation and a 2D convolution using PyTorch: ```python import torch import torch.nn as nn
# Simulate a single grayscale image: batch=1, channels=1, 6x6 pixels image = torch.randn(1, 1, 6, 6)
# A single conv layer: 1 input channel, 4 output filters, 3x3 kernel conv = nn.Conv2d(in_channels=1, out_channels=4, kernel_size=3, padding=1) relu = nn.ReLU() feature_map = relu(conv(image)) print(feature_map.shape) # → torch.Size([1, 4, 6, 6]) ``` A simplified version of the full AlexNet architecture in PyTorch looks like this: ```python import torch.nn as nn class SimpleAlexNet(nn.Module): def __init__(self, num_classes=1000): super().__init__() self.features = nn.Sequential(
# Layer 1: large 11x11 filters to capture coarse patterns nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=2), nn.ReLU(), nn.MaxPool2d(kernel_size=3, stride=2),
# Layer 2: smaller 5x5 filters nn.Conv2d(96, 256, kernel_size=5, padding=2), nn.ReLU(), nn.MaxPool2d(kernel_size=3, stride=2),
# Layers 3-5: 3x3 filters, deeper features nn.Conv2d(256, 384, kernel_size=3, padding=1), nn.ReLU(), nn.Conv2d(384, 384, kernel_size=3, padding=1), nn.ReLU(), nn.Conv2d(384, 256, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(kernel_size=3, stride=2), ) self.classifier = nn.Sequential( nn.Dropout(p=0.5), # Dropout prevents overfitting nn.Linear(256 6 6, 4096), nn.ReLU(), nn.Dropout(p=0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Linear(4096, num_classes), ) def forward(self, x): x = self.features(x) x = x.view(x.size(0), -1) # Flatten return self.classifier(x) model = SimpleAlexNet(num_classes=1000) print(model) ```
Dropout, applied with probability 0.5 in the fully connected layers, randomly zeros out half the neurons during each training step. This forces the network to learn redundant representations and dramatically reduces overfitting. At test time, dropout is turned off and all neurons are active.
MaxPooling downsamples feature maps by taking the maximum value in each small window, making the representation slightly translation-invariant and reducing spatial dimensions:
Explanation: Each pooled output value is simply the largest activation found within a local neighborhood window centered near position (i, j).
- y_{i,j} — the pooled output at position (i, j)
- 𝒩(i,j) — the set of positions within the pooling window (e.g., 3×3)
- z_{p,q} — the pre-pool activation at position (p, q)
Geoffrey Hinton is a British-Canadian cognitive psychologist and computer scientist who spent decades championing neural networks when the field was deeply unfashionable. He supervised Alex Krizhevsky and Ilya Sutskever at the University of Toronto, providing the intellectual environment and prior work on RBMs and backpropagation that made AlexNet possible. Remarkably, Hinton left Google Brain in 2023 partly to speak freely about AI risks — and shared the 2024 Nobel Prize in Physics for his foundational work on neural networks.
Alex Krizhevsky is a Ukrainian-Canadian computer scientist who built AlexNet as his PhD work under Geoffrey Hinton at the University of Toronto. His key practical contributions were implementing the entire network to run efficiently across two NVIDIA GTX 580 GPUs and creating the training pipeline that handled the full 1.2-million-image ImageNet dataset. He also created the CIFAR-10 and CIFAR-100 datasets, which remain standard benchmarks for teaching deep learning to this day.
Intuition
Imagine hiring a team of investigators to identify objects in photos. A rookie investigator looks at tiny 11×11-pixel patches and learns to spot simple things: "this patch has a diagonal edge," "this patch is red and circular," "this patch has a striped texture." Their reports get passed to slightly senior investigators who combine those observations: "diagonal edge near a circular blob = maybe an eye." Those reports go to experienced investigators who combine those: "two eyes + pointy triangular shapes + fur texture = probably a cat face." The fully connected layers at the end act like a senior analyst who reads all those reports and makes the final call. AlexNet automates this entire hierarchy — you don't hire or instruct anyone; you just show the team millions of labeled photos and let them figure out what clues to look for.