LeNet and the CNN Blueprint
In 1989 the US Postal Service wanted machines to read handwritten zip codes off millions of envelopes. Bank checks needed automated digit recognition too. The scale was too large for any team of human clerks to sustain. The images were small, noisy, and full of variation. The same digit "7" could be tall, slanted, or crossed. Yet a human reads it instantly. This is because the eye recognizes local strokes and combines them into shapes. Yann LeCun and collaborators built a machine that mimicked this idea directly in a neural network. By the mid-1990s it was reading roughly 10-20% of all checks processed in the United States. That machine, culminating in LeNet-5 (1998), established a template so influential that nearly every image classifier since is a variation on its blueprint. This includes AlexNet, VGG, ResNet, and even parts of modern vision transformers' convolutional stems. The blueprint is deceptively simple. It alternates convolution layers with pooling layers. Convolution layers detect local patterns while sharing parameters across space. Pooling layers shrink the spatial map and add tolerance to small shifts. What remains is then flattened into a small vector. That vector is fed to fully-connected layers that make the final decision. Why does this matter beyond a history lesson? Because the conv-pool-FC pattern encodes a strong, correct prior about images. The identity of a stroke does not depend on where it sits in the frame. Small translations or distortions should not change the digit's label. Before LeNet, a fully-connected network applied to a 32×32 image needed roughly a thousand weights per neuron in the first layer alone. It had no way to know that a "loop" detector useful in the top-left corner is equally useful in the bottom-right. LeNet enforces weight sharing so the same small filter slides everywhere. This cuts parameters by orders of magnitude. It also makes learning dramatically more sample-efficient. This article gives the exact layer-by-layer architecture of LeNet-5. It works through the shape and parameter arithmetic for every layer. It explains why the design is correct as an inductive bias. It traces a digit through the network by hand. Finally, it connects the blueprint forward to the deeper CNNs that inherited it.
- LeNet-5 alternates convolution → pooling → convolution → pooling, then flattens into fully-connected layers ending in a classifier — the "conv-pool-FC" template.
- Convolution layers exploit parameter sharing and locality: a small filter slides over the image, detecting the same pattern anywhere it appears.
- Pooling layers downsample, shrinking spatial resolution while adding robustness to small translations and reducing computation for later layers.
- Parameter counts are computed exactly per layer: LeNet-5 has about 60,000 trainable parameters, tiny by modern standards but huge relative to input size at the time.
- The architecture was validated on MNIST-style digit recognition and deployed commercially for check reading — an early, real production success of deep learning.
- LeNet's skeleton — feature extractor (conv+pool) followed by a classifier head (FC) — persists in AlexNet and Deep CNNs and remains visible even as depth and residual connections were added later.
LeNet showed that stacking local, weight-shared convolutions with pooling — instead of a fully-connected network — turns image recognition into a tractable, sample-efficient learning problem.
The idea
A convolutional neural network in the LeNet mold processes an image through a sequence of stages with two distinct roles. Feature-extraction stages consist of a convolution layer followed by a nonlinearity and a pooling layer; they transform the raw pixel grid into increasingly abstract, increasingly coarse feature maps. Classification stages consist of one or more fully-connected layers that take the (now small, flattened) feature vector and output class scores. This is the conv-pool-FC blueprint: the network narrows spatially while it widens in the number of feature channels, then collapses entirely into a vector for decision-making. Three architectural principles justify this design, and each is a direct response to a weakness of plain fully-connected networks on images. Locality: a stroke, edge, or corner is defined by a small neighborhood of pixels, so a detector only needs a small receptive field (e.g. 5×5), not a connection to every pixel in the image. Parameter sharing: the same edge-detector is useful at every spatial location, so instead of learning a separate detector per location (as a fully-connected layer would), the network learns one filter and convolves it across the whole image — one set of weights, reused thousands of times. Subsampling (pooling): exact pixel position is usually irrelevant to the digit's identity, so after detecting a feature, the network can afford to blur where exactly it occurred, shrinking the map and gaining invariance to small translations and elastic distortions in the input. LeNet-5 specifically stacks two feature-extraction stages (C1-S2, C3-S4) and then three fully-connected-style stages (C5, F6, output), operating on 32×32 grayscale images and producing 10 class scores for the digits 0-9. Every design choice — the filter sizes, the number of channels, the pooling factors — was tuned so that the receptive field of a unit at the top of the network eventually spans the entire input, letting later layers reason globally about combinations of local strokes detected earlier.
How it works
The exact LeNet-5 layer sequence, with the tensor shape flowing through it (format: channels × height × width), is:
| Layer | Type | Output shape | Kernel | Stride | Trainable params |
|---|---|---|---|---|---|
| Input | — | 1×32×32 | — | — | 0 |
| C1 | Conv (6 filters) | 6×28×28 | 5×5 | 1 | (5·5·1+1)·6 = 156 |
| S2 | Avg-pool | 6×14×14 | 2×2 | 2 | 6·2 = 12 (scale+bias) |
| C3 | Conv (16 filters, partial connectivity) | 16×10×10 | 5×5 | 1 | 1,516 (see below) |
| S4 | Avg-pool | 16×5×5 | 2×2 | 2 | 16·2 = 32 |
| C5 | Conv (120 filters, acts as FC) | 120×1×1 | 5×5 | 1 | (5·5·16+1)·120 = 48,120 |
| F6 | Fully-connected | 84 | — | — | (120+1)·84 = 10,164 |
| Output | Fully-connected (RBF in original / softmax in modern versions) | 10 | — | — | (84+1)·10 = 850 |
Summing the trainable parameters gives 156 + 12 + 1,516 + 32 + 48,120 + 10,164 + 850 ≈ 60,850, matching the commonly cited "~60K parameters" for LeNet-5.
Why C1 produces 28×28 from a 32×32 input. With no padding ("valid" convolution) and stride 1, a 5×5 kernel applied to an `n×n` input produces an `(n−4)×(n−4)` output, since the kernel's top-left corner can occupy `n−5+1` positions per dimension. So `32 − 5 + 1 = 28`. This is why LeNet pads the original 28×28 MNIST digit to 32×32 first — it wants the very center of each stroke to be visible to the first layer without truncation at the border, and it wants the arithmetic to work out to the classic 28→14→10→5 cascade.
Why S2 produces 14×14. Average pooling with a 2×2 window and stride 2 is non-overlapping: it partitions the 28×28 map into 14×14 disjoint blocks and averages each, output `28/2 = 14`. Each of the 6 channels gets its own learned scalar multiplier and bias applied after averaging (this is why S2 has 12 parameters, not 0) — a detail specific to the original LeNet design, though modern implementations typically use parameter-free average or max pooling.
Why C3 has only 1,516 parameters instead of the naive `(5·5·6+1)·16 = 2,416`. LeNet-5's C3 layer deliberately does not connect every output channel to every input channel. Each of the 16 output feature maps is connected to only a subset of the 6 input maps (some to 3, some to 4, one to all 6), following a fixed connection table designed by the original authors. The stated reasons were (a) to break symmetry so different feature maps learn different combinations rather than converging to redundant features, and (b) to keep the parameter count and computation manageable given 1998 hardware. Concretely, LeCun et al.'s table gives 6 maps connected to 3 inputs, 9 maps connected to 4 inputs, and 1 map connected to all 6, yielding: ``` 6·(3·25+1) + 6·(4·25+1) + 3·(4·25+1) + 1·(6·25+1) = 6·76 + 6·101 + 3·101 + 1·151 = 456 + 606 + 303 + 151 = 1,516 ``` Modern reimplementations often simplify C3 to full connectivity (accepting the larger 2,416-parameter count) since GPUs make the saving irrelevant; the sparse table is a historical artifact worth understanding but not essential to reproduce.
Why C5 is "fully connected but drawn as convolutional." After S4, the feature map is 16×5×5. A 5×5 convolution over a 5×5 input with no padding produces a 1×1 output — meaning C5 sees the entire spatial extent at once and reduces it to a single scalar per output channel, of which there are 120. This is mathematically identical to flattening the 16×5×5 = 400 values and applying a fully-connected layer with 120 outputs; LeCun's paper implements it as a convolution purely so that if the input image were larger, C5 would still slide and produce a small spatial map rather than breaking. ```pseudocode
# LeNet-5 forward pass (shapes shown as C x H x W) function LENET5_FORWARD(image): # image: 1 x 32 x 32 x = CONV(image, filters=6, kernel=5, stride=1) # -> 6 x 28 x 28 x = ACTIVATION(x) # tanh (original) or ReLU (modern) x = AVG_POOL(x, window=2, stride=2) # -> 6 x 14 x 14 (S2) x = CONV(x, filters=16, kernel=5, stride=1, connectivity=LENET_C3_TABLE) # -> 16 x 10 x 10 (C3, sparse wiring) x = ACTIVATION(x) x = AVG_POOL(x, window=2, stride=2) # -> 16 x 5 x 5 (S4) x = CONV(x, filters=120, kernel=5, stride=1) # -> 120 x 1 x 1 (C5, full receptive field) x = ACTIVATION(x) x = FLATTEN(x) # -> 120 x = FULLY_CONNECTED(x, out=84) # -> 84 (F6) x = ACTIVATION(x) scores = FULLY_CONNECTED(x, out=10) # -> 10 class scores return SOFTMAX(scores) # class probabilities function CONV(x, filters, kernel, stride, connectivity=FULL): for each output channel o in filters: acc = bias[o] for each input channel i connected to o (per `connectivity`): acc += CROSS_CORRELATE(x[i], weight[o][i], stride) out[o] = acc return out function AVG_POOL(x, window, stride): for each channel c: for each output position (r, c2): patch = x[c, rstride : rstride+window, c2stride : c2stride+window] out[c, r, c2] = MEAN(patch) # non-overlapping when stride == window return out ```
Complexity & correctness
Parameter complexity. Fully-connected networks applied directly to a `32×32` image would need, just for a first layer with `k` hidden units, `1024·k` weights — e.g. 100 hidden units already costs over 100,000 parameters, more than LeNet-5's entire network. Convolutional weight sharing reduces a layer's parameter count from `O(H·W·C_in·C_out)` (fully-connected) to `O(k_h·k_w·C_in·C_out)` (convolutional), where `k_h, k_w` are kernel dimensions — independent of the spatial size `H×W`. This is the central efficiency argument: parameter count no longer scales with image resolution, only with kernel size and channel counts, so the same filter bank can in principle be applied to a much larger image without adding parameters.
Time complexity. A convolution layer with `C_in` input channels, `C_out` output channels, kernel `k×k`, applied to a spatial output of size `H'×W'`, costs ``` O(H' · W' · C_in · C_out · k²) ``` multiply-adds — every output pixel requires a `k×k×C_in` dot product, repeated `C_out` times. For LeNet's C1: `28·28·1·6·25 ≈ 118K` MACs; for C3 (full connectivity): `10·10·6·16·25 ≈ 240K` MACs. Pooling layers cost `O(H'·W'·C·k²)` but with no learned weights involved in the reduction itself (beyond the optional scale/bias), so they are cheap relative to convolutions. Fully-connected layers cost `O(n_in · n_out)`; C5 is the single most expensive layer here at `48,120` MACs comparable to weights, F6 costs `120·84 ≈ 10K`. Total forward-pass cost is on the order of a few hundred thousand MACs — trivial for even 1990s hardware per image, which is exactly why LeNet-5 could run in real time on bank check scanners.
Space complexity. Memory for weights is the parameter count itself (~61K numbers, a few hundred KB at 32-bit precision — minuscule by any modern standard). Activation memory at inference is dominated by the largest intermediate feature map, `6×28×28 = 4,704` values after C1, again negligible.
Correctness as an inductive bias (why the architecture is "right" for images). The claim to justify is not a theorem in the sense of a guaranteed error bound, but a modeling assumption that provably reduces the hypothesis space to one matching the true structure of image data: 1. Translation equivariance of convolution. If `f` is a convolution operator and `T_v` denotes a spatial shift by vector `v`, then `f(T_v(x)) = T_v(f(x))` (up to boundary effects). This is a direct consequence of using the same kernel at every position — shifting the input shifts the output identically. A fully-connected layer has no such property; it would need to re-learn a shifted-version detector from scratch for every possible shift, requiring far more data to generalize. 2. Translation invariance from pooling. Average or max pooling over a window `w` makes the output insensitive to shifts smaller than `w`: if a feature activates anywhere inside a pooling window, the pooled output changes only slightly (average) or not at all (max), so the representation becomes progressively less sensitive to exact location as it propagates upward — exactly the property digit recognition needs, since a "7" is a "7" regardless of a one-pixel jitter. 3. Compositionality via depth. Layer C1 detects small strokes/edges (5×5 receptive field). Layer C3, after pooling, has an effective receptive field back in the input covering roughly `14×14` input pixels (since each C3 unit looks at a 5×5 window of the already-pooled S2 map, and each S2 unit already summarizes a 2×2 patch of C1's 28×28 map, whose own units each see a 5×5 patch of the input). By C5 the receptive field covers the entire 32×32 input. This growing receptive field lets the network build a hierarchy: edges → strokes/loops → whole-digit templates, mirroring how a human would describe recognizing a digit compositionally. Taken together, these three properties mean the hypothesis class LeNet searches over already contains "translation-equivariant, locally-compositional, shift-tolerant classifiers" — a class that (empirically and by construction) contains good digit classifiers with far fewer parameters than the class of arbitrary fully-connected functions, hence needs less data and less computation to find one that generalizes.
Honest tradeoffs. The inductive bias is a double-edged sword: convolution assumes translation equivariance is always useful, which can hurt when absolute position matters (e.g. some medical imaging tasks). Average pooling with fixed non-overlapping windows discards information irrecoverably — a large stride pooling layer cannot be undone, which is one reason later architectures moved to max pooling, overlapping pooling, or learned strided convolutions instead. And the sparse C3 connectivity table, while parameter-efficient, is a hand-engineered artifact — a decision later architectures abandoned entirely in favor of full connectivity, since GPUs made the parameter savings unnecessary and the hand-designed sparsity pattern doesn't generalize to new problems.
Worked example
Trace an actual (schematic) 32×32 padded MNIST digit "3" through LeNet-5, tracking shapes and what each stage plausibly represents:
| Stage | Operation | Output shape | What it captures |
|---|---|---|---|
| Input | zero-padded MNIST digit | 1×32×32 | Raw pixel intensities of a handwritten "3" |
| C1 | 6 learned 5×5 filters, valid conv | 6×28×28 | 6 maps of low-level strokes: e.g. map 1 fires on horizontal top-curve, map 2 on the middle inward dent, map 3 on the bottom curve, others on diagonal edges |
| S2 | 2×2 avg-pool, stride 2 | 6×14×14 | Same 6 stroke types, but each spatial cell now summarizes a 2×2 neighborhood — jitter of ±1 pixel in the original stroke no longer changes the map much |
| C3 | 16 filters, sparse connectivity over subsets of the 6 S2 maps | 16×10×10 | Combinations: e.g. one map might combine "top-curve" + "middle-dent" evidence to detect the upper loop of the "3"; another combines "bottom-curve" + "middle-dent" for the lower loop |
| S4 | 2×2 avg-pool, stride 2 | 16×5×5 | Coarse loop/curve detectors, tolerant to shift; total 400 values remain |
| C5 | 120 filters, kernel spans full 5×5 | 120×1×1 | Each of 120 units is a learned weighted combination of all 400 S4 values — effectively a "global template match" against holistic patterns like "double-loop stacked vertically" |
| F6 | fully connected | 84 | A further nonlinear recombination into an 84-dim code (LeCun's original design chose 84 to match a 7×12 stylized bitmap used for the RBF output layer) |
| Output | fully connected + softmax | 10 scores | The class score for "3" should dominate; softmax turns the 10 raw scores into probabilities |
Concretely, suppose the 10 raw output scores (logits) before softmax are: ``` digit: 0 1 2 3 4 5 6 7 8 9 logit: -1.2 -3.4 0.8 5.1 -0.3 1.0 -2.0 -0.7 2.2 0.1 ``` Applying softmax, `p_i = exp(z_i) / Σ_j exp(z_j)`, the digit "3" logit of 5.1 is far larger than any competitor (next closest is "8" at 2.2, a gap of 2.9), so after exponentiating, `p_3` dominates the sum overwhelmingly — the network confidently predicts "3", and correctly so if trained well, since 5.1 vs. 2.2 in logit space corresponds to roughly `e^{2.9} ≈ 18×` more unnormalized mass for the correct class over the nearest confusable digit (an "8", whose loops are indeed the most visually similar competitor to a "3" — a sensible confusion for the network to almost make and then resolve). This trace illustrates why the architecture works, not just that it does: strokes (C1) combine into partial loops (C3), partial loops combine into whole-digit templates (C5), and the FC layers linearly separate those templates into class scores — a hierarchy of increasing abstraction and shrinking spatial resolution, exactly matching the conv-pool-FC blueprint's intent.
Common pitfalls & traps
| Pitfall | Why it happens | Fix |
|---|---|---|
| Forgetting to pad 28×28 MNIST to 32×32 | Naively feeding native MNIST size breaks the classic 28→14→10→5 shape cascade LeNet-5 assumes | Zero-pad by 2 pixels on each side, or adjust kernel/padding to match whatever input size is actually used |
| Assuming all conv layers must be fully connected across channels | The historical C3 layer uses sparse connectivity; modern reimplementations often silently switch to full connectivity, changing parameter count | State explicitly whether you're using the original sparse table (1,516 params) or full connectivity (2,416 params) when reporting parameter counts |
| Using max activation after pool without realizing original LeNet used tanh + average pooling with learned scale/bias | Later CNNs standardized on ReLU + max-pool; mixing eras causes confusing accuracy or gradient comparisons | Be explicit about which nonlinearity/pooling variant is being used; both are valid but not interchangeable when reproducing historical results |
| Treating C5 as "just another conv layer" and adding padding | If padding is added to a 5×5 input with a 5×5 kernel, the output is no longer 1×1, and the intended flatten-into-FC equivalence breaks | Keep C5 as a valid (no-padding) convolution whenever the preceding spatial size is exactly equal to the kernel size |
| Ignoring receptive field growth when scaling to larger images | Assuming the same filter sizes will "just work" on say 224×224 images without adjusting depth/pooling | Recompute receptive field arithmetic (`RF_out = RF_in + (k-1)·stride_so_far`) whenever input resolution changes materially |
| Confusing average pooling's smoothing with max pooling's sparsity | Average pooling blends all activations in a window (never discards outliers), max pooling keeps only the single strongest — they produce different invariances and gradients | Choose deliberately: average pooling for smoother features, max pooling when sharp peak detection matters (a choice AlexNet later revisited) |
Pioneers & history
Origins · 1 minYann LeCun and how this was discovered
Where this matters
LeNet's blueprint is the direct ancestor of essentially every modern convolutional vision model. AlexNet and Deep CNNs scaled the same conv-pool-FC skeleton to much larger images and many more layers, adding ReLU activations, dropout, and GPU training — but kept the alternating feature-extraction-then-classification structure intact. Later, ResNet addressed the problem of training very deep versions of this blueprint by adding skip connections, again without abandoning the underlying convolution-pooling philosophy. Object detectors (Faster R-CNN, YOLO), semantic segmentation networks (U-Net, which explicitly mirrors an encoder built from this blueprint), and even the convolutional stems still used in some hybrid vision transformers all trace their lineage to LeNet's decision to interleave local, weight-shared feature extraction with progressive downsampling. Beyond vision, the same locality-plus-sharing principle underlies 1D convolutional networks for audio and time series, and the general idea of "learn a small shared filter, slide it, then pool" recurs anywhere translation-invariant structure exists in the data. Industrially, optical character recognition, document processing pipelines, and the check-reading systems descended from LeNet remain in production at banks today, a direct commercial legacy of this exact architecture.
Ask the tutor: Why does the C3 layer's partial connectivity matter for symmetry-breaking, and what happens if you make it fully connected instead?
Why does LeNet-5 use average pooling instead of max pooling, and does it matter?
The original LeNet-5 used average pooling with a learned per-channel scale and bias applied after averaging, which was the natural choice at the time — it produces smooth, differentiable downsampling and blends all activations in a window rather than discarding all but the largest. Max pooling, popularized later by AlexNet and subsequent architectures, instead keeps only the strongest activation in each window, which tends to preserve sharp, discriminative features and often works better empirically for tasks where the presence of a strong feature matters more than its average strength. Both are valid choices and neither is "correct" in an absolute sense; they represent different inductive biases about what to keep when downsampling. Modern reimplementations of LeNet sometimes swap in max pooling for convenience, but this changes the architecture from its historical form and should be reported explicitly when comparing results.
How is LeNet-5 different from a modern CNN like AlexNet or ResNet?
LeNet-5 is shallow (roughly 7 layers with weights) and small (about 60,000 parameters), trained on 32×32 grayscale digit images with tanh activations and average pooling. AlexNet, by contrast, is deeper, uses ReLU activations (avoiding the vanishing-gradient-prone saturating tanh), operates on 224×224 color images, adds dropout for regularization, and was trained on GPUs across a dataset (ImageNet) many orders of magnitude larger and more diverse than MNIST. ResNet goes further, adding skip connections that let networks reach over a hundred layers without the optimization difficulties that plague plain deep stacks. All three, however, share the same underlying blueprint: alternating convolution/pooling stages for feature extraction followed by fully-connected layers for classification.
Why did the original LeNet-5 paper deliberately avoid full connectivity between S2 and C3?
The authors wanted the 16 feature maps in C3 to learn different combinations of the 6 feature maps produced by S2, rather than all 16 converging toward similar or redundant representations. By restricting each C3 map to see only a hand-designed subset (3, 4, or all 6) of the S2 maps, they forced diversity in what each output channel could "see" and reduced the parameter count from 2,416 to 1,516 in that layer. This kind of hand-engineered sparsity pattern was practical in 1998 given limited compute, but was largely abandoned in later architectures once GPU throughput made full connectivity computationally cheap and let the network discover useful combinations on its own via gradient descent rather than a manually designed table.
What role does pooling play beyond just making the network faster?
Pooling serves two purposes simultaneously: computational efficiency and statistical robustness. Computationally, halving spatial resolution at each stage quarters the number of activations, and since later convolution layers' cost scales with `H'·W'`, this substantially reduces the total multiply-add count in the deeper layers. Statistically, pooling introduces tolerance to small translations — if a stroke shifts by one pixel due to handwriting variation, an average or max over a 2×2 window is far less sensitive to that shift than the raw pixel value would be. This connects directly to the broader concept of building invariances into an architecture rather than requiring the training data to contain every possible shifted example, which is the same motivation behind Convolutional Layers and Parameter Sharing more generally.
Is LeNet-5 still useful today, or is it purely of historical interest?
LeNet-5 remains an excellent teaching architecture and a genuinely practical baseline for small, low-resolution classification tasks such as digit recognition, simple character recognition, or other problems where compute and data are limited. Its tiny footprint (~60K parameters, a few hundred thousand multiply-adds per image) makes it suitable for embedded or resource-constrained deployment where a modern deep CNN would be overkill. It is not competitive with modern architectures on complex, high-resolution, many-class problems like ImageNet, where depth, more sophisticated regularization, and architectural innovations like residual connections give large accuracy gains that a shallow LeNet-style network cannot match.
How do I compute the receptive field of a unit deep in a LeNet-style network by hand?
Track it layer by layer using the recurrence `RF_out = RF_in + (k − 1) · stride_product_so_far`, where `k` is the current layer's kernel size and `stride_product_so_far` is the product of all strides applied in earlier layers. Starting from `RF = 1` at the input: after C1 (`k=5`, stride 1), `RF = 1 + 4·1 = 5`. After S2 (`k=2`, stride 2), `RF = 5 + 1·1 = 6`, and the cumulative stride becomes 2. After C3 (`k=5`, stride 1, but applied to the pooled map so its effective stride contribution is scaled by the cumulative stride of 2), `RF = 6 + 4·2 = 14`. After S4 (`k=2`, stride 2), `RF = 14 + 1·2 = 16`, cumulative stride becomes 4. After C5 (`k=5`, stride 1), `RF = 16 + 4·4 = 32` — exactly the full input size, confirming that a C5 unit "sees" the entire 32×32 image, which is exactly the architectural intent.
Why did LeNet pad its input to 32×32 instead of using the native 28×28 MNIST size?
The choice of 32×32 was deliberate so that distinctive features like stroke endpoints or corners could appear near the center of the receptive field of the first-layer kernels even when they are near the physical edge of the original digit, rather than being clipped by a convolution with no padding. It also produces the clean 28→14→10→5 cascade of shapes across the network without any layer needing padding beyond the input, which simplifies the architecture and keeps every internal convolution a strict "valid" (unpadded) convolution. This is largely an engineering convenience specific to LeNet-5's exact shape choices; the same underlying architecture idea works with different input sizes and padding schemes as long as receptive fields and downsampling factors are consistently recomputed.