Convolutional Layers and Parameter Sharing

A fully connected layer maps every input unit to every output unit with an independent weight, giving a weight matrix of shape $(\text{output size}) \times (\text{input size})$. For image-sized inputs this is enormous, and it discards a structural fact about images: statistics are roughly stationary across space — an edge detector useful near pixel $(10, 10)$ is usually just as useful near pixel $(200, 150)$. A convolutional layer encodes this fact through two coupled design choices:

Local connectivity. Each output unit connects only to a small spatial patch of the input — a $k \times k$ window — rather than to the entire input. This window is called the receptive field of that unit with respect to that layer. If the input has $C_{in}$ channels, the connectivity of one output unit spans $k \times k \times C_{in}$ input values, not $H \times W \times C_{in}$.

Parameter sharing (weight tying). Rather than learning a separate $k \times k \times C_{in}$ weight tensor for every output spatial position, the layer learns one such tensor — called a kernel or filter — and reuses it at all positions, sliding it across the input (the convolution/cross-correlation operation itself, covered in The Convolution Operation). The output at position $(i,j)$ for output channel $c$ is

Note that $W$ has no dependence on $i,j$ — that is the sharing. Contrast this with a locally connected (unshared) layer, which keeps the small receptive field but gives every output position its own private kernel; that intermediate design has the connectivity benefit but not the parameter-economy or equivariance benefit. A convolutional layer typically learns $C_{out}$ independent filters at once, each of shape $k \times k \times C_{in}$, producing $C_{out}$ output channels (also called feature maps). Each channel specializes: one might respond to vertical edges, another to a particular color contrast, another to a corner pattern. Because filters operate over all input channels jointly but produce one output channel each, the total number of learnable weights in a layer is

completely independent of the spatial size $H \times W$ of the input. This is the crux of the efficiency argument: a $3\times3$ convolution with 64 input and 64 output channels has $3\cdot3\cdot64\cdot64 + 64 = 36{,}928$ parameters whether the image is $32\times32$ or $2048\times2048$.

The forward pass of a convolutional layer is: for every output spatial position and every output channel, gather the corresponding local patch across all input channels, take the dot product with that channel's shared kernel, add the bias, and (usually) apply a nonlinearity afterward. Padding and stride control the output size and how densely the kernel samples the input; both are properties of the sliding mechanism from The Convolution Operation, reused here unchanged. ``` function CONV2D_LAYER(X, W, b, stride, padding):

# X: input tensor of shape (H, W, C_in)

# W: kernel tensor of shape (k, k, C_in, C_out) -- SHARED across all (i, j)

# b: bias vector of shape (C_out,)

# stride: integer step size

# padding: integer amount of zero-padding on each spatial side X_padded <- ZERO_PAD(X, padding) H_p, W_p, C_in <- shape(X_padded) k, k, C_in, C_out <- shape(W) H_out <- floor((H_p - k) / stride) + 1 W_out <- floor((W_p - k) / stride) + 1 Y <- ZEROS(H_out, W_out, C_out) for i in 0 .. H_out - 1: for j in 0 .. W_out - 1: row_start <- i stride col_start <- j stride

# extract the local receptive field, all input channels patch <- X_padded[row_start : row_start + k, col_start : col_start + k, 0 : C_in] for c_out in 0 .. C_out - 1:

# the SAME W[:, :, :, c_out] is used at every (i, j) Y[i, j, c_out] <- SUM(patch W[:, :, :, c_out]) + b[c_out] return Y # shape (H_out, W_out, C_out) ``` Two invariants are worth stating explicitly, because they are exactly what makes this a convolutional layer rather than an arbitrary sparse layer: 1. <strong>Sparsity invariant:</strong> for fixed $(i,j,c_{out})$, the output depends on at most $k \times k \times C_{in}$ input entries — a constant, independent of $H, W$. 2. <strong>Sharing invariant:</strong> the weight tensor $W[:,:,:,c_{out}]$ used to compute $Y[i,j,c_{out}]$ is literally the same array (same memory, same gradient accumulator) for every $(i,j)$. During backpropagation, gradients from every spatial position flow into and accumulate on the same* shared parameters. That last point deserves elaboration since it is the mechanical reason sharing works with ordinary gradient descent: if $L$ is the loss, then

a sum over all spatial positions where that weight was used. The shared kernel is updated by an average (well, sum) of evidence gathered across the entire image, rather than by evidence from a single location — this is precisely why convolutional layers can learn good filters from far fewer examples than a fully connected layer would need: every image patch, not every whole image, becomes a training signal for the shared weights.

Parameter count. As derived above, a convolutional layer with kernel size $k$, $C_{in}$ input channels, and $C_{out}$ output channels has

independent of spatial size $H, W$. Compare to a fully connected layer mapping the same $H \times W \times C_{in}$ input to an $H' \times W' \times C_{out}$ output: it needs

which scales with the product of input and output spatial resolutions — for even modest images this is many orders of magnitude larger. This is not a heuristic gain; it is a direct consequence of the two invariants (sparsity + sharing) stated earlier.

Time complexity (forward pass). Computing one output value costs $O(k^2 C_{in})$ multiply-adds. There are $H_{out} \times W_{out} \times C_{out}$ output values, so the naive direct-convolution forward pass costs

With "same" padding and stride 1, $H_{out}\approx H$, $W_{out}\approx W$, giving $O(HWC_{in}C_{out}k^2)$ — linear in the number of pixels and in the number of channel pairs, and quadratic only in the (typically small, e.g. $3$ or $5$) kernel size. In practice this is implemented via im2col + matrix multiply, or via FFT-based / Winograd convolution algorithms for further constant-factor speedups, but the asymptotic FLOP count above is what any implementation must compute, since it equals the number of scalar multiply-adds mathematically required.

Space complexity. Parameters require $O(k^2 C_{in} C_{out})$ memory — independent of image size, as emphasized above. Activations (the output feature map, needed for the backward pass) require $O(H_{out} W_{out} C_{out})$ memory, which does scale with image size, same as for a fully connected layer's output — sharing reduces the weight memory, not the activation memory.

Backward pass. Gradients with respect to the input and the shared kernel are themselves expressible as convolutions/cross-correlations of the upstream gradient with (flipped/transposed) kernels, so the backward pass has the same asymptotic cost as the forward pass, $O(HWC_{in}C_{out}k^2)$, and requires no additional big-O memory beyond storing one gradient tensor per parameter and per activation.

Correctness argument — why sharing doesn't cost expressiveness for this problem class. The concern with tying weights together is: are we throwing away necessary flexibility? The argument for why not, for image-like data, rests on translation equivariance. Define the shift operator $S_\Delta$ that translates an image by $\Delta$ pixels. A function $f$ is translation-equivariant if

for all shifts $\Delta$ (ignoring boundary effects). One can verify directly from the convolution formula that a shared-kernel convolutional layer satisfies this exactly: shifting the input by $\Delta$ shifts every extracted patch by $\Delta$, and since the same kernel is applied to the shifted patch as would have been applied to the original patch at its new location, the output shifts by the same $\Delta$. A fully connected layer, or a locally-connected (unshared) layer, does not have this property in general — nothing forces the weights at output position $(i,j)$ to relate to the weights at $(i+\Delta, j+\Delta)$. Because natural images exhibit approximate translation invariance of local statistics (an edge is an edge wherever it appears), building translation equivariance into the layer is a correct prior, not an arbitrary restriction — it matches the true structure of the hypothesis class we want to search over. The cost is real but bounded: sharing removes the ability to have different detectors at different absolute positions (position-specific behavior), which is why architectures that need that (e.g. detecting "top of image" vs "bottom of image" features differently) sometimes add positional encodings or use non-shared layers at select points — but for the vast majority of low/mid-level image features, the shared hypothesis class contains the right answer, and restricting to it only helps sample efficiency, never asymptotic accuracy.

Take a small but non-trivial trace: a $5\times5$ single-channel input, a $3\times3$ kernel, stride $1$, no padding.

This $W$ is a classic vertical-edge detector (large left column minus large right column). Output size: $H_{out} = W_{out} = \lfloor(5-3)/1\rfloor + 1 = 3$, so $Y$ is $3\times3$. Compute $Y[0,0]$ using the top-left $3\times3$ patch:

Compute $Y[0,1]$, sliding the same kernel one column right, over patch $\begin{bmatrix}2&0&1\\1&2&1\\0&1&3\end{bmatrix}$:

Continuing this process (same $W$, no new parameters, just a new patch each time) for all 9 output positions gives:

The table above required 9 kernel applications but only 9 weights total (reused 9 times, not 81 separate weights) — this is the parameter-sharing saving made concrete: a locally-connected (unshared) layer solving the identical connectivity pattern would need $9 \times 9 = 81$ independent weights, one $3\times3$ kernel per output cell, for this tiny example alone; the gap widens combinatorially as images grow. Now suppose $C_{out} = 2$: add a second $3\times3$ kernel, say a horizontal-edge detector $W' = W^{\top}$. The layer now produces $Y$ of shape $3\times3\times2$: channel 0 is the table above (vertical edges), channel 1 is the analogous horizontal-edge response, computed with the second shared kernel applied at all 9 positions. Total parameters for this two-filter layer: $3\cdot3\cdot1\cdot2 + 2 = 20$, regardless of whether the input were $5\times5$ or $500\times500$ — only the number of output positions (and hence compute) grows with image size, not the parameter count.

Convolutional layers descend from a line of work connecting biological vision, hierarchical feature extraction, and neural network learning rules. The core insight — local receptive fields whose weights are shared and reused across the visual field — was first cast into a trainable neural network architecture by Yann LeCun and collaborators, building on Kunihiko Fukushima's earlier Neocognitron, which itself drew on Hubel and Wiesel's neurophysiological findings about simple and complex cells in the visual cortex.

Convolutional layers with parameter sharing are the computational backbone of essentially every production image, video, and much of audio and biomedical signal processing today. Image classifiers, object detectors (e.g. region-proposal and single-shot detectors), semantic segmentation networks, and image generators (GANs, diffusion model U-Nets) all use stacks of convolutional layers as their core feature extractors, precisely because the parameter count stays bounded regardless of image resolution, letting the same architecture scale to megapixel inputs. Medical imaging systems (tumor detection in CT/MRI scans), autonomous-vehicle perception stacks (lane and pedestrian detection from camera feeds), satellite and remote-sensing analysis, and on-device mobile vision models (which must fit in tens of megabytes) all depend on the parameter-sharing efficiency argument made precise in this article. Beyond vision, 1-D convolutions with shared weights are used for audio waveform and spectrogram processing (speech recognition, music generation), and the same local-connectivity-plus-sharing idea, generalized, underlies graph convolutional networks. Even the attention mechanisms in Transformers can be viewed as a data-dependent relaxation of the fixed spatial weight-sharing pattern that convolutions impose by design — understanding convolutional inductive bias is a prerequisite for appreciating what later architectures deliberately give up or keep.

Ask the tutor: How does the receptive-field growth formula change if some layers use stride > 1 or dilation instead of plain stride-1 convolutions?