InceptionNet and GoogLeNet

Naive Inception Module

Consider an input feature map $X$ of shape $(H, W, C_{in})$. The naive Inception module applies four branches and concatenates along the channel axis:

Explanation: Four transformations of the same input are stacked depth-wise, giving the next layer simultaneous access to 1×1, 3×3, 5×5, and pooled information. Here $W_k * X$ denotes a convolution with kernel size $k \times k$ (with appropriate padding to preserve spatial dimensions), and $\text{Concat}$ is concatenation along the channel axis.

Computational Cost of Naive Module

Let the input have $C_{in}$ channels and the output of each branch have $C_k$ channels. The multiply-add (FLOP) cost of a single $k \times k$ convolution over an $H \times W$ map is:

Explanation: Every output pixel in every output channel requires a dot product of depth $C_{in}$ over a $k \times k$ spatial window, giving this quadratic dependence on kernel size. For $k=5$, $C_{in}=256$, $C_5=128$, $H=W=28$: this is $28^2 \times 256 \times 128 \times 25 \approx 638\text{M}$ FLOPs for a single branch in a single layer — far too expensive to stack.

Bottleneck with 1×1 Convolution

The reduction Inception module inserts a 1×1 convolution of output depth $C_r \ll C_{in}$ before each expensive branch:

Explanation: By first projecting from $C_{in}$ channels down to $C_r$ channels, the expensive $k \times k$ convolution operates on a much smaller channel space, reducing total FLOPs dramatically. For the same example with $C_r=32$: $28^2 \times 256 \times 32 + 28^2 \times 32 \times 128 \times 25 \approx 7.2\text{M} + 71.7\text{M} \approx 79\text{M}$ FLOPs — an 8× reduction for the 5×5 branch alone.

Auxiliary Classifiers and the Loss

GoogLeNet adds two intermediate classification heads at intermediate layers during training. The total training loss is:

Explanation: Auxiliary losses inject gradient directly into middle layers, combating vanishing gradients in a 22-layer network trained before batch normalization was standard practice. $\mathcal{L}_{\text{main}}$ is the softmax cross-entropy at the final layer; $\mathcal{L}_{\text{aux},j}$ are softmax cross-entropies at intermediate layers; $\lambda=0.3$ is a discount weight so auxiliary losses guide but do not dominate. At inference, the auxiliary heads are discarded entirely.

Inception-v2/v3: Factorized Convolutions

Inception-v3 (Szegedy et al., 2016) further decomposes spatial convolutions. A $k \times k$ convolution is replaced by a $1 \times k$ followed by a $k \times 1$:

Explanation: Factorizing an $n \times n$ convolution into two asymmetric convolutions reduces the FLOP count from $k^2$ to $2k$ per output element, a saving of roughly $\frac{k}{2}$ for large kernels. For $k=7$ this is a $3.5\times$ FLOP reduction; Inception-v3 achieves 78.8% top-1 on ImageNet at 25M parameters. ---

PyTorch Implementation — Inception Module

```python import torch import torch.nn as nn import torch.nn.functional as F class InceptionModule(nn.Module): """ Inception module with dimensionality reduction (as in GoogLeNet). Args: in_channels: number of input feature map channels out_1x1: output channels for the 1×1 branch red_3x3: channel reduction before 3×3 branch out_3x3: output channels for the 3×3 branch red_5x5: channel reduction before 5×5 branch out_5x5: output channels for the 5×5 branch out_pool: output channels for the max-pool branch (1×1 after pool) """ def __init__(self, in_channels, out_1x1, red_3x3, out_3x3, red_5x5, out_5x5, out_pool): super().__init__()

# Branch 1: just 1×1 conv self.branch1 = nn.Sequential( nn.Conv2d(in_channels, out_1x1, kernel_size=1), nn.BatchNorm2d(out_1x1), nn.ReLU(inplace=True), )

# Branch 2: 1×1 bottleneck → 3×3 conv self.branch2 = nn.Sequential( nn.Conv2d(in_channels, red_3x3, kernel_size=1), nn.BatchNorm2d(red_3x3), nn.ReLU(inplace=True), nn.Conv2d(red_3x3, out_3x3, kernel_size=3, padding=1), nn.BatchNorm2d(out_3x3), nn.ReLU(inplace=True), )

# Branch 3: 1×1 bottleneck → 5×5 conv self.branch3 = nn.Sequential( nn.Conv2d(in_channels, red_5x5, kernel_size=1), nn.BatchNorm2d(red_5x5), nn.ReLU(inplace=True), nn.Conv2d(red_5x5, out_5x5, kernel_size=5, padding=2), nn.BatchNorm2d(out_5x5), nn.ReLU(inplace=True), )

# Branch 4: 3×3 max-pool → 1×1 projection self.branch4 = nn.Sequential( nn.MaxPool2d(kernel_size=3, stride=1, padding=1), nn.Conv2d(in_channels, out_pool, kernel_size=1), nn.BatchNorm2d(out_pool), nn.ReLU(inplace=True), ) def forward(self, x): b1 = self.branch1(x) b2 = self.branch2(x) b3 = self.branch3(x) b4 = self.branch4(x)

# Concatenate along channel dimension return torch.cat([b1, b2, b3, b4], dim=1)

# Quick sanity check if __name__ == "__main__": module = InceptionModule( in_channels=192, out_1x1=64, red_3x3=96, out_3x3=128, red_5x5=16, out_5x5=32, out_pool=32, ) x = torch.randn(2, 192, 28, 28) # batch=2, C=192, H=W=28 y = module(x)

# Output channels = 64 + 128 + 32 + 32 = 256 print(y.shape) # → torch.Size([2, 256, 28, 28]) ```