EfficientNet and Compound Scaling

EfficientNet is a family of convolutional neural networks introduced by Tan and Le (2019) that achieves state-of-the-art accuracy with dramatically fewer parameters by systematically scaling network width, depth, and input resolution together using a principled method called compound scaling. It matters because it shattered the assumption that bigger models must be wasteful — EfficientNet-B7 matched the accuracy of GPT-era vision giants while using 8.4× fewer parameters than the best ResNet variant. The counterintuitive fact: naively making a network deeper alone often yields less improvement than a balanced, simultaneous increase across all three dimensions, even when the total FLOP budget is identical.

The core insight

The central insight of EfficientNet is that width, depth, and resolution are not independent knobs — they are coupled, and scaling one without the others creates bottlenecks that waste capacity. If you double the depth of a network processing small images, the extra layers have nothing richer to learn from; you must also increase resolution so there is more spatial detail to exploit, and increase width so channels can represent the richer feature space. Compound scaling formalizes this coupling: find the optimal ratio between the three dimensions once (on a tiny baseline model), then scale all three together as a single composite exponent of your compute budget.

How It Works

The Compound Scaling Formulation

Given a baseline network $\mathcal{N}$ , EfficientNet parameterises the three scaling dimensions as:

\quad d = \alpha^{\phi}, \quad w = \beta^{\phi}, \quad r = \gamma^{\phi}

Explanation: Depth $d$ , width $w$ , and resolution $r$ each grow as a power of the compound coefficient $\phi$ , where $\alpha$ , $\beta$ , $\gamma$ are fixed ratios and $\phi$ is the single user-controlled knob for compute budget.

Symbol guide:

$d$ — depth multiplier (scales number of layers)
$w$ — width multiplier (scales number of channels per layer)
$r$ — resolution multiplier (scales input image spatial dimensions)
$\alpha, \beta, \gamma \geq 1$ — per-dimension scaling ratios found by grid search
$\phi \geq 0$ — compound coefficient; $\phi=0$ gives the baseline B0

The total FLOP scaling can be approximated. Since FLOPs grow linearly with depth, quadratically with resolution, and quadratically with width for convolutions:

\quad \text{FLOPs} \propto d \cdot w^{2} \cdot r^{2} = \alpha^{\phi} \cdot \beta^{2\phi} \cdot \gamma^{2\phi} = \left(\alpha \cdot \beta^{2} \cdot \gamma^{2}\right)^{\phi}

Explanation: Total computation scales as the $\phi$ -th power of the compound base $\alpha\beta^2\gamma^2$ , so constraining that base to approximately 2 means every unit increase in $\phi$ roughly doubles FLOPs — a clean, predictable budget. The constraint imposed during the grid search is:

\quad \alpha \cdot \beta^{2} \cdot \gamma^{2} \approx 2

Explanation: This constraint fixes the FLOP multiplier per $\phi$ -step to ≈2, so each EfficientNet variant (B0→B7) costs roughly $2^\phi$ times the baseline compute — making the model family a geometric sequence in compute. For EfficientNet-B0, the grid search found $\alpha = 1.2$ , $\beta = 1.1$ , $\gamma = 1.15$ , satisfying $1.2 \times 1.1^2 \times 1.15^2 \approx 1.2 \times 1.21 \times 1.3225 \approx 1.92 \approx 2$ .

The Neural Architecture Search Baseline

EfficientNet-B0 is not a hand-designed network — it was found via Neural Architecture Search using a multi-objective reward that balances accuracy and FLOP count:

\quad \text{Reward}(m, a) = \text{ACC}(m)^w \cdot \left[\frac{\text{FLOPs}(m)}{T}\right]^{v}

Explanation: The NAS reward multiplies accuracy raised to a positive power $w$ by a FLOP penalty term, steering the search toward Pareto-optimal models that are simultaneously accurate and efficient.

Symbol guide:

$\text{ACC}(m)$ — validation accuracy of candidate model $m$
$T$ — target FLOP budget
$w, v$ — hyperparameters controlling the accuracy-efficiency trade-off (in the paper, $w=0.07$ , $v=-0.07$ )

MBConv: The Building Block

The B0 baseline uses Mobile Inverted Bottleneck Convolution (MBConv) blocks from MobileNetV2: ``` Input → Expand (1×1 conv, expand_ratio × channels) → Depthwise 3×3 or 5×5 conv → SE block (Squeeze-and-Excitation) → Project (1×1 conv, back to output channels) → Skip connection (if stride=1 and channels match) ``` ```python import torch import torch.nn as nn class MBConv(nn.Module): def __init__(self, in_ch, out_ch, kernel_size=3, stride=1, expand_ratio=6, se_ratio=0.25): super().__init__() hidden = in_ch * expand_ratio self.use_skip = (stride == 1 and in_ch == out_ch) layers = []

# Expansion phase if expand_ratio != 1: layers += [ nn.Conv2d(in_ch, hidden, 1, bias=False), nn.BatchNorm2d(hidden), nn.SiLU() ]

# Depthwise conv layers += [ nn.Conv2d(hidden, hidden, kernel_size, stride=stride, padding=kernel_size//2, groups=hidden, bias=False), nn.BatchNorm2d(hidden), nn.SiLU() ]

# Squeeze-and-Excitation se_ch = max(1, int(in_ch * se_ratio)) layers += [ SqueezeExcitation(hidden, se_ch) ]

# Projection layers += [ nn.Conv2d(hidden, out_ch, 1, bias=False), nn.BatchNorm2d(out_ch) ] self.block = nn.Sequential(*layers) def forward(self, x): out = self.block(x) return x + out if self.use_skip else out

class SqueezeExcitation(nn.Module): def __init__(self, in_ch, se_ch): super().__init__() self.se = nn.Sequential( nn.AdaptiveAvgPool2d(1), nn.Conv2d(in_ch, se_ch, 1), nn.SiLU(), nn.Conv2d(se_ch, in_ch, 1), nn.Sigmoid() ) def forward(self, x): return x * self.se(x) ```

Applying Compound Scaling in Practice

```python import math def efficientnet_params(phi: float, alpha: float = 1.2, beta: float = 1.1, gamma: float = 1.15, base_depth: int = 18, base_width: int = 32, base_res: int = 224): """Return scaled (depth, width, resolution) for a given phi.""" depth = math.ceil(base_depth alpha <strong> phi) width = math.ceil(base_width beta phi) res = math.ceil(base_res * gamma ** phi / 2) * 2 # keep even return depth, width, res

# EfficientNet B0 through B7 for phi in range(8): d, w, r = efficientnet_params(phi) print(f"B{phi}: depth={d}, width={w}, resolution={r}") ```

researcher · Mingxing Tan | b. 1985 | EfficientNet Lead Author

Mingxing Tan is a Senior Staff Research Scientist at Google Brain, specialising in efficient neural network design and neural architecture search. His 2019 paper "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" (co-authored with Quoc V. Le) introduced compound scaling and achieved top-1 accuracy on ImageNet with 8× fewer parameters than contemporaries. A surprising fact: Tan was simultaneously working on EfficientDet (efficient object detection) during the same period, demonstrating that compound scaling principles generalise beyond classification to detection heads and feature pyramids.

researcher · Quoc V. Le | b. 1982 | Neural Architecture Search Pioneer

Quoc V. Le is a Distinguished Scientist at Google Brain and one of the most influential figures in modern deep learning, best known for pioneering Neural Architecture Search (NAS) and co-authoring the Word2Vec paper. His collaboration with Tan on EfficientNet grounded compound scaling in a NAS-discovered baseline, ensuring the scaling ratios were optimised for a provably strong architecture rather than a hand-crafted one. Surprisingly, Le's PhD thesis at Stanford was on unsupervised feature learning from unlabelled video — far from the efficient supervised architectures he became famous for.

Figure

Compound Scaling Diagram

Figure

EfficientNet-B0 Architecture Overview

Intuition

The Mental Model

Imagine you are a film production company and your "budget" is the total runtime of the movie. You have three ways to spend it: hire more cast members (width — more channels to represent diverse features), add more scenes (depth — more layers to learn hierarchical abstractions), or film in higher resolution (resolution — richer spatial detail per frame). A naive director might just keep adding scenes until the budget runs out, but the film becomes repetitive: scene 47 can't show anything new if the actors are too few and the camera is still standard-definition. The optimal director balances all three: a few more actors, a few more scenes, slightly better cameras — and does so in a fixed ratio each time the budget increases. This is exactly compound scaling. The