EfficientNet: Complete Guide — Compound Scaling of Neural Networks

Summary — EfficientNet, proposed by Tan and Le from Google Brain in 2019, is a family of vision models that uses a systematic scaling approach to achieve state-of-the-art image classification with far fewer parameters and FLOPs than previous architectures. Instead of arbitrarily scaling a single dimension (depth, width, or resolution), EfficientNet applies compound scaling: a mathematically optimized balance of all three dimensions simultaneously. The result: EfficientNet-B7 achieves 84.3% top-1 accuracy on ImageNet with 8.4x fewer parameters than GPipe, the most accurate architecture of its time.

Mathematical Principles

1. The problem: how do you scale a neural network?

Before EfficientNet, there were three scaling approaches:
– Depth (ResNet): adding layers (from 18 to 152)
– Width (Wide ResNet): increasing channels
– Resolution (GPipe): increasing input image size

Each method improves accuracy but with diminishing returns. More importantly, scaling one dimension without the others wastes resources: a very deep network with few channels lacks the capacity to process features, and a wide network with few layers cannot capture complex patterns.

2. Compound scaling

EfficientNet proposes scaling all three dimensions together with optimized coefficients:

depth: d = alpha^phi
width: w = beta^phi
resolution: r = gamma^phi

s.t.  alpha · beta² · gamma² ≈ 2,  alpha >= 1, beta >= 1, gamma >= 1

Where phi is a user coefficient (resource budget) and alpha, beta, gamma are constants determined by grid search on a small model. The constraint alpha · beta² · gamma² ≈ 2 comes from the fact that doubling depth doubles the number of operations (FLOPs), while doubling width or resolution quadruples FLOPs.

For the base B0 model, the coefficients found are:

alpha = 1.2, beta = 1.1, gamma = 1.15

These coefficients are then applied to generate B1 through B7:

B0: phi = 0   → 5.3M params, 76.3% top-1
B1: phi = 1.0 → 7.8M params, 78.4% top-1
B2: phi = 2.0 → 9.2M params, 79.8% top-1
B3: phi = 3.0 → 12M params,  80.9% top-1
B4: phi = 4.0 → 19M params,  82.4% top-1
B5: phi = 5.0 → 30M params,  83.0% top-1
B6: phi = 6.0 → 43M params,  83.6% top-1
B7: phi = 7.0 → 66M params,  84.3% top-1

3. The MBConv block (Mobile Inverted Bottleneck Convolution)

Each EfficientNet block is an MBConv with squeeze-and-excitation (inspired by MobileNetV2):

1. 1×1 Expansion: increases channels by an expansion factor (typically 6)
2. 3×3 or 5×5 Depthwise: per-channel convolution (far fewer parameters than a standard conv)
3. Squeeze-and-Excitation: global average pooling → FC → sigmoid → per-channel weights
4. 1×1 Projection: reduces channels to the output count

The residual connection is added only when input and output channels are identical.

4. Detailed architecture of B0

Step 1: Conv 3x3, stride=2, 32 channels
Step 2: MBConv1, k=3x3, 16 channels, 1 layer
Step 3: MBConv6, k=3x3, 24 channels, 2 layers
Step 4: MBConv6, k=5x5, 40 channels, 2 layers
Step 5: MBConv6, k=3x3, 80 channels, 3 layers
Step 6: MBConv6, k=5x5, 112 channels, 3 layers
Step 7: MBConv6, k=5x5, 192 channels, 4 layers
Step 8: MBConv6, k=3x3, 320 channels, 1 layer
Step 9: Conv 1x1, 1280 channels → GAP → FC 1000 classes

Intuition

Scaling a deep learning model is like preparing a meal for more guests.

If you have 10 guests instead of 5, you don’t JUST:
– Increase the quantity of food (more layers = deeper)
– Improve the quality of ingredients (more channels = wider)
– Use bigger plates (more resolution = more detail)

You do all three, in proportion. EfficientNet’s compound scaling is exactly that balanced dosage. And the base recipe (B0) was tested and optimized to be the best starting point before increasing the portions.

Python Implementation

1. MBConv Block with Squeeze-and-Excitation

import torch
import torch.nn as nn
import torch.nn.functional as F
import math


class SEBlock(nn.Module):
    """Squeeze-and-Excitation: recalibrates channels by importance."""
    def __init__(self, channels, se_ratio=0.25):
        super().__init__()
        se_channels = max(1, int(channels * se_ratio))
        self.pool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Sequential(
            nn.Linear(channels, se_channels), nn.SiLU(inplace=True),
            nn.Linear(se_channels, channels), nn.Sigmoid()
        )

    def forward(self, x):
        b, c, h, w = x.shape
        y = self.pool(x).view(b, c)
        y = self.fc(y).view(b, c, 1, 1)
        return x * y


class MBConv(nn.Module):
    """MBConv Block with expansion, depthwise, and squeeze-excitation."""
    def __init__(self, in_channels, out_channels, kernel_size, stride, expansion, se_ratio=0.25):
        super().__init__()
        self.use_residual = (stride == 1) and (in_channels == out_channels)
        expanded = in_channels * expansion

        layers = []
        # 1x1 Expansion
        if expansion != 1:
            layers.extend([nn.Conv2d(in_channels, expanded, 1, bias=False),
                          nn.BatchNorm2d(expanded), nn.SiLU(inplace=True)])

        # Depthwise convolution
        padding = (kernel_size - 1) // 2
        layers.extend([nn.Conv2d(expanded, expanded, kernel_size, stride, padding,
                                groups=expanded, bias=False),
                      nn.BatchNorm2d(expanded), nn.SiLU(inplace=True)])

        # Squeeze-and-Excitation
        layers.append(SEBlock(expanded, se_ratio))

        # 1x1 Projection
        layers.extend([nn.Conv2d(expanded, out_channels, 1, bias=False),
                      nn.BatchNorm2d(out_channels)])

        self.block = nn.Sequential(*layers)

    def forward(self, x):
        out = self.block(x)
        if self.use_residual:
            out = out + x
        return out


### 2. EfficientNet-B0 from scratch


def efficientnet_b0_cfg():
    """EfficientNet-B0 configuration."""
    return [
        # repeat, out, kernel, stride, expand_ratio
        (1,  16,  3,  1,  1),
        (2,  24,  3,  2,  6),
        (2,  40,  5,  2,  6),
        (3,  80,  3,  2,  6),
        (3,  112, 5,  1,  6),
        (4,  192, 5,  2,  6),
        (1,  320, 3,  1,  6),
    ]


def get_width_depth_coeff(phi, base_w=1.0, base_d=1.0):
    """Calculates the scaling coefficients for a given phi (B0: phi=0, B7: phi=7)."""
    alpha, beta, gamma = 1.2, 1.1, 1.15
    w = int(beta ** phi * 32)  # base 32 channels
    d = alpha ** phi  # depth multiplier
    r = int(gamma ** phi * 224)  # input resolution
    return max(8, int(w * base_w)), max(1.0, base_d * d), r


def create_efficientnet(cfg, input_channels=3, num_classes=1000, drop_connect_rate=0.2):
    layers = [nn.Conv2d(input_channels, 32, 3, 2, 1, bias=False), nn.BatchNorm2d(32), nn.SiLU(inplace=True)]

    in_c = 32
    for repeat, out_c, kernel, stride, expand in cfg:
        for i in range(repeat):
            s = stride if i == 0 else 1
            layers.append(MBConv(in_c, out_c, kernel, s, expand))
            in_c = out_c

    layers.extend([nn.Conv2d(in_c, 1280, 1, bias=False), nn.BatchNorm2d(1280), nn.SiLU(inplace=True),
                   nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Dropout(0.2), nn.Linear(1280, num_classes)])

    return nn.Sequential(*layers)

# Create EfficientNet-B0
model_b0 = create_efficientnet(efficientnet_b0_cfg())
params_b0 = sum(p.numel() for p in model_b0.parameters())
print(f'EfficientNet-B0: {params_b0/1e6:.1f}M parameters')

# Create EfficientNet-B7
w_b7, d_b7, r_b7 = get_width_depth_coeff(7)
print(f'EfficientNet-B7: width={w_b7}, depth_mult={d_b7:.2f}, resolution={r_b7}')

# Training
optimizer = torch.optim.Adam(model_b0.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()
for images, labels in train_loader:
    preds = model_b0(images)
    loss = criterion(preds, labels)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Hyperparameters

Advantages

1. Efficiency: The same level of accuracy with 10x fewer FLOPs than previous architectures.
2. Systematic scalability: The compound scaling method applies to any architecture, not just EfficientNet.
3. Complete family: 8 models (B0-B7) covering all resource constraints, from mobile to server.
4. Transfer learning: EfficientNet pretrained weights are among the best available vision backbones.

Limitations

1. Large models are slow: B7 is very accurate but slow at inference. For real-time applications, B0-B3 are more suitable.
2. Fixed architecture: The MBConv blocks are not automatically optimized — they are manually chosen as the backbone.
3. Less popular: Fewer frameworks support EfficientNet compared to ResNet, although the situation is improving.

4 Concrete Use Cases

1. Industrial classification on edge devices

EfficientNet-B0 and B1, with their 5-8M parameters, are perfect for deployment on embedded devices (Raspberry Pi, smartphones) for factory product classification.

2. Disease detection on X-rays

EfficientNet-B3 pretrained on ImageNet is fine-tuned on chest X-rays to detect pneumonia, achieving an AUC of 0.98 with far less training data than an equivalent ResNet.

3. E-commerce image search

Features extracted from EfficientNet (layer before the classifier) serve as embeddings for similar image search across millions of product catalogs.

4. Backbone for detection/segmentation

EfficientNet replaces ResNet as the backbone in detection (RetinaNet, EfficientDet) and segmentation architectures, offering a better accuracy/speed tradeoff.

Conclusion

EfficientNet proved that a systematic, mathematical approach to scaling can outperform the ad hoc scaling that dominated research. Compound scaling is now a standard practice, and the B0-B7 family covers a range of applications without equal.

The successors (EfficientNetV2 with Fused-MBConv and training-aware scaling) have further improved these results, but the fundamental principle remains the same: balance depth, width, and resolution in a calculated way, not an arbitrary one.

See Also

• Path Sum in Python: Four Essential Methods to Master the Algorithm
• Demystifying the Champernowne Constant in Python: Complete Guide and Practical Tutorial