Residual Networks: Complete Guide — Residual Networks and Skip Connections

Summary — ResNet, introduced by He et al. in 2015, is a deep neural network architecture that uses skip connections to enable training of networks with over 1000 layers. By learning residual functions rather than direct transformations, ResNet solves the degradation problem where deep networks see their performance decrease with the addition of layers, not because of overfitting but because optimization becomes impossible. ResNet won ImageNet 2015 with a top-5 error of 3.57% — surpassing estimated human accuracy.

Mathematical Principle

1. The Problem: Degradation of Deep Networks

Before ResNet, VGGNet (2014) had 19 layers, GoogLeNet had 22. Intuitively, more layers = more capacity. But in practice, beyond a certain depth, the training accuracy itself begins to decrease. It’s not overfitting (the test error is also worse) — it’s that the identity function (doing nothing) is harder to learn than a non-trivial function.

2. The Solution: Residual Learning

Instead of directly learning the function H(x) that a layer should perform, ResNet learns the residual function F(x) = H(x) – x. The output becomes:

y = H(x) = F(x, {W_i}) + x

Where F(x, {W_i}) is the residual network (the convolutions) and x is the input added directly via the skip connection. If the optimal transformation is the identity, the network simply learns F(x) = 0, which is much easier than learning F(x) = x.

3. Why Gradients Flow Better

Without skip connection, the gradient in a network of L layers is:

dL/dx_1 = dL/dx_L · d x_L/d x_{L-1} · ... · d x_2/d x_1

Each term d x_{i+1}/d x_i = d sigma(W_i · x_i)/d x_i can be < 1 (vanishing) or > 1 (exploding).

With skip connection, x_L = F(x_{L-1}) + x_{L-2}, so:

dL/dx_l = dL/dx_L · (d F/d x_l + 1)

The “1” guarantees that even if d F/d x_l is tiny, the gradient passes directly through. It’s the gradient highway.

4. Bottleneck Architecture

For very deep networks (50+ layers), a 3-layer bottleneck block replaces the 2-layer block:

1x1 Conv (reduction) → 3x3 Conv (processing) → 1x1 Conv (expansion)

The 1×1 layer first reduces the channels (e.g., 256 → 64), the 3×3 works on fewer channels, and the last 1×1 restores (64 → 256). This reduces parameters by a factor of 3.5 compared to 2 convolutions 3×3 with 256 channels.

5. Pre-activation (ResNet v2)

The ResNet v2 from 2016 reverses the order: instead of Conv → BN → ReLU → Add, it proposes BN → ReLU → Conv → Add. This pre-activation allows an even cleaner gradient flow because there is no longer a non-linear transformation on the summation path.

Intuition

Imagine a production line. In a classical network, each team (layer) must completely transform the product. If one team makes a mistake, the entire chain is compromised.

In a ResNet, each team receives two things: the raw product from the previous step AND a “correction note” to add. If the product is already perfect, the team adds zero (F(x) = 0). If a small correction is enough, it just adds what’s needed.

It’s like asking a student to correct a draft rather than rewriting it from scratch — correcting is always easier than creating. And the teacher (the gradient) can communicate directly with each student through the skip connection corridor that runs through the entire building.

Python Implementation

1. ResNet-18 from Scratch with PyTorch

import torch
import torch.nn as nn
import torch.nn.functional as F


class BasicBlock(nn.Module):
    """Basic ResNet block (2 convolutions 3x3)."""
    expansion = 1

    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample

    def forward(self, x):
        identity = x
        if self.downsample:
            identity = self.downsample(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += identity  # Skip connection
        out = self.relu(out)
        return out


class Bottleneck(nn.Module):
    """Bottleneck ResNet block (1x1 → 3x3 → 1x1)."""
    expansion = 4

    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, stride, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.conv3 = nn.Conv2d(out_channels, out_channels * 4, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(out_channels * 4)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample

    def forward(self, x):
        identity = self.downsample(x) if self.downsample else x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += identity  # Skip connection
        return self.relu(out)


class ResNet(nn.Module):
    def __init__(self, block, layers, num_classes=10, base_channels=64):
        super().__init__()
        self.in_channels = base_channels
        self.conv1 = nn.Conv2d(3, base_channels, 7, 2, 3, bias=False)
        self.bn1 = nn.BatchNorm2d(base_channels)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(3, 2, 1)

        self.layer1 = self._make_layer(block, base_channels, layers[0], stride=1)
        self.layer2 = self._make_layer(block, base_channels*2, layers[1], stride=2)
        self.layer3 = self._make_layer(block, base_channels*4, layers[2], stride=2)
        self.layer4 = self._make_layer(block, base_channels*8, layers[3], stride=2)

        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(base_channels*8 * block.expansion, num_classes)

    def _make_layer(self, block, out_channels, num_blocks, stride):
        downsample = None
        if stride != 1 or self.in_channels != out_channels * block.expansion:
            downsample = nn.Sequential(
                nn.Conv2d(self.in_channels, out_channels * block.expansion, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels * block.expansion)
            )
        layers = [block(self.in_channels, out_channels, stride, downsample)]
        self.in_channels = out_channels * block.expansion
        for _ in range(1, num_blocks):
            layers.append(block(self.in_channels, out_channels))
        return nn.Sequential(*layers)

    def forward(self, x):
        x = self.relu(self.bn1(self.conv1(x)))
        x = self.maxpool(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.avgpool(x).flatten(1)
        return self.fc(x)

# Create classic architectures
def ResNet18(): return ResNet(BasicBlock, [2,2,2,2])           # 18 layers
def ResNet34(): return ResNet(BasicBlock, [3,4,6,3])           # 34 layers
def ResNet50(): return ResNet(Bottleneck, [3,4,6,3], base_channels=64)  # 50 layers

# Size comparison
models = {"ResNet18": ResNet18(), "ResNet50": ResNet50()}
for name, model in models.items():
    params = sum(p.numel() for p in model.parameters())
    print(f'{name}: {params/1e6:.1f}M parameters')

# Training on CIFAR-10
model = ResNet18()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
criterion = nn.CrossEntropyLoss()

for epoch in range(100):
    model.train()
    for images, labels in train_loader:
        preds = model(images)
        loss = criterion(preds, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    scheduler.step()
    print(f'Epoch {epoch} | LR: {scheduler.get_last_lr()[0]:.4f}')

Hyperparameters

Advantages

1. Very deep: Enables training of networks with 152, 1000+ layers, previously impossible.
2. Easy to optimize: Skip connections stabilize the gradient, making training simpler.
3. Transfer learning: Pre-trained ResNets (ImageNet) are the foundation of most vision models.
4. Extensible: The bottleneck architecture allows increasing depth without parameter explosion.

Limitations

1. Computational cost: ResNet-152 requires 60M+ parameters and weeks of training.
2. Feature redundancy: Skip connections create redundant paths that may compute the same transformations.
3. Memory intensive: Activations from all layers must be stored for the backward pass, even if some are skipped.

4 Concrete Use Cases

1. ImageNet Image Classification

ResNet-50 and ResNet-152 are reference models for ImageNet. ResNet-50 achieves 76% top-1 accuracy, and ResNet-152 achieves 78%. They serve as the starting point for any vision task.

2. Transfer Learning for Detection

Object detection models (Faster R-CNN, Mask R-CNN) use ResNet as a backbone. Features pre-trained on ImageNet are fine-tuned for detection-specific data, reducing training time by 10x.

3. Medical Models

In medical imaging (X-ray, MRI), a pre-trained ResNet is fine-tuned on a few thousand annotated images, achieving diagnostic performance comparable to human experts.

4. Embedding for Image Search

Features extracted from a ResNet at the penultimate layer (before the classifier) serve as embedding vectors for similar image search: cosine similarity between image embeddings.

Conclusion

ResNet is without question one of the most influential architectures in modern AI. By introducing skip connections, it solved a fundamental problem of deep learning and paved the way for the ultra-deep architectures that dominate today.

Even the Transformer architectures of Vision Transformers (ViT) draw inspiration from ResNet’s skip connections — they use the same shortcut mechanisms between their attention layers.

See Also

• Implementation of Cramer’s Algorithm and Determinant Calculation in Python
• Optimize Your Python Code with SOP and POS: Complete Guide for Beginners