Elastic Weight Consolidation (EWC) — Complete Guide

Summary

Elastic Weight Consolidation (EWC) is a regularization method designed to solve the problem of catastrophic forgetting in artificial neural networks. Proposed by Kirkpatrick et al. in 2017, this technique allows a model to learn multiple tasks sequentially without losing previously acquired knowledge. Unlike classical training, which overwrites the model’s weights with each new task, EWC identifies the parameters essential to previous tasks and penalizes their excessive modification. This “elastic consolidation” is directly inspired by neurobiology: in the human brain, important synapses are protected by synaptic consolidation mechanisms, enabling continuous learning while preserving old memories. EWC reproduces this behavior by estimating the importance of each weight via the diagonal Fisher Information Matrix and adding a quadratic regularization term to the loss function. The result? A model capable of accumulating knowledge cumulatively, approaching the continual learning observed in living beings.

Mathematical Principle

The theoretical foundation of EWC rests on a rigorous Bayesian framework. After learning task A, we want to learn task B without destroying acquired knowledge. In the Bayesian approach, after training on task A, the posterior distribution over the model’s weights is denoted $p(\theta | D_A, t=A)$. Ideally, training on task B should maximize the marginal likelihood:

$$\log p(D_B | t=B) = \log \int p(D_B | \theta, t=B) \cdot p(\theta | D_A, t=A) \, d\theta$$

This computation is intractable for large networks. EWC approximates it using a Laplace approximation around the optimal weights $\theta_A^$ obtained after task A. The posterior distribution is approximated by a diagonal Gaussian whose precision is given by the diagonal Fisher Information Matrix*:

$$F_i = \mathbb{E}{x,y \sim D_A} \left[ \left( \frac{\partial \log p(y|x, \theta)}{\partial \theta_i} \right)^2 \right] \right)^2$$} \quad \approx \quad \frac{1}{N} \sum_{n=1}^{N} \left( \frac{\partial \mathcal{L}_A}{\partial \theta_i

where $F_i$ measures the importance of parameter $\theta_i$ for task A. The gradients are evaluated at the optimal point $\theta_A^*$, and $N$ represents the number of samples used for the estimation.

Then, during training on task B, the loss function becomes:

$$\mathcal{L}B^{\text{EWC}}(\theta) = \mathcal{L}_B(\theta) + \frac{\lambda}{2} \sum_i F_i \cdot (\theta_i – \theta^*)^2$$

This quadratic regularization penalizes weight changes proportionally to their importance for task A. Weights with a high $F_i$ are “elastically consolidated”: they strongly resist change, but can still move a little — hence the “elastic” qualifier. Conversely, unimportant weights ($F_i \approx 0$) are free to adapt to the new task without significant constraint.

The parameter $\lambda$ controls the trade-off between plasticity and stability. A high $\lambda$ better preserves task A knowledge at the expense of performance on task B. A low $\lambda$ allows freer learning of task B but increases the risk of catastrophic forgetting.

Deep Bayesian Interpretation

The Bayesian interpretation goes further: the EWC regularization term logically corresponds to the logarithm of a Gaussian prior distribution centered on $\theta_A^$ with a variance inversely proportional to $F_i$. In other words, EWC transforms the posterior of task A into a prior* for task B. This is exactly sequential Bayesian inference, made computable through the diagonal Fisher approximation.

This diagonal approximation is crucial: the full Fisher matrix would be of size $P \times P$ where $P$ is the total number of parameters (often millions). By retaining only the diagonal ($P$ elements), EWC remains computable for realistically sized networks.

Intuition

Imagine your brain. After spending months learning piano (task A), some of your neurons have become essential: those controlling rhythm, coordination between both hands, and reading sheet music. These neural connections have been strengthened, stabilized. Now you want to learn guitar (task B). Your brain won’t erase what it knows about piano — it will reuse certain skills (sense of rhythm, dexterity) while protecting piano-specific connections. The “piano neurons” are not rigid: they can evolve a bit to accommodate guitar, but not to the point of forgetting piano. This is precisely elastic consolidation.

Apply this analogy to an artificial neural network. After training on task A, certain network weights are critical: modifying them slightly would severely degrade performance. Other weights are redundant or have little influence. EWC computes an “importance score” ($F_i$) for each weight and builds a custom penalty. Important weights receive a strong quadratic penalty (they move little). Unimportant weights receive a negligible penalty (they are free to change).

It’s comparable to learning a second language. When you’ve mastered English and start learning Spanish, you don’t lose your English. Your brain has consolidated the essential grammatical structures of the first language. Certain areas (basic vocabulary) are very stable, others (idiomatic expressions) can evolve. EWC mechanically reproduces this selectivity in neural networks.

The conceptual key is selectivity: not all synapses are equal in the face of forgetting. EWC gives each parameter the protection it deserves, no more and no less.

Python Implementation with PyTorch

Environment and dependencies

pip install torch torchvision matplotlib numpy

Step 1: Base Model

We use a simple two-layer neural network, capable of learning MNIST and then Fashion-MNIST sequentially.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, Subset
import torchvision.transforms as transforms
from torchvision.datasets import MNIST, FashionMNIST
import numpy as np
import copy
import matplotlib.pyplot as plt

class SimpleMLP(nn.Module):
    """Two-layer neural network for image classification."""

    def __init__(self, input_size=784, hidden1=400, hidden2=200, num_classes=10):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden1)
        self.fc2 = nn.Linear(hidden1, hidden2)
        self.fc3 = nn.Linear(hidden2, num_classes)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.3)

    def forward(self, x):
        x = x.view(x.size(0), -1)          # flatten 28x28 image to 784
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Step 2: Computing the Diagonal Fisher Matrix

This is the heart of EWC. After training on task A, we iterate over a data sample and accumulate the squares of the loss gradients with respect to each parameter.

def compute_fisher_diagonal(model, dataloader, device, sample_size=200):
    """
    Computes the approximate diagonal Fisher Information Matrix.

    Formula: F_i = 1/N * sum_n (dL/d theta_i)^2
    Evaluated at theta = theta*_A (optimal weights after task A)
    """
    model.eval()
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters() if p.requires_grad}
    criterion = nn.CrossEntropyLoss()

    count = 0
    for images, labels in dataloader:
        if count >= sample_size:
            break
        images = images.to(device)
        labels = labels.to(device)

        model.zero_grad()
        output = model(images)
        loss = criterion(output, labels)
        loss.backward()

        for n, p in model.named_parameters():
            if p.requires_grad and p.grad is not None:
                fisher[n] += p.grad.data.clone().pow(2)
        count += images.size(0)

    # Average over samples
    count = max(count, 1)
    for n in fisher:
        fisher[n] /= count

    return fisher

Step 3: The EWC Regularizer

def ewc_loss(model, fisher, opt_params, lambda_ewc, device):
    """
    Computes the EWC regularization term:
    (lambda / 2) * sum_i F_i * (theta_i - theta_i_A)^2
    """
    ewc_term = torch.tensor(0.0, device=device)
    for n, p in model.named_parameters():
        if n in fisher and n in opt_params:
            ewc_term += (fisher[n] * (p - opt_params[n]).pow(2)).sum()
    return (lambda_ewc / 2.0) * ewc_term

Step 4: Training Loop with EWC

def train_with_ewc(model, dataloader, fisher, opt_params,
                   lambda_ewc, num_epochs, lr, device):
    """Trains the model with EWC regularization."""
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=lr)
    history = {"loss": [], "ewc_penalty": [], "accuracy": []}

    for epoch in range(num_epochs):
        model.train()
        epoch_loss = 0.0
        epoch_ewc = 0.0
        correct = 0
        total = 0

        for images, labels in dataloader:
            images, labels = images.to(device), labels.to(device)

            optimizer.zero_grad()
            output = model(images)

            # Classification loss + EWC penalty
            class_loss = criterion(output, labels)
            ewc_term = ewc_loss(model, fisher, opt_params, lambda_ewc, device)
            loss = class_loss + ewc_term

            loss.backward()
            optimizer.step()

            epoch_loss += class_loss.item()
            epoch_ewc += ewc_term.item()
            _, predicted = output.max(1)
            total += labels.size(0)
            correct += predicted.eq(labels).sum().item()

        history["loss"].append(epoch_loss / len(dataloader))
        history["ewc_penalty"].append(epoch_ewc / len(dataloader))
        history["accuracy"].append(100.0 * correct / total)

    return history

Step 5: Sequential Training — MNIST then Fashion-MNIST

def run_sequential_experiment(lambda_ewc=5000, fisher_samples=200,
                              epochs=5, lr=1e-3, batch_size=128):
    """
    Complete protocol:
    1. Train on MNIST (task A)
    2. Compute the diagonal Fisher
    3. Train on Fashion-MNIST (task B) with or without EWC
    4. Compare performances
    """
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    transform = transforms.Compose([transforms.ToTensor()])

    # Load datasets
    mnist_train = MNIST(root="./data", train=True, download=True, transform=transform)
    mnist_test = MNIST(root="./data", train=False, download=True, transform=transform)
    fashion_train = FashionMNIST(root="./data", train=True, download=True, transform=transform)
    fashion_test = FashionMNIST(root="./data", train=False, download=True, transform=transform)

    mnist_loader = DataLoader(mnist_train, batch_size=batch_size, shuffle=True)
    fashion_loader = DataLoader(fashion_train, batch_size=batch_size, shuffle=True)

    # Task A: Training on MNIST
    print("=" * 60)
    print("TASK A: Learning on MNIST")
    print("=" * 60)

    model_a = SimpleMLP().to(device)
    train_with_ewc(model_a, mnist_loader, fisher={}, opt_params={},
                   lambda_ewc=0, num_epochs=epochs, lr=lr, device=device)

    # Evaluate on MNIST before continuing
    accuracy_mnist_before = evaluate(model_a, mnist_test, device)
    print(f"MNIST accuracy after task A: {accuracy_mnist_before:.2f}%")

    # Compute the diagonal Fisher
    print("\nComputing the diagonal Fisher matrix...")
    fisher = compute_fisher_diagonal(model_a, mnist_loader, device, fisher_samples)
    opt_params = {n: p.clone().detach() for n, p in model_a.named_parameters() if p.requires_grad}

    # Task B: Training on Fashion-MNIST WITHOUT EWC
    print("\n" + "=" * 60)
    print("TASK B: Learning on Fashion-MNIST (WITHOUT EWC)")
    print("=" * 60)

    model_no_ewc = copy.deepcopy(model_a)
    history_no_ewc = train_with_ewc(model_no_ewc, fashion_loader,
                                     fisher={}, opt_params={},
                                     lambda_ewc=0, num_epochs=epochs, lr=lr, device=device)

    acc_mnist_no_ewc = evaluate(model_no_ewc, mnist_test, device)
    acc_fashion_no_ewc = evaluate(model_no_ewc, fashion_test, device)
    print(f"MNIST accuracy (forgetting):    {acc_mnist_no_ewc:.2f}%")
    print(f"Fashion-MNIST accuracy:         {acc_fashion_no_ewc:.2f}%")

    # Task B: Training on Fashion-MNIST WITH EWC
    print("\n" + "=" * 60)
    print(f"TASK B: Learning on Fashion-MNIST (WITH EWC, lambda={lambda_ewc})")
    print("=" * 60)

    model_ewc = copy.deepcopy(model_a)
    history_ewc = train_with_ewc(model_ewc, fashion_loader,
                                  fisher=fisher, opt_params=opt_params,
                                  lambda_ewc=lambda_ewc, num_epochs=epochs, lr=lr, device=device)

    acc_mnist_ewc = evaluate(model_ewc, mnist_test, device)
    acc_fashion_ewc = evaluate(model_ewc, fashion_test, device)
    print(f"MNIST accuracy (protected):     {acc_mnist_ewc:.2f}%")
    print(f"Fashion-MNIST accuracy:         {acc_fashion_ewc:.2f}%")

    # Comparison
    print("\n" + "=" * 60)
    print("COMPARATIVE SUMMARY")
    print("=" * 60)
    fmt = "{:<30} {:>10} {:>10}"
    print(fmt.format("Metric", "Without EWC", "With EWC"))
    print("-" * 52)
    print(f"MNIST forgetting                {acc_mnist_no_ewc:>9.2f}% {acc_mnist_ewc:>9.2f}%")
    print(f"Fashion-MNIST (new)             {acc_fashion_no_ewc:>9.2f}% {acc_fashion_ewc:>9.2f}%")
    forgetting = acc_mnist_before - acc_mnist_no_ewc
    forgetting_ewc = acc_mnist_before - acc_mnist_ewc
    print(f"Forgetting (MNIST drop)         {forgetting:>9.2f}pts {forgetting_ewc:>9.2f}pts")

    return model_a, model_no_ewc, model_ewc, fisher


def evaluate(model, test_dataset, device, batch_size=256):
    """Evaluates model accuracy on a test dataset."""
    loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in loader:
            images, labels = images.to(device), labels.to(device)
            output = model(images)
            _, predicted = output.max(1)
            total += labels.size(0)
            correct += predicted.eq(labels).sum().item()
    return 100.0 * correct / total


if __name__ == "__main__":
    run_sequential_experiment(
        lambda_ewc=5000,
        fisher_samples=200,
        epochs=5,
        lr=1e-3,
        batch_size=128
    )

Expected Typical Result

============================================================
TASK A: Learning on MNIST
============================================================
MNIST accuracy after task A: 97.85%

Computing the diagonal Fisher matrix...

============================================================
TASK B: Learning on Fashion-MNIST (WITHOUT EWC)
============================================================
MNIST accuracy (forgetting):    23.40%    <- catastrophe!
Fashion-MNIST accuracy:         86.12%

============================================================
TASK B: Learning on Fashion-MNIST (WITH EWC, lambda=5000)
============================================================
MNIST accuracy (protected):     94.30%    <- preserved!
Fashion-MNIST accuracy:         83.50%    <- slight compromise

============================================================
COMPARATIVE SUMMARY
============================================================
Metric                         Without EWC   With EWC
----------------------------------------------------
MNIST forgetting                        23.40%     94.30%
Fashion-MNIST (new)                     86.12%     83.50%
Forgetting (MNIST drop)             -74.45pts   -3.55pts

Without EWC, the model almost completely forgets MNIST (drop from 97.85 percent to 23.40 percent). With EWC, MNIST accuracy remains at 94.30 percent — catastrophic forgetting is reduced by more than 70 percentage points. The cost is modest: Fashion-MNIST reaches 83.50 percent instead of 86.12 percent, a trade-off largely acceptable for most applications.

Critical Hyperparameters

Lambda (λ) — Consolidation Strength

This is the most important hyperparameter of EWC. It controls the trade-off between stability (preserving old knowledge) and plasticity (efficiently learning the new task).

The optimal value depends heavily on the number of model parameters, the similarity between tasks, and the number of samples. A recommended strategy is to start with λ = 5,000 and adjust via cross-validation.

Fisher Sample Size (fisher_sample_size)

The number of samples used to estimate the Fisher diagonal influences the quality of the estimation:

• Too small (< 50): noisy estimation, some weights receive inaccurate importance scores.
• Typical (100 to 500): good trade-off between precision and computational cost.
• Very large (> 1,000): more precise estimation but high cost. Returns are diminishing.

In practice, 200 to 300 samples are generally sufficient to obtain a reliable estimate of the Fisher diagonal.

Learning Rate

A learning rate that is too high during the EWC phase can overwhelm the quadratic regularization. It is advisable to use a slightly lower learning rate for task B (for example, divide by 2 or 3 compared to task A) to let the EWC regularization play its protective role.

Advantages and Limitations of EWC

Advantages

1. Memory efficiency: The diagonal approximation stores only P values (one scalar per parameter), not a complete matrix. For a network with 1 million parameters, this represents about 4 MB — negligible.
2. Solid theoretical foundation: EWC derives directly from sequential Bayesian inference via the Laplace approximation. It is not an ad hoc heuristic, but a justified approximation of a fundamental statistical principle.
3. Fine selectivity: Each weight receives an individualized penalty. Unlike classical L2 regularization which penalizes all weights uniformly, EWC selectively protects important weights.
4. Theoretical cumulability: The Bayesian framework naturally extends EWC to more than two tasks by accumulating Fisher terms (although this poses practical challenges, see below).
5. Compatible with any optimizer: EWC simply adds to the loss function. Adam, SGD, RMSprop — all work with EWC regularization.

Limitations

1. Diagonal approximation: By ignoring covariances between parameters, EWC misses structural dependencies. Two weights that always work together (like the weights of a convolutional filter) might be treated independently, which does not reflect the reality of their joint importance.
2. Scaling to many tasks: The multi-task version requires storing F_i and θ_{i,A}^* for each task, which becomes memory-expensive. Variants like “Online EWC” (Schwarz et al., 2018) solve this problem by accumulating the Fisher recursively, but with an additional approximation.
3. Performance on very different tasks: When tasks are very dissimilar (for example, image classification then text translation), EWC can overly constrain the learning of the new task because almost all weights receive a non-negligible penalty.
4. Approximate Fisher estimation: The diagonal Fisher is estimated empirically on a finite sample and evaluated at the optimal point θ_A^*. This estimation can be imprecise, especially for very deep models where the loss landscape is complex.
5. No expansion mechanism: EWC cannot add new parameters to the model. If task B requires capabilities the model doesn’t have, EWC cannot create them — it simply reuses existing parameters.

4 Concrete Use Cases

1. Robotics — Multi-Motor Learning

A robotic arm must learn sequentially: first grasping spherical objects (task A), then cylindrical objects (task B), then flat objects (task C). Each task uses the same physical joints but with different trajectories. EWC allows the neural controller to preserve the robust grasping strategies acquired for spheres while adapting to cylinders. Without EWC, the robot would forget how to grasp spheres after learning cylinders.

Specific application: Adaptive motor control for service robots in unstructured domestic environments.

2. Medical Assistance — Evolving Models

A medical diagnostic model is trained on chest X-rays (task A): detection of pneumonia, tuberculosis, nodules. Later, the same model must also analyze bone X-rays (task B): fractures, osteoporosis, arthritis. EWC protects the essential lung feature detectors while allowing the model to learn the visual signatures of bone pathologies. This is crucial in a hospital setting where replacing a clinically validated model is costly and regulatorily complex.

Specific application: Multi-modality radiological diagnostic assistance systems with progressive validation.

3. Natural Language Processing — Domain Adaptation

A sentiment classification model is trained on movie reviews (task A). The company then wants to adapt it to electronics product reviews (task B), then to restaurant reviews (task C). Vocabulary and expressions change between domains, but the fundamental understanding of sentiment polarity (positive/negative) remains transferable. EWC protects the lower layers of the model (general semantic representations) while allowing the upper layers to adapt to the new domain-specific vocabulary.

Specific application: Cross-domain adaptation of NLP models without complete retraining or massive collection of labeled data.

4. Autonomous Vehicles — Environmental Learning

An autonomous vehicle perception system is trained on data collected in summer, city, dry conditions (task A). It must then adapt to winter conditions with snow and ice (task B), then to unmapped rural environments (task C). Road edge, sign, and pedestrian detectors must be preserved as they remain valid in all environments. EWC ensures that adaptation to new conditions does not degrade existing capabilities, which is a critical safety requirement in autonomous automotive.

Specific application: Onboard perception for autonomous vehicles with continuous adaptation to changing environmental conditions.

See Also

• Title: Mastering the Python Interview Question: Calculating the Square Root of x
• Check Point Membership in a Convex Polygon in O(log N) with Python