MAML: Complete Guide — Model-Agnostic Meta-Learning

Summary — MAML (Model-Agnostic Meta-Learning) is a meta-learning algorithm proposed by Finn, Abbeel, and Levine in 2017. It learns an initialization of model parameters such that after a few gradient steps on a new task, performance is good. Unlike architecture-specific meta-learning (such as memory networks), MAML works with any model trainable by gradient (hence “model-agnostic”). It is the reference method for few-shot learning: learning from 1-5 examples per class.

Mathematical Principle

1. The few-shot learning problem

In classical learning, a model needs thousands of examples per class. Few-shot learning aims to learn new tasks with only K examples per class (K=1 or 5 typically). Training a model from scratch on K examples is impossible (massive overfitting). MAML solves this problem by learning a “good” initialization.

2. Meta-learning formulation

We have a task distribution p(T). Each task T_i has a support set S_i (K examples per class, used for adaptation) and a query set Q_i (used to evaluate adaptation):

min_θ sum_{T_i ~ p(T)} L_{T_i}(f_{θ_i'})
where θ_i' = θ - α · ∇_θ L_{T_i}(f_θ)

The meta-model θ is updated based on performance after adaptation (θ_i’), not before.

3. The meta-gradient

The update is a gradient descent on the meta-gradient:

θ ← θ - β · ∇_θ sum_i L_{T_i}(f_{θ - α∇L})

This requires differentiating through the gradient step (hence “gradient of a gradient”, or second-order meta-gradient). In practice, the first-order approximation (FOMAML) is used by ignoring second derivatives:

∇_θ L_FOMAML ~ sum_i ∇_{θ_i'} L_{T_i}(f_{θ_i'})

4. Complete algorithm

Initialize θ randomly
For each meta-batch:
  For each task T_i:
    1. Compute gradients on the support set: g_i = ∇_θ L_{T_i}(f_θ)
    2. Adapt parameters: θ_i' = θ - α · g_i
    3. Evaluate on the query set: L_{T_i}(f_{θ_i'})
  Update θ with meta-gradients: θ ← θ - β · ∇_θ sum_i L_{T_i}(f_{θ_i'})

Intuition

Instead of learning a model from scratch for each new task, MAML learns “how to learn”. It’s like a student who has a good work method: when they encounter a new subject, they learn quickly because they already know how to approach it.

MAML finds the right “learning reflexes” — a weight initialization that is equidistant from all possible tasks, ready to adapt in any direction with just a few examples.

It’s comparable to a multilingual athlete: they don’t know every language in the world, but they’ve developed a language-learning ability that allows them to master a new one in weeks where a beginner would take years.

Python Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F
from copy import deepcopy


class MAML:
    def __init__(self, model, inner_lr=0.01, outer_lr=1e-3, inner_steps=5):
        self.model = model
        self.inner_lr = inner_lr
        self.outer_lr = outer_lr
        self.inner_steps = inner_steps
        self.outer_opt = torch.optim.Adam(self.model.parameters(), lr=outer_lr)

    def inner_adapt(self, task_support_x, task_support_y):
        """Inner loop: adapt weights on the support set."""
        adapted = deepcopy(self.model)
        adapted_opt = torch.optim.SGD(adapted.parameters(), lr=self.inner_lr)

        for step in range(self.inner_steps):
            preds = adapted(task_support_x)
            loss = F.cross_entropy(preds, task_support_y)
            adapted_opt.zero_grad()
            loss.backward()
            adapted_opt.step()

        return adapted

    def meta_update(self, tasks):
        """Outer loop: meta-update on multiple tasks."""
        meta_loss = 0

        for task in tasks:
            support_x, support_y, query_x, query_y = task

            # Adapt on the support set
            adapted = self.inner_adapt(support_x, support_y)

            # Evaluate on the query set
            query_pred = adapted(query_x)
            task_loss = F.cross_entropy(query_pred, query_y)
            meta_loss += task_loss

        meta_loss = meta_loss / len(tasks)

        self.outer_opt.zero_grad()
        meta_loss.backward()
        self.outer_opt.step()

        return meta_loss.item()


# Few-shot classification on Omniglot (5-way 1-shot)
def create_task(dataset, n_way=5, k_shot=1, q_per_class=15):
    """Create a few-shot task from the dataset."""
    classes = torch.randperm(len(dataset.classes))[:n_way]
    support_x, support_y = [], []
    query_x, query_y = [], []

    for c_idx, cls in enumerate(classes):
        samples = dataset.get_samples(cls, k_shot + q_per_class)
        support_x.append(samples[:k_shot])
        support_y.extend([c_idx] * k_shot)
        query_x.append(samples[k_shot:])
        query_y.extend([c_idx] * q_per_class)

    return (torch.cat(support_x), torch.tensor(support_y),
            torch.cat(query_x), torch.tensor(query_y))


# Training
model = nn.Sequential(
    nn.Conv2d(1, 32, 3), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(32, 32, 3), nn.ReLU(), nn.MaxPool2d(2),
    nn.Flatten(), nn.Linear(32*12*12, 5)
)

maml = MAML(model, inner_lr=0.01, outer_lr=1e-3, inner_steps=5)

for meta_epoch in range(1000):
    tasks = [create_task(dataset, n_way=5, k_shot=1) for _ in range(4)]
    loss = maml.meta_update(tasks)

    if meta_epoch % 100 == 0:
        print(f'Meta-epoch {meta_epoch} | Loss: {loss:.4f}')

# Evaluation: test on a new unseen task
new_task = create_task(dataset, n_way=5, k_shot=1)
adapted = maml.inner_adapt(new_task[0], new_task[1])
with torch.no_grad():
    preds = adapted(new_task[2])
    acc = (preds.argmax(dim=1) == new_task[3]).float().mean()
    print(f'5-way 1-shot accuracy: {acc.item()*100:.1f}%')

Hyperparameters

Comparison with other meta-learning methods

MAML vs Metric-Based (ProtoNet, Matching Networks)

Metric-based approaches (ProtoNet, Matching Networks, Relation Networks) learn an embedding space where examples of the same class are close together. MAML, by contrast, learns an initialization of the model’s weights. Metric approaches require re-classifying at each new task (costly), while MAML can directly fine-tune the existing classifier.

MAML vs Reptile

Reptile is a first-order approximation of MAML: instead of backpropagating through the adaptation steps, it simply adjusts the original parameters toward the adapted parameters: θ ← θ + β · (θ’ – θ). It is much simpler and faster, with comparable performance in most cases.

MAML vs Standard Fine-tuning

Fine-tuning starts from an initialization pre-trained on a source task. MAML goes further: it specifically finds the initialization that is easiest to adapt, even if it is not optimal for the source task. It’s the difference between being a generalist expert and being an expert who knows how to adapt.

Advantages

1. Model-agnostic: Works with any architecture trainable by gradient (CNN, RNN, Transformers).
2. Low data requirement: Adapts to new tasks with only 1-5 examples per class.
3. Fast adaptation: Adaptation requires only a few gradient steps, quick to compute.

Limitations

1. Expensive meta-training: Each meta-batch requires adaptation on multiple tasks, each with inner_steps forward/backward passes.
2. Memory: Second-order gradients require storing the entire computation graph of the adaptation.
3. Variable performance: MAML can be unstable — meta-gradients have high variance.

4 Concrete Use Cases

1. Rare character recognition

In OCR systems for languages with thousands of characters (Chinese, Japanese), new characters can appear. MAML makes it possible to recognize a new character after only 1-3 examples.

2. Personalized recommendation

For a new user on a platform, MAML quickly adapts the recommendation model to their first 5-10 interactions, providing relevant recommendations from day one.

3. Robotic control transfer

A robot trained with MAML on dozens of different terrains (grass, sand, ice) quickly adapts to a new unseen terrain with just a few minutes of trials.

4. Medical diagnosis for rare cases

For rare pathologies with few available cases, MAML adapts a general medical imaging model to the specific case with the few available examples.

5. Language model adaptation to dialects

A language model trained on standard English can quickly adapt to a regional dialect thanks to the MAML initialization that captures basic linguistic skills. A few hundred sentences are enough to achieve satisfactory performance on the target dialect, avoiding a complete retraining of the model. This approach is particularly useful for languages or dialects with limited resources.

Conclusion

MAML popularized the idea of learning to learn through optimization. Its influence can be seen in modern few-shot learning algorithms (ProtoNet, Relation Network, MetaOptNet) and has inspired knowledge transfer between language models (fine-tuning on multiple tasks).

Variants such as Reptile (First-order MAML simplified without second-order gradients) and ANIL (Almost No Inner Loop) have made meta-learning more accessible and less costly. These advances show that MAML’s fundamental principle — learning an adaptive initialization rather than a fixed model — is robust and extends beyond the original use cases.

See Also

• Mastering Binomial Product Dividers with Python: Complete Guide
• Mastering Primary Repunit Property Composites with Python: Guide and Applications