Active Learning: Complete Guide

Summary

Active Learning is an approach to machine learning where the model itself chooses the most informative examples to submit to a human annotator. Rather than passively training an algorithm on a vast corpus of already-labeled data — a process that is often costly and time-consuming — active learning asks the model to ask the right questions and precisely target ambiguous, uncertain, or boundary cases.

The result is spectacular: in many scenarios, an Active Learning-guided model achieves the same accuracy as a passive model with only 20% to 30% of the annotations required. This massive savings in annotation cost represents a decisive advantage when data labeling requires the intervention of specialized experts — doctors, lawyers, linguists, or financial analysts.

This guide explores the mathematical foundations, different selection strategies, practical implementation in Python with the modAL library, and concrete use cases that make Active Learning an indispensable tool for the modern practitioner.

Mathematical Principle of Active Learning

The core of active learning is based on a fundamental principle: iterative selection by maximization of a utility function. Formally, at each round, the model chooses the example $x^*$ from the pool of unlabeled examples that maximizes a utility function $U(x)$:

$$x^* = \underset{x}{\operatorname{argmax}} \; U(x)$$

where $U(x)$ measures how beneficial adding the annotation of $x$ to the training set will be for the model. It is this utility function that distinguishes the different Active Learning strategies.

Uncertainty Sampling

The most intuitive strategy selects the examples on which the model is least certain of its prediction. The entropy of the probability distribution is generally used as the uncertainty measure:

$$H(p(y|x)) = -\sum_{y_i \in \mathcal{C}} p(y_i|x) \cdot \log p(y_i|x)$$

where $\mathcal{C}$ is the set of possible classes and $p(y_i|x)$ is the probability estimated by the model that example $x$ belongs to class $y_i$. Entropy is maximum when the model assigns uniform probabilities to all classes: the model is then completely uncertain. Conversely, zero entropy means the model is absolutely confident in its prediction.

The model thus selects the example that maximizes entropy, i.e., the one where it hesitates the most between several classes. This approach naturally extends to margin sampling:

$$x^* = \underset{x}{\operatorname{argmin}} \left( p(y_1|x) – p(y_2|x) \right)$$

where $y_1$ and $y_2$ are respectively the two most probable classes according to the model for example $x$. Here, we select the example for which the difference between the probability of the most likely class and that of the second most likely class is minimal — in other words, the one where the model most hesitates between two candidate classes.

Query-by-Committee

This strategy maintains a collection of distinct models (a “committee”), each trained on the same labeled dataset but with different different initializations or distinct bootstrap subsets. The committee selects examples on which the individual models most disagree.

The agreement measure is based on vote entropy (VE):

$$VE(x) = -\sum_{y_i \in \mathcal{C}} V(y_i) \cdot \log V(y_i)$$

where $V(y_i)$ is the fraction of committee members who vote for class $y_i$. If all committee models agree, $V(y_i) = 1$ for a single class and entropy is zero. If models are evenly split across all classes, entropy reaches its maximum.

The major advantage of query-by-committee is that it captures not only the individual uncertainty of a model but also structural zones of disagreement between different hypotheses — thus revealing regions of the feature space where the model is sorely lacking discriminative information.

Expected Model Change

This approach selects examples that would change the model the most if they were annotated and added to the training set. The idea is profound: rather than measuring the model’s current uncertainty, it directly estimates the future impact of each potential annotation.

Several metrics exist to quantify this change:

• The norm of the expected gradient: the example that maximizes the norm of the loss function gradient is chosen, indicating a strong update of the model weights.
• The variation of predictions: an estimate of how much the predicted probabilities would change for all unlabeled data if this example were added.
• The expected Fisher information: examples that most increase the Fisher information matrix are selected, thus measuring the information gain on the model parameters.

Formally, we seek to maximize the expected change:

$$x^* = \underset{x}{\operatorname{argmax}} \; \mathbb{E}{y \sim p(y|x)} \left[ | \theta | \right]$$} – \theta_{\text{current}

where $\theta_{x,y}$ represents the model parameters after including the example $(x, y)$ in training.

Intuition — Why Active Learning Works

Imagine a student preparing for a marine biology exam. Faced with thousands of flashcards, they have two possible strategies. The first is to go through everything in order — from obvious animals like clownfish or blue whales to obscure species. The second, much smarter strategy is to focus on the borderline cases — those creatures that look like both a shark and a ray, those ambiguous mollusks, those corals whose classification is debated among specialists. Each answer to a well-chosen question is worth ten answers to obvious questions. The clever student doesn’t need to see a thousand dolphin photos to know what a dolphin is. But they need to understand precisely what distinguishes a porpoise from a dolphin.

Active Learning works exactly like this. Instead of learning passively on thousands of randomly drawn examples — a large majority of which will be trivial or redundant — the model actively identifies instances that lie close to the decision boundary. These boundary examples are where the model hesitates, the ones that reveal the most information about the underlying structure of the data.

Each annotation obtained through Active Learning provides disproportionate informational value. A single well-chosen example near the boundary between two classes can refine that boundary more than hundreds of examples far from the boundary. It is the algorithmic equivalent of asking the question that makes all the difference.

Python Implementation with modAL

The modAL library is the reference framework for implementing Active Learning in Python. It integrates seamlessly with scikit-learn and offers an elegant abstraction for query strategies. Here is a complete implementation illustrating Active Learning applied to a classification problem.

Installing dependencies

pip install modAL scikit-learn numpy matplotlib

Complete example: Active Learning vs. random learning

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from modAL.models import ActiveLearner
from modAL.uncertainty import uncertainty_sampling

# Generate synthetic data
X, y = make_classification(
    n_samples=1000,
    n_features=20,
    n_informative=15,
    n_redundant=5,
    n_classes=3,
    random_state=42
)

# Data split: unlabeled pool and test set
X_pool, X_test, y_pool, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Initialize with a small labeled set (5 examples per class)
np.random.seed(42)
initial_idx = []
for cls in np.unique(y_pool):
    class_indices = np.where(y_pool == cls)[0]
    selected = np.random.choice(class_indices, size=5, replace=False)
    initial_idx.extend(selected)

X_initial = X_pool[initial_idx]
y_initial = y_pool[initial_idx]

# Create the base model
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)

# Initialize the active learner
learner = ActiveLearner(
    estimator=rf_model,
    query_strategy=uncertainty_sampling,
    X_training=X_initial,
    y_training=y_initial
)

# Active Learning query loop
n_queries = 30
active_performances = []
random_performances = []

for iteration in range(n_queries):
    # The model selects the most uncertain examples
    query_idx, query_instance = learner.query(X_pool)

    # Simulated annotation (using the true labels)
    y_new = y_pool[query_idx]

    # Incremental training
    learner.teach(X_pool[query_idx], y_new)

    # Evaluation
    y_pred = learner.predict(X_test)
    perf = accuracy_score(y_test, y_pred)
    active_performances.append(perf)

    # Parallel: random learning for comparison
    ref_model = RandomForestClassifier(n_estimators=100, random_state=42)
    training_size = len(initial_idx) + (iteration + 1)
    random_idx = np.random.choice(
        len(X_pool), size=training_size, replace=False
    )
    ref_model.fit(X_pool[random_idx], y_pool[random_idx])
    y_pred_ref = ref_model.predict(X_test)
    perf_ref = accuracy_score(y_test, y_pred_ref)
    random_performances.append(perf_ref)

    print(f"Round {iteration + 1:2d} | Active: {perf:.4f} | Random: {perf_ref:.4f}")

# Visualization
plt.figure(figsize=(10, 6))
plt.plot(
    range(1, n_queries + 1), active_performances,
    'b-o', label='Active Learning (Uncertainty Sampling)', linewidth=2
)
plt.plot(
    range(1, n_queries + 1), random_performances,
    'r--s', label='Random Learning', linewidth=2
)
plt.xlabel("Number of queries (annotation batches)", fontsize=12)
plt.ylabel("Accuracy on test set", fontsize=12)
plt.title("Active Learning vs Random Learning", fontsize=14)
plt.legend(fontsize=11)
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('active_learning_comparison.png', dpi=150)
plt.show()

Advanced example: Query-by-Committee with modAL

from modAL.models import Committee
from modAL.disagreement import vote_entropy_sampling

# Create a committee of classifiers
n_committee = 5
classifiers = [
    RandomForestClassifier(n_estimators=50, random_state=i)
    for i in range(n_committee)
]

committee = Committee(
    learner_list=classifiers,
    query_strategy=vote_entropy_sampling
)

# Initial training
committee.teach(X_initial, y_initial)

# Committee query loop
for iteration in range(20):
    query_idx, _ = committee.query(X_pool)
    committee.teach(X_pool[query_idx], y_pool[query_idx])
    y_pred = committee.predict(X_test)
    print(f"Committee - Round {iteration + 1}: {accuracy_score(y_test, y_pred):.4f}")

The committee approach presents a significant advantage in situations where a single model might converge prematurely to a suboptimal solution. Disagreement between committee members reveals areas of ambiguity that an isolated model might not detect.

Essential Hyperparameters

Fine-tuning the parameters of an Active Learning pipeline directly influences the effectiveness of the selection. Here are the key hyperparameters to master.

n_queries — Number of query rounds

This parameter determines how many times the model will request new annotations. Too few queries — the model does not acquire enough information. Too many queries — the last annotations provide negligible marginal gain (diminishing returns). In practice, start with 20 to 50 queries and monitor the performance curve to identify the plateau.

batch_size — Annotation batch size

Instead of requesting a single annotation at a time, you can query a batch of examples. The batch_size controls this number. A batch that is too large dilutes the selection quality (the last examples in the batch are less informative). A batch of 5 to 20 examples represents a realistic compromise for most applications. Batch processing also allows parallelizing human annotation — multiple annotators work simultaneously.

learner — The base classifier

The choice of the underlying model is crucial. Random Forests are an excellent starting point: they provide well-calibrated probabilities, are robust to overfitting on small initial sets, and work well with uncertainty sampling. Support Vector Machines (SVMs) — coupled with margin sampling — offer excellent results when classes are linearly separable in a transformed feature space. Neural networks can be used with Monte Carlo Dropout techniques to estimate epistemic uncertainty, although this approach is more complex to implement and more computationally expensive.

uncertainty_measure — Uncertainty measure

modAL offers several selection functions:

• uncertainty_sampling: uses the probability of the most likely class (1 – p_max_class)
• entropy_sampling: uses the Shannon entropy of the probability distribution
• margin_sampling: uses the difference between the two most probable classes

In practice, margin sampling tends to outperform the other two on multi-class problems, as it explicitly considers the two most probable candidate classes instead of focusing solely on maximum confidence or global entropy.

Advantages and Limitations of Active Learning

Advantages

1. Massive reduction in annotation cost — This is the primary and most obvious advantage. By targeting the most informative examples, Active Learning achieves equivalent performance with significantly less labeled data.
2. Better generalization — Actively selected examples cover the feature space more effectively, particularly the boundary regions critical for model accuracy.
3. Adaptability — The model dynamically adapts to the distribution of unlabeled data. If the distribution changes, the selection strategy automatically adjusts.
4. Synergy with human expertise — The human annotator focuses exclusively on ambiguous cases that truly require their judgment, instead of wasting time on trivial examples.

Limitations

1. Selection bias — By selecting only boundary examples, there is a risk of neglecting regions of the feature space that, although dense, are not identified as uncertain. This bias can lead to an imprecise model for minority classes or atypical distributions.
2. Sensitivity to initialization — The quality of the first labeled examples strongly influences the early iterations. An unfortunate initialization can steer the model toward a poor initial understanding.
3. Computational cost — Each query round requires computing uncertainty over the entire unlabeled pool, which can become prohibitive with millions of examples.
4. Dependency on the annotator — If the annotator makes systematic errors on difficult cases — which is precisely what they are being asked to label — the model’s quality suffers directly.

4 Concrete Use Cases

Use Case #1: Legal Document Classification

A law firm wants to automatically categorize thousands of court decisions by legal domain. Annotation requires the expertise of specialized lawyers — each document takes between 10 and 30 minutes to analyze. With Active Learning, the model identifies decisions on the borderline between two domains (for example, between commercial law and corporate law). Lawyers annotate these ambiguous cases as a priority. Result: 92% accuracy achieved with only 800 annotations instead of the 4,000 required by random sampling.

Use Case #2: Anomaly Detection in Medical Imaging

A hospital deploys a system for detecting rare pathologies on chest radiographs. Radiologists have limited time to label hundreds of thousands of images. Active Learning targets images close to the boundary between healthy and pathological — difficult nodules, subtle infiltrates, ambiguous shadows. Each targeted annotation considerably refines the model’s decision boundary, enabling early detection with a reduced number of expert validations.

Use Case #3: Sentiment Analysis for Low-Resource Languages

For languages with few annotated resources — such as Swahili, Amharic, or Quechua — Active Learning makes it possible to build performant sentiment classifiers with only a few hundred annotations. The model identifies sentences whose sentiment is ambivalent or culturally nuanced — irony, double meanings, idiomatic expressions — and submits them to native speakers for annotation.

Use Case #4: Quality Control in Industrial Production

An automotive factory wants to automatically detect defects on metal parts using computer vision. Rare and subtle defects — micro-cracks, inclusions, minimal deformations — are the hardest to identify and the most critical for safety. Active Learning concentrates human inspections on parts whose appearance is ambiguous — neither clearly defective nor clearly conforming — thus optimizing the time of qualified inspectors and maximizing the detection rate of critical defects.

See Also

• Bellman-Ford Algorithm: Finding Shortest Paths with Negative Weights in Python
• Mastering Power Totient Sequences with Python: Complete Guide and Tips