Contrastive Learning: Complete Guide — Contrastive Learning

Summary — Contrastive Learning is a self-supervised learning method that learns quality visual representations without any labels. By bringing together representations of two augmented versions of the same image (positive pairs) and pushing apart those of different images (negative pairs), the model develops a deep understanding of visual semantics. SimCLR, MoCo, and BYOL have demonstrated that this approach rivals supervised learning on classic vision tasks.

Mathematical Principle

1. InfoNCE Loss

The central cost function of Contrastive Learning is the Normalized Information Noise-Contrastive Estimation (InfoNCE):

$$L = -\log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]} \exp(\text{sim}(z_i, z_k) / \tau)}$$

Where:
– z_i, z_j: projections of two augmented views of the same image (positive pair)
– sim(u,v): cosine similarity = (u · v) / (||u|| · ||v||)
– τ: temperature, controls the sharpness of the distribution (typically 0.07-0.5)
– 2N: all samples in the batch (each image has 2 views, so N images = 2N views)
– The denominator includes all other views as negatives

2. SimCLR Architecture

SimCLR uses a three-stage pipeline:
1. Augmentation: for each image $x$, create two augmented views $\tilde{x}_i$ and $\tilde{x}_j$ (crop, color, blur)
2. Encoder: f(x) extracts features (ResNet-50) → h = f(x) ∈ R^d
3. Projection head: g(h) = W_2 · ReLU(W_1 · h) → z ∈ R^128

After pre-training, we discard g() and use f(x) as a feature extractor for downstream tasks.

3. MoCo (Momentum Contrast)

MoCo solves the batch size problem (SimCLR requires thousands of samples) by maintaining a queue of negatives and a momentum encoder:

$$k_{m} = m \cdot k_{m} + (1 – m) \cdot k$$

The momentum encoder k_m is an exponential moving average of the main encoder k. This guarantees consistent representations for negatives stored in the queue.

Intuition

Imagine a child sorting photos without knowing the names of objects.

We give them two photos of the same cat — one cropped on the head, the other with slightly different colors. The child learns that these two photos come from the same cat because they look alike despite the modifications.

At the same time, they look at photos of dogs, cars, trees — all different. They learn that these images are distinct.

Without ever hearing the word “cat,” the child develops an internal representation that groups similar views and separates different views. This is exactly what Contrastive Learning does.

Python Implementation

1. Simple SimCLR with PyTorch

[Python code block preserved as-is from original]

2. Linear Probing Evaluation

[Python code block preserved as-is from original]

Hyperparameters

Advantages

1. No labels needed: Learns directly from raw data, saving thousands of hours of annotation.
2. Generic representations: Learned features transfer to multiple tasks (classification, detection, segmentation).
3. Competitive performance: SimCLR with ResNet-50 achieves 76.5% top-1 on ImageNet, close to the 77% of direct supervised learning.
4. Robustness: Representations are more robust to image corruptions and perturbations than supervised models.

Limitations

1. High compute cost: SimCLR requires thousands of epochs and large batches. MoCo is more memory efficient.
2. Augmentation dependency: Augmentations that are too weak don’t provide enough variety; too strong and positive pairs become too different.
3. Redundant negatives: In a large batch, the majority of negatives are easy (very different images), providing little information.

4 Concrete Use Cases

1. Medical Classification with Few Labeled Data

Pre-trained with Contrastive Learning on 100,000 unlabeled X-rays, then fine-tuned with only 500 annotated X-rays to detect pathologies. Performance surpasses a model trained with only supervised learning on the same 500 examples.

2. Image Similarity Search

An e-commerce search engine uses features extracted by a SimCLR model to find images visually similar to a query image. Contrastive representations capture semantic similarity better than features from a classical supervised model.

3. Audio Representation Learning

By adapting augmentations to the audio domain (time stretching, pitch shifting, noise addition), Contrastive Learning learns robust representations for speech recognition or environmental sound classification.

4. Self-Supervised NLP

Contrastive multi-task learning for sentence representations: two versions of a sentence (with different deleted words) are brought together, while random sentences are pushed apart. This complements MLM approaches like BERT and improves embedding robustness on text similarity tasks.

Comparison of Contrastive Methods

The field has evolved rapidly since SimCLR (2020). Here are the main approaches:

Methods with negatives (explicit contrast):
– SimCLR: large batch for many negatives, simple but effective
– MoCo (v1-v3): negative queue + momentum encoder, memory efficient
– SupCon: Supervised Contrastive Learning, uses labels to create more positive pairs

Methods without negatives (implicit contrast):
– BYOL (Bootstrap Your Own Latent): two networks with momentum, no negatives needed
– SimSiam: even simpler, no momentum or negatives, just two branches with stop-gradient
– DINO: self-distillation with a teacher and student, spontaneous emergence of attention

Masked reconstruction methods:
– MAE (Masked Autoencoder): masks 75% of the image and reconstructs pixels, competitive with contrastive
– iBOT: combines distillation and masked reconstruction

Conclusion

Contrastive Learning ushered in the era of modern self-supervised learning. By learning rich representations without any labels, it drastically reduces the need for manual annotations while maintaining competitive performance.

SimCLR and MoCo paved the way for even more advanced methods like DINO (self-distillation without negatives) and MAE (Masked Autoencoder), confirming that the future of deep learning is self-supervised.

See Also

• Master Mahjong with Python: Complete Guide for Developers and Enthusiasts
• Master Grid Graphs with Python: Complete Guide and Optimization Tips