Mask R-CNN: Complete Guide — Instance Detection and Segmentation

Summary — Mask R-CNN, proposed by He, Gkioxari, Dollár, and Girshick from Facebook AI Research in 2017, is an extension of Faster R-CNN that adds a segmentation branch to produce pixel-perfect masks for each detected object. It is the reference algorithm for instance segmentation: it detects each individual object in an image, classifies it, and draws its precise contour pixel by pixel. The key contribution is RoI Align, which replaces Faster R-CNN’s approximate RoI Pooling with exact bilinear interpolation, eliminating mask misalignment.

Mathematical Principles

1. Overall architecture

Mask R-CNN = Faster R-CNN + Mask Branch:

1. Backbone + FPN: a ResNet-FPN extracts multi-scale features
2. Region Proposal Network (RPN): proposes ~1000 Regions of Interest (RoIs) via anchor boxes
3. RoI Align: extracts fixed-size (7×7) features for each RoI exactly
4. Branches: classification + bounding box + binary mask (m×m per class)

2. RoI Align — the key contribution

Faster R-CNN’s RoI Pooling quantizes RoI coordinates to integers, creating offsets of up to 0.5 pixels. For segmentation, this is catastrophic: the mask is shifted.

RoI Align removes this quantization and uses bilinear interpolation:

For each sampling point (x, y) in the extracted feature map:
– Compute the 4 nearest neighbor pixels: p1, p2, p3, p4
– Interpolate: V(x,y) = w1·V(p1) + w2·V(p2) + w3·V(p3) + w4·V(p4)

Where the weights w_i are computed from the relative distance of (x, y) to each pixel.

In practice, 4 points are sampled per bin cell (typically 2×2 = 4 points) and the max of each cell is taken.

3. Total loss

L = L_cls + L_box + L_mask

Where:
– L_cls: Cross-Entropy for class classification
– L_box: Smooth L1 loss for bounding box regression (4 coordinates)
– L_mask: Pixel-wise BCE (Binary Cross-Entropy) for each class key (K masks of m×m)

Crucially, L_mask is computed independently for each class: the mask for class k uses only the k-th output of the mask branch. This allows the model to generate masks for all classes without competition between them. The correct class is selected at inference time.

4. Feature Pyramid Network (FPN)

The FPN extracts features at different spatial scales:
– P2: 1/4 of input resolution (small objects)
– P3: 1/8 (medium objects)
– P4: 1/16 (large objects)
– P5: 1/32 (very large objects)

Each FPN level is used by the RPN to generate proposals adapted to object size.

Intuition

Faster R-CNN says “there’s a cat here” with a bounding box. Mask R-CNN goes one step further: it says exactly which pixels belong to the cat and which pixels are the background. It’s the difference between outlining an object with a thick marker (bounding box) and coloring it precisely with a fine brush (mask).

The key improvement (RoI Align) is like switching from an artist who draws freehand (approximation) to an artist who uses a ruler and compass (pixel-level precision). This precision is essential when you want to know exactly where the cat pixel ends and the couch pixel begins.

Python Implementation

1. Inference with pretrained Mask R-CNN

import torch
import torchvision
from torchvision.models.detection import maskrcnn_resnet50_fpn, MaskRCNN_ResNet50_FPN_Weights
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np

# Load pretrained model on COCO
weights = MaskRCNN_ResNet50_FPN_Weights.DEFAULT
model = maskrcnn_resnet50_fpn(weights=weights, num_classes=91)
model.eval()

# Load an image
img = Image.open("photo.jpg").convert("RGB")
transform = weights.transforms()
img_tensor = transform(img)

# Inference
with torch.no_grad():
    predictions = model([img_tensor])[0]

# Display detections
COCO_CATEGORIES = weights.meta["categories"]
threshold = 0.7

fig, ax = plt.subplots(1, 1, figsize=(12, 8))
ax.imshow(np.array(img))

for i in range(len(predictions["scores"])):
    if predictions["scores"][i] < threshold:
        continue

    box = predictions["boxes"][i].cpu().numpy()
    mask = predictions["masks"][i, 0].cpu().numpy()
    label = COCO_CATEGORIES[predictions["labels"][i]]
    score = predictions["scores"][i].item()

    # Bounding box
    from matplotlib.patches import Rectangle
    rect = Rectangle((box[0], box[1]), box[2]-box[0], box[3]-box[1],
                     edgecolor="lime", facecolor="none", linewidth=2)
    ax.add_patch(rect)

    # Semi-transparent mask
    ax.contour(mask, levels=[0.5], colors="cyan", linewidths=1.5, alpha=0.7)

    # Label
    ax.text(box[0], box[1]-5, f"{label}: {score:.2f}", color="white",
            bbox=dict(facecolor="red", alpha=0.7, pad=2), fontsize=10)

plt.axis("off")
plt.tight_layout()
plt.savefig("mask_rcnn_detections.png", dpi=150)
plt.show()

2. Training on a custom dataset

import torch
import torchvision
from torchvision.models.detection import maskrcnn_resnet50_fpn, MaskRCNN_ResNet50_FPN_Weights
from torchvision.datasets import VOCDetection

class PennFudanDataset(torch.utils.data.Dataset):
    """Custom dataset for Mask R-CNN with masks."""
    def __init__(self, root, transforms=None):
        self.root = root
        self.transforms = transforms
        self.imgs = sorted(os.listdir(os.path.join(root, "PNGImages")))
        self.masks = sorted(os.listdir(os.path.join(root, "PedMasks")))

    def __getitem__(self, idx):
        img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
        mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])

        img = Image.open(img_path).convert("RGB")
        mask = Image.open(mask_path)
        mask = np.array(mask)

        # Extract individual instances from the mask
        obj_ids = np.unique(mask)[1:]  # Ignore background (0)
        masks = (mask == np.expand_dims(obj_ids, axis=(1, 2))).astype(np.uint8)

        boxes = []
        for i in range(len(obj_ids)):
            pos = np.where(masks[i])
            xmin = np.min(pos[1]); xmax = np.max(pos[1])
            ymin = np.min(pos[0]); ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        target = {
            "boxes": torch.as_tensor(boxes, dtype=torch.float32),
            "labels": torch.ones((len(obj_ids),), dtype=torch.int64),
            "masks": torch.as_tensor(masks),
            "image_id": torch.tensor([idx])
        }

        if self.transforms:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

# Training
model = maskrcnn_resnet50_fpn(weights=None, num_classes=2)
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

dataset = PennFudanDataset("PennFudanPed/")
loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, collate_fn=lambda x: tuple(zip(*x)))

for epoch in range(10):
    model.train()
    total_loss = 0
    for images, targets in loader:
        loss_dict = model(images, targets)
        losses = sum(loss for loss in loss_dict.values())
        optimizer.zero_grad()
        losses.backward()
        optimizer.step()
        total_loss += losses.item()
    lr_scheduler.step()
    print(f'Epoch {epoch}: Loss {total_loss/len(loader):.4f}')

Hyperparameters

Advantages

1. Pixel-perfect accuracy: RoI Align eliminates quantization offsets, producing masks 10-100x more accurate than previous methods.
2. Multi-task: Detection + classification + segmentation in a single pass.
3. General-purpose: Works on any object class, not just the COCO classes.
4. Mature framework: Integrated in torchvision, easy to use with custom datasets.

Limitations

1. Slow at inference: ~5-10 FPS on GPU, insufficient for real-time applications.
2. Memory-hungry: FPN + masks require a lot of VRAM.
3. Small objects: Very small objects (< 32×32 pixels) are poorly detected because the FPN has limited resolution.
4. RPN dependency: If the RPN misses an object (false negative), the mask branch cannot recover it.

4 Concrete Use Cases

1. Medical analysis: cell segmentation

Mask R-CNN individually segments blood cells in smears, enabling automated counting and type classification (red blood cells, white blood cells, platelets). Each cell has its precise mask.

2. Robotics: object manipulation

A robotic arm uses Mask R-CNN to identify and locate objects on a table. The precise masks allow computing the center of mass and exact shape of each object for optimal grasping.

3. Intelligent video surveillance

Blurring the faces of detected pedestrians to comply with GDPR while retaining detection boxes for crowd counting and motion analysis.

4. Agriculture: fruit counting

Mask R-CNN detects and segments each fruit on a tree to estimate yield. The precise masks separate overlapping fruits, improving counting accuracy by 15% compared to bounding box methods.

Conclusion

Mask R-CNN remains the reference algorithm for instance segmentation. Its main contribution — RoI Align — is now used in numerous subsequent detection architectures.

Although faster architectures such as SOLO and YOLO-seg have emerged, Mask R-CNN retains its advantage in terms of precision and implementation maturity, especially for applications where mask accuracy is critical.

See Also

• Mastering Circular Logic in Python: Complete Guide for Beginners and Experts
• Mastering Maximum Arrangements in Python: Complete Guide and Tips for Developers