DeepSORT: Complete Guide — Multi-Object Tracking with Deep Learning

Summary — DeepSORT (Simple Online and Realtime Tracking with Deep Associations) is one of the most widely used multi-object tracking algorithms in computer vision. It combines Kalman filter motion predictions with appearance descriptors from a neural network to associate detections frame by frame. Designed for real time, it handles temporary occlusions, crossings, and object appearances/disappearances with remarkable efficiency.

Mathematical Principle

1. Kalman Filter — Motion Prediction

The Kalman filter models the state of a track as an 8-dimensional vector:

$$
x = (u, v, s, r, \dot{u}, \dot{v}, \dot{s}, 0)
$$

Where ( (u, v) ) is the center of the bounding box, ( s ) the area, ( r ) the aspect ratio (width/height), and ( (\dot{u}, \dot{v}, \dot{s}) ) the corresponding linear velocities.

Prediction Step (Predict):
– State prediction: x_pred = F × x_prev where F is the transition matrix (constant velocity model)
– Covariance prediction: P_pred = F × P_prev × F^T + Q with Q the process noise

Update Step (Update):
– Kalman gain: K = P_pred × H^T × (H × P_pred × H^T + R)^-1
– State correction: x_k = x_pred + K × (z_k – H × x_pred)
– Covariance correction: P_k = (I – K × H) × P_pred

The Kalman filter maintains a probabilistic estimate of the future position of each tracked object, with uncertainty that grows during occlusions and decreases with each successful update.

2. Mahalanobis Distance

To measure the compatibility between a track prediction and a detection, the Mahalanobis distance is used, accounting for directional uncertainty:

$$
d^{(1)}(i, j) = (d_j – y_i)^T \cdot S_i^{-1} \cdot (d_j – y_i)
$$

Where ( d_j ) is detection ( j ), ( y_i ) is the predicted position of track ( i ), and ( S_i ) is the innovation covariance. A threshold of 9.481 (95th percentile of chi-squared with 4 degrees of freedom) eliminates statistically improbable associations.

3. Appearance Descriptor (CNN features)

A deep neural network (Wide Residual Network with ~1.2M parameters) is trained on a large person re-identification dataset (Market-1501) to extract a 128-dimensional vector representing the visual appearance of each detection. The appearance cost is the minimum cosine similarity between the current detection and the last 100 descriptors of the track:

$$
d^{(2)}(i, j) = \min { r_k^T \cdot \tilde{d}_j \mid r_k \in R_i }
$$

4. Combined Distance

$$
c_{i,j} = \lambda \cdot d^{(1)}(i,j) + (1 – \lambda) \cdot d^{(2)}(i,j)
$$

When the Mahalanobis distance exceeds the threshold, the association is rejected regardless of the appearance score. Otherwise, motion and appearance are combined. In practice, the matching cascade is used, which prioritizes recently observed tracks.

5. Hungarian Algorithm (Optimal Assignment)

The Hungarian algorithm (Kuhn-Munkres) solves the optimal assignment problem: minimizing the total cost by associating each track with at most one detection. This is a classic combinatorial optimization problem solved in polynomial time ( O(n^3) ).

6. Track Lifecycle

• Tentative: new detection not yet confirmed (n_init frames)
• Confirmed: active track maintained by successive associations
• Deleted: track not associated for max_age frames

Intuition

Imagine a football referee who must track every player on the pitch throughout the entire match.

• Detection (YOLO): on each frame, the referee sees 22 player positions
• Kalman Filter: the referee anticipates where each player will be on the next frame based on their speed and direction
• Appearance (CNN): the referee recognizes each player by their jersey, body type, haircut
• Matching: the referee associates each detected player with the tracked player that best matches in position AND appearance

When two players cross paths: the Kalman filter alone might confuse them (similar positions at crossing), appearance alone might confuse them (similar jerseys). But combining both signals allows them to be distinguished with precision.

When a player temporarily leaves the field (occlusion): the Kalman continues to predict their trajectory. When they reappear, the appearance allows instant re-identification, even several seconds later.

Python Implementation

1. Detection + Tracking with YOLO and DeepSORT

import cv2
import numpy as np
from ultralytics import YOLO
from deep_sort_realtime.deepsort_tracker import DeepSort

# Load the detector and the DeepSORT tracker
detector = YOLO("yolov8n.pt")
tracker = DeepSort(
    max_age=30,
    n_init=3,
    max_cosine_distance=0.2,
    nn_budget=100,
)

cap = cv2.VideoCapture("traffic.mp4")
colors = {}
frame_count = 0
allowed_classes = {0, 2, 3, 5, 7}  # person, car, motorcycle, bus, truck

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    results = detector(frame, conf=0.4)
    detections = []

    for result in results:
        for box in result.boxes:
            cls_id = int(box.cls[0])
            if cls_id not in allowed_classes:
                continue

            x1, y1, x2, y2 = box.xyxy[0].tolist()
            conf = float(box.conf[0])
            label = detector.model.names[cls_id]
            detections.append(([x1, y1, x2 - x1, y2 - y1], conf, label))

    tracks = tracker.update_tracks(detections, frame=frame)

    for track in tracks:
        if not track.is_confirmed():
            continue

        track_id = track.track_id
        left, top, right, bottom = map(int, track.to_ltrb())
        color = colors.setdefault(
            track_id, tuple(np.random.randint(0, 255, 3).tolist())
        )

        cv2.rectangle(frame, (left, top), (right, bottom), color, 2)
        cv2.putText(
            frame,
            f"ID {track_id}",
            (left, top - 10),
            cv2.FONT_HERSHEY_SIMPLEX,
            0.5,
            color,
            2,
        )

    cv2.imshow("DeepSORT Tracking", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

    frame_count += 1

cap.release()
cv2.destroyAllWindows()

2. Trajectory Extraction and Counting

from collections import defaultdict
import numpy as np

trajectories = defaultdict(list)

# ... to be placed right after `tracks = tracker.update_tracks(...)`
for track in tracks:
    if not track.is_confirmed():
        continue

    left, top, right, bottom = track.to_ltrb()
    cx = (left + right) / 2.0
    cy = (top + bottom) / 2.0
    trajectories[track.track_id].append((frame_count, cx, cy))

# Example analysis after video processing
for track_id, points in trajectories.items():
    if len(points) < 10:
        continue

    crossings = 0
    for i in range(1, len(points)):
        if points[i - 1][2] < 300 <= points[i][2]:
            crossings += 1

    segment_distances = [
        np.hypot(points[i][1] - points[i - 1][1], points[i][2] - points[i - 1][2])
        for i in range(1, len(points))
    ]
    avg_speed = np.mean(segment_distances) if segment_distances else 0.0

    print(
        f"Track {track_id}: {len(points)} frames, "
        f"{crossings} crossings, avg speed {avg_speed:.1f} px/frame"
    )

Hyperparameters

Advantages of DeepSORT

1. Real time: With YOLOv8-nano as the detector, DeepSORT achieves 30+ FPS on a modest GPU
2. Robustness to occlusions: The Kalman filter predicts during occlusions, appearance enables re-identification
3. Easy to integrate: Just a few lines to add tracking to any detector
4. ID stability: Unlike IoU-only tracking, DeepSORT maintains IDs through crossings

Limitations of DeepSORT

1. Detector dependency: If the detector misses an object for several frames, the Kalman filter eventually loses it
2. Sensitive to abrupt appearance changes: A vehicle changing viewing angle may lose association
3. No 3D tracking: Works only in 2D image space
4. Memory: The appearance gallery (100 descriptors × 128 dimensions per track) consumes a lot with hundreds of simultaneous tracks

4 Concrete Use Cases

1. Highway Vehicle Counting

Surveillance cameras equipped with YOLO + DeepSORT count vehicles in each lane, measure their average speed, and detect traffic jams in real time. Tracking avoids double-counting when a vehicle passes through multiple camera fields of view.

2. Sports Behavior Analysis

In a basketball game, DeepSORT tracks each player and the ball. Extracted trajectories enable tactical pattern analysis, measurement of distances covered by each player, and identification of the most frequented areas of the court.

3. Airport Security Surveillance

A video surveillance system tracks luggage and people to detect abandoned baggage (detected object without an associated track for more than X minutes) and suspicious movements in restricted areas.

4. Crowd Analysis and Flow Management

In train stations, counting pedestrians by direction makes it possible to estimate waiting times, detect panic movements (abnormal speed), and optimize signage in real time to distribute passenger flows.

Conclusion

DeepSORT is the reference in multi-object tracking for its simplicity and real-time performance. By combining motion prediction (Kalman), visual appearance (CNN), and optimal assignment (Hungarian algorithm), it offers a robust and elegant solution to the problem of tracking objects in videos. Although newer approaches such as ByteTrack (without appearance learning) and Transformer-based trackers are emerging, DeepSORT remains an excellent choice for most practical applications in 2026.

See Also

• Implement the SHA-1 Hashing Algorithm in Python: Step-by-Step Guide
• Draw a Triangle with Circle Arcs in Python: Complete Guide for Developers