CatBoost: Complete Guide — Categorical Boosting

Summary — CatBoost (Categorical Boosting) is a gradient boosting algorithm developed by Yandex (the Russian research team behind the Yandex search engine) in 2017. Its particularity is that it natively handles categorical features without preprocessing (one-hot encoding, label encoding), thanks to ordered statistical encoding that prevents target leakage. Combined with its ordered boosting (a gradient boosting variant that reduces overfitting), CatBoost offers competitive performance with minimal tuning.

Mathematical Principles

1. Category Encoding via Target Statistics

Rather than converting categories to one-hot encoding (which explodes dimensionality for features with many modalities), CatBoost encodes them using a target statistic:

$$\text{Encoding}(c) = \frac{\sum_{i=1}^{n} I(x_i = c) \cdot y_i + a \cdot P}{\sum_{i=1}^{n} I(x_i = c) + a}$$

where I(x_i = c) is 1 if example i has category c, y_i is the target, a is a smoothing parameter (prior), and P is the prior value.

2. Ordered Target Statistics: Preventing Target Leakage

The classic problem of target mean encoding is target leakage: if you compute the target mean for a category by including the example itself, the model indirectly sees the answer during training, causing massive overfitting.

CatBoost solves this problem with Ordered Target Statistics: to encode the category of example i, only the examples that precede it in a random order (a permutation σ of the data) are considered:

$$\text{OrderedEncoding}(c_i) = \frac{\sum_{j=1}^{i-1} I(x_{\sigma(j)} = c) \cdot y_{\sigma(j)} + a \cdot P}{\sum_{j=1}^{i-1} I(x_{\sigma(j)} = c) + a}$$

This is a sliding window that guarantees each example only sees the past (the examples that precede it in order σ). It’s analogous to a time series where future data is inaccessible.

3. Ordered Boosting

CatBoost goes further: it applies the same ordering principle not only to target encoding but also to gradient computation (the boosting itself). Instead of computing gradients over all examples, each tree is trained using gradients computed only on preceding examples in order σ.

This ordered boosting approach significantly reduces overfitting that affects classic gradient boosting algorithms, especially on small datasets.

4. Category Combinations

To capture interactions between categorical features, CatBoost automatically builds combinations:

• At a tree’s split level, CatBoost considers not only each individual categorical feature but also combinations of 2, 3, or even 4 categorical features.
• This is effective because the number of combinations remains limited through filtering by previous splits.

Intuition

Imagine you are a bookseller and you want to recommend books to your customers.

The classical approach (one-hot encoding): each customer receives a unique identification number, and you create one column per customer in your table. With 10,000 customers, your table has 10,000 columns. It’s unmanageable.

CatBoost’s approach: instead of numbering customers, you directly learn what type of book each customer likes by looking at past purchase history. You know your customers by name, not by number — and you use the past to predict the future.

Ordered boosting is like reading a book in order: you don’t know the ending when you’re on chapter 3. Similarly, CatBoost never looks at the future when encoding a category.

Python Implementation

Example 1: Basic CatBoostClassifier

import pandas as pd
from catboost import CatBoostClassifier, Pool
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Data with categorical features
df = pd.DataFrame({
    'age': [25, 30, 35, 45, 50, 22, 28, 38, 42, 55],
    'revenue': [30000, 45000, 60000, 80000, 95000, 25000, 38000, 70000, 75000, 100000],
    'city': ['Paris', 'Lyon', 'Marseille', 'Paris', 'Bordeaux', 'Lyon', 'Paris', 'Toulouse', 'Marseille', 'Paris'],
    'education': ['Bachelor', 'Master', 'PhD', 'Master', 'Bachelor', 'High School', 'Bachelor', 'Master', 'PhD', 'High School'],
    'purchase': [0, 1, 1, 1, 1, 0, 0, 1, 1, 0]
})

X = df.drop('purchase', axis=1)
y = df['purchase']

# Identify categorical columns
cat_cols = ['city', 'education']
cat_idx = [X.columns.get_loc(c) for c in cat_cols]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# CatBoost
model = CatBoostClassifier(
    iterations=100,
    learning_rate=0.1,
    depth=6,
    loss_function='Logloss',
    cat_features=cat_idx,
    verbose=False,
    random_state=42
)
model.fit(X_train, y_train)
pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, pred):.3f}")

Example 2: CatBoost vs XGBoost Comparison with Encoding

from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import accuracy_score
import time

# Prepare data for XGBoost (encoding required)
from sklearn.datasets import fetch_openml
X_full, y_full = fetch_openml('adult', version=2, return_X_y=True, as_frame=True)
X_full = X_full.select_dtypes(include=['number', 'category'])
y_full = (y_full == '>50K').astype(int)

X_train, X_test, y_train, y_test = train_test_split(X_full, y_full, test_size=0.2, random_state=42)

# CatBoost (with native categoricals)
cat_full_idx = X_train.select_dtypes(include=['category']).columns.tolist()

start = time.time()
cb = CatBoostClassifier(iterations=200, learning_rate=0.1, depth=6,
    cat_features=cat_full_idx, verbose=False, random_state=42)
cb.fit(X_train, y_train)
cb_time = time.time() - start
cb_acc = accuracy_score(y_test, cb.predict(X_test))
print(f"CatBoost: {cb_time:.2f}s | Acc: {cb_acc:.3f}")

Example 3: Feature Importance and Visualization

import matplotlib.pyplot as plt

# Feature importance
importances = model.get_feature_importance()
feature_names = X_train.columns

plt.figure(figsize=(8, 5))
plt.barh(range(len(importances)), importances)
plt.yticks(range(len(importances)), feature_names)
plt.xlabel('Importance')
plt.title("CatBoost - Feature Importance")
plt.tight_layout()
plt.savefig('catboost_importance.png', dpi=150)

Example 4: CatBoost with Pool and Early Stopping

# Pool API for optimization
train_pool = Pool(X_train, y_train, cat_features=cat_full_idx)
val_pool = Pool(X_test, y_test, cat_features=cat_full_idx)

model_es = CatBoostClassifier(
    iterations=1000,
    learning_rate=0.05,
    depth=6,
    cat_features=cat_full_idx,
    eval_metric='AUC',
    early_stopping_rounds=50,
    verbose=100,
    random_state=42
)
model_es.fit(train_pool, eval_set=val_pool)
print(f"\nBest iteration: {model_es.best_iteration_}")
print(f"Best AUC: {model_es.best_score_['validation']['AUC']:.3f}")

Hyperparameters

Advantages of CatBoost

1. Native categorical handling: No need for LabelEncoder or OneHotEncoder upstream. You pass categorical columns directly to CatBoost, which handles them. Huge time savings in preprocessing.
2. Ordered Target Statistics eliminates target leakage: Unlike LightGBM which uses simple category encoding, CatBoost prevents the model from seeing the future during encoding. It’s safer and more reliable.
3. Ordered Boosting reduces overfitting: CatBoost is generally more stable than XGBoost and LightGBM on small datasets thanks to this mechanism. Less risk of having to exhaustively tune hyperparameters.
4. Minimal tuning needed: CatBoost’s default values are often very good. In practice, CatBoost is the boosting algorithm that requires the least grid search to achieve good results.
5. Fast prediction: A trained CatBoost model is a compact binary file, and the prediction phase is very fast, compatible with production constraints.

Limitations of CatBoost

1. Sometimes slow training: Ordered boosting is more expensive than standard boosting. On very large datasets, CatBoost can be 2-3x slower to train than LightGBM.
2. High memory usage: CatBoost’s symmetric (oblivious) trees require more memory than LightGBM’s asymmetric trees. On memory-limited machines, this is a disadvantage.
3. Less popular than XGBoost: Fewer tutorials, fewer Stack Overflow questions, less integration in standard pipelines. The community is smaller.
4. Very high cardinality categoricals: If a feature has millions of unique modalities, even ordered target encoding can lose precision. Binning techniques may be necessary.
5. Sometimes problematic installation: CatBoost requires a C++ compiler for installation on certain environments (notably older Python versions on Windows).

4 Concrete Use Cases

1. Credit Scoring with Heterogeneous Data

Credit scoring datasets combine numerical features (revenue, number of loans) and categorical features (employment type, marital status, region). CatBoost handles this mix without preprocessing, and its ordered boosting prevents critical overfitting biases in the financial domain.

2. Content Recommendation with Rich Categorical Data

For recommendation systems (movies, products, articles), categorical features dominate: genre, director, actor, brand, product category. CatBoost captures interactions between these categories automatically, without manual feature engineering.

3. Customer Churn Prediction with Categoricals

Churn data often contains variables such as subscription type, acquisition channel, geographic localization — all categorical with many modalities. CatBoost handles these features directly and offers competitive performance with minimal tuning.

4. Text Classification with Mixed Features

In document classification, you can combine text embeddings (numerical) with categorical metadata (source, author, main topic, language). CatBoost merges these two types of features naturally.

Conclusion

CatBoost stands out in the boosting ecosystem by its ability to natively handle categorical features, combined with ordered boosting that significantly reduces overfitting. It’s the ideal choice when data contains many categories and preprocessing time is valuable.

In competition, the trio XGBoost + LightGBM + CatBoost in stacking is often unbeatable on tabular data. In production, CatBoost is preferred when inference speed and robustness to overfitting are priorities.

See Also

• Master the Calculation of Divisor Sums with Python: Complete Guide and Efficient Tips
• Create a Sliding Puzzle Game in Python: Complete Guide and Tips for Developers