Bernoulli Naive Bayes: Complete Guide — Principles, Examples, and Python Implementation

Bernoulli Naive Bayes is one of the most efficient supervised classification algorithms for processing strictly binary data. In this complete guide, we will explore its mathematical foundations, deep intuition, and practical implementation in Python with scikit-learn.

Mathematical Principle

Bernoulli Naive Bayes is based on a fundamental assumption: each feature in the dataset follows a Bernoulli distribution. Unlike Gaussian Naive Bayes — which models continuous variables — or Multinomial Naive Bayes — which handles integer counts, Bernoulli Naive Bayes only knows two possible states: 0 or 1, absent or present, yes or no.

The Bernoulli Model

For each feature xᵢ and each class y, the model estimates the conditional probability:

P(xᵢ | y) = θᵢ,y^xᵢ × (1 − θᵢ,y)^(1 − xᵢ)

where:

• θᵢ,y is the probability that feature i is present (equal to 1) given that the observation belongs to class y.
• xᵢ ∈ {0, 1} is the binary value of feature i.
• If xᵢ = 1, the probability is θᵢ,y.
• If xᵢ = 0, the probability is 1 − θᵢ,y.

Estimating θᵢ,y Parameters

Given a training set, θᵢ,y is estimated by the frequency of occurrence of feature i in observations of class y:

θᵢ,y = (number of observations in class y where xᵢ = 1) / (total number of observations in class y)

Laplace Smoothing

A major problem occurs when a feature never appears in a given class: the maximum likelihood estimator would produce θᵢ,y = 0, which would completely zero out the probability product (recall: the model assumes conditional independence of features). To solve this catastrophic problem, Laplace smoothing is applied:

θᵢ,y = (Nᵢ,y + α) / (N_y + 2α)

where:

• Nᵢ,y is the number of observations in class y with xᵢ = 1.
• N_y is the total number of observations in class y.
• α is the smoothing parameter (default α = 1, which corresponds to classical Laplace smoothing).

The denominator contains 2α because the Bernoulli distribution has exactly two outcomes (0 and 1), unlike the multinomial case where the denominator would be N_y + kα with k being the number of modalities.

Decision Rule

The prediction for an observation x is obtained by computing for each class y:

P(y | x) ∝ P(y) × ∏ᵢ P(xᵢ | y)

The predicted class is the one that maximizes this posterior probability:

ŷ = argmax_y [ log P(y) + Σᵢ log P(xᵢ | y) ]

In practice, we work in logarithmic space to avoid floating-point underflow issues when multiplying many small probabilities.

Intuition

Imagine you need to sort emails into two categories: spam or non-spam. Rather than counting how many times the word “free” appears — which is what Multinomial Naive Bayes would do — Bernoulli Naive Bayes simply asks: is the word “free” present or absent in this email?

This binary approach simplifies the model considerably. Each feature becomes a closed-ended question:

• Is the word “urgent” present? → Yes (1) / No (0)
• Does the recipient contain “.edu”? → Yes (1) / No (0)
• Does the message contain more than 3 exclamation marks? → Yes (1) / No (0)

Why Not Multinomial Naive Bayes Instead?

That’s an excellent question. Multinomial Naive Bayes is more suitable when the frequency of a word or event is informative. For example, if an email contains the word “winner” 50 times, it is almost certainly spam. Bernoulli Naive Bayes would ignore this valuable information: it would simply record present = 1.

However, Bernoulli Naive Bayes excels in situations where:

1. Presence matters more than frequency. Whether a word appears one or a hundred times, the qualitative information is the same.
2. Data is naturally binary. For example, a patient profile in medicine: presence or absence of symptoms.
3. The feature space is very high-dimensional (thousands or tens of thousands of words). Binarization reduces noise caused by extreme counts.

Complete Python Implementation

Here is a complete and executable implementation of Bernoulli Naive Bayes with scikit-learn, including synthetic data, a comparison with Multinomial Naive Bayes, and visual analysis.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.naive_bayes import BernoulliNB, MultinomialNB
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import (
    confusion_matrix, classification_report,
    accuracy_score, ConfusionMatrixDisplay
)
from sklearn.feature_extraction.text import CountVectorizer

# ============================================================
# 1. Synthetic Binary Data Generation
# ============================================================
np.random.seed(42)

# Create a binary dataset with 3 classes
n_samples = 600
n_features = 15

# Define distinct profiles for each class
theta_class0 = np.array([0.9, 0.8, 0.7, 0.1, 0.1, 0.2, 0.3, 0.1, 0.9, 0.8, 0.7, 0.1, 0.2, 0.3, 0.1])
theta_class1 = np.array([0.1, 0.2, 0.1, 0.8, 0.9, 0.8, 0.1, 0.9, 0.1, 0.2, 0.1, 0.8, 0.9, 0.8, 0.1])
theta_class2 = np.array([0.5, 0.1, 0.9, 0.5, 0.1, 0.9, 0.8, 0.1, 0.5, 0.9, 0.1, 0.9, 0.1, 0.9, 0.8])

X_list = []
y_list = []

n_per_class = n_samples // 3
for theta, label in zip([theta_class0, theta_class1, theta_class2], [0, 1, 2]):
    for _ in range(n_per_class):
        # Each feature follows a Bernoulli distribution
        row = np.random.binomial(1, theta)
        X_list.append(row)
        y_list.append(label)

X = np.array(X_list)
y = np.array(y_list)

print(f"Dataset: {n_samples} samples, {n_features} binary features")
print(f"Classes: {np.unique(y, return_counts=True)}")

# ============================================================
# 2. Training / Test Split
# ============================================================
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"\nTraining: {X_train.shape[0]} samples")
print(f"Test:     {X_test.shape[0]} samples")

# ============================================================
# 3. Bernoulli Naive Bayes with Different Alphas
# ============================================================
# Evaluate the impact of the smoothing parameter alpha
alphas = [0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
train_scores = []
test_scores = []

for alpha in alphas:
    bnb = BernoulliNB(alpha=alpha)
    bnb.fit(X_train, y_train)
    train_scores.append(bnb.score(X_train, y_train))
    test_scores.append(bnb.score(X_test, y_test))

# Visualizing the impact of alpha
plt.figure(figsize=(10, 6))
plt.plot(alphas, train_scores, 'o-', label="Training Score", linewidth=2, markersize=8)
plt.plot(alphas, test_scores, 's-', label="Test Score", linewidth=2, markersize=8)
plt.xscale('log')
plt.xlabel("Smoothing Parameter Alpha (Log Scale)", fontsize=12)
plt.ylabel("Accuracy", fontsize=12)
plt.title("Impact of Laplace Smoothing on Bernoulli Naive Bayes", fontsize=14, fontweight='bold')
plt.legend(fontsize=11)
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('bnb_alpha_impact.png', dpi=150, bbox_inches='tight')
plt.close()
print("Alpha impact chart saved: bnb_alpha_impact.png")

# ============================================================
# 4. Best Model and Classification Report
# ============================================================
best_alpha = alphas[np.argmax(test_scores)]
print(f"\nBest alpha: {best_alpha} (test accuracy: {max(test_scores):.4f})")

bnb_best = BernoulliNB(alpha=best_alpha)
bnb_best.fit(X_train, y_train)
y_pred = bnb_best.predict(X_test)

print("\n" + "="*60)
print("CLASSIFICATION REPORT — BERNOULLI NAIVE BAYES")
print("="*60)
print(classification_report(y_test, y_pred))

# ============================================================
# 5. Confusion Matrix
# ============================================================
cm = confusion_matrix(y_test, y_pred)
fig, ax = plt.subplots(figsize=(8, 6))
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Class 0", "Class 1", "Class 2"])
disp.plot(ax=ax, cmap='Blues', values_format='d')
ax.set_title('Confusion Matrix — Bernoulli Naive Bayes', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.savefig('bnb_confusion_matrix.png', dpi=150, bbox_inches='tight')
plt.close()
print("Confusion matrix saved: bnb_confusion_matrix.png")

# ============================================================
# 6. Comparison with Multinomial Naive Bayes
# ============================================================
print("\n" + "="*60)
print("COMPARISON: BERNOULLI vs MULTINOMIAL")
print("="*60)

# For MultinomialNB, binary data is still valid
# (0 and 1 are also non-negative integers)
mnb = MultinomialNB(alpha=best_alpha)
mnb.fit(X_train, y_train)
y_pred_mnb = mnb.predict(X_test)

print(f"BernoulliNB   — Test Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"MultinomialNB — Test Accuracy: {accuracy_score(y_test, y_pred_mnb):.4f}")

print("\nMultinomialNB Report:")
print(classification_report(y_test, y_pred_mnb))

# Cross-validation for robust comparison
bnb_cv = cross_val_score(BernoulliNB(alpha=best_alpha), X, y, cv=5, scoring='accuracy')
mnb_cv = cross_val_score(MultinomialNB(alpha=best_alpha), X, y, cv=5, scoring='accuracy')

print(f"\nCross-validation (5-fold):")
print(f"BernoulliNB   : {bnb_cv.mean():.4f} (+/- {bnb_cv.std():.4f})")
print(f"MultinomialNB : {mnb_cv.mean():.4f} (+/- {mnb_cv.std():.4f})")

# ============================================================
# 7. Text Application with CountVectorizer(binary=True)
# ============================================================
print("\n" + "="*60)
print("TEXT CLASSIFICATION WITH BERNOULLI NAIVE BAYES")
print("="*60)

documents = [
    "purchase free offer special price unbeatable",
    "meeting tomorrow morning ten o'clock office",
    "win money quickly bank account",
    "project report delivery friday team",
    "congratulations you have been selected click here",
    "memo policy update company",
    "exceptional promotion last day don't miss out",
    "financial quarterly analysis report",
    "lottery jackpot millionaire instant fortune",
    "agenda monthly board administration"
]

labels_text = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0]  # 1 = spam, 0 = normal

# CountVectorizer with binary=True is essential for BernoulliNB
vectorizer = CountVectorizer(binary=True)
X_text = vectorizer.fit_transform(documents)
print(f"Vocabulary ({len(vectorizer.get_feature_names_out())} terms): {vectorizer.get_feature_names_out()}")
print(f"Binary matrix: {X_text.shape}")

# Training on text dataset
bnb_text = BernoulliNB()
bnb_text.fit(X_text, labels_text)
y_pred_text = bnb_text.predict(X_text)
print(f"\nText accuracy (training): {accuracy_score(labels_text, y_pred_text):.4f}")

# Display probabilities for a new document
new_docs = [
    "exceptional offer free click now",
    "annual results presentation finance department"
]
X_new = vectorizer.transform(new_docs)
probas = bnb_text.predict_proba(X_new)

for i, doc in enumerate(new_docs):
    pred = bnb_text.predict(X_new)[i]
    label = "SPAM" if pred == 1 else "NORMAL"
    print(f'\nDocument: "{doc}"')
    print(f"Prediction: {label}")
    print(f"Probabilities — Normal: {probas[i][0]:.4f}, Spam: {probas[i][1]:.4f}")

print("\n" + "="*60)
print("Implementation completed successfully!")
print("="*60)

This code is fully executable and demonstrates all aspects of Bernoulli Naive Bayes: synthetic binary data generation, exploration of the alpha hyperparameter, comparison with MultinomialNB, confusion matrices, and text classification with binarization via CountVectorizer(binary=True).

Hyperparameters

Advantages / Limitations

Advantages

• Extremely fast: Training amounts to counting and multiplying probabilities. Complexity is linear with respect to the number of samples and features: O(n × d) where n is the number of samples and d the number of features.
• Little data required: Thanks to Laplace smoothing, the model works correctly even with few observations per class.
• Interpretability: The probabilities θᵢ,y are directly readable: you know exactly how much each feature contributes to the decision.
• No overfitting with binary data: Unlike complex models, Bernoulli Naive Bayes generalizes well even with a large number of features.
• Perfect for binary data: It is the mathematically appropriate model when features are inherently binary (presence/absence, yes/no).
• Probabilistic predictions: The model directly produces predict_proba(), which is valuable for sorting, scoring, and nuanced decision-making.

Limitations

• Conditional independence assumption: Like all Naive Bayes models, Bernoulli Naive Bayes assumes that features are independent given the class. This assumption is almost always violated in practice (words in a text are not independent!). Paradoxically, the model still works well.
• Information loss from binarization: Reducing features to 0 or 1 discards frequency information. A word appearing 100 times and a word appearing 1 time receive the same treatment.
• Sensitive to correlated features: Strongly correlated features are counted multiple times, which biases probability estimation.
• Poorly calibrated probabilities: Predicted probabilities tend to be extreme (very close to 0 or 1), even when the model’s actual confidence should be moderate.

4 Concrete Use Cases

1. Spam Detection by Keyword Presence

The historical and most classic use case for Bernoulli Naive Bayes. A vocabulary of spam-indicator words is built (“free”, “win”, “urgent”, “click”) and each email is represented as a binary vector: is the word present or not? Binarization prevents a spammer from circumventing the filter by repeating a suspicious word hundreds of times.

2. Medical Diagnosis from Binary Symptoms

Does a patient have a fever? Nausea? Headaches? Each symptom is coded binary (present/absent). Bernoulli Naive Bayes can quickly classify the type of pathology from this binary profile. This is particularly well-suited for structured medical questionnaires where the response is systematically yes or no.

3. Behavioral Biometric Authentication

In security systems, a user’s behavior can be modeled by binary features: does the user log in from their usual device? At their usual time? From their usual location? Each deviation from the norm is a 1, each conformity a 0. The model efficiently and quickly detects behavioral anomalies.

4. Document Classification by Tag Presence

In natural language processing, web documents can be classified according to the HTML tags or metadata they contain: presence of a form (1/0), presence of <script> tags (1/0), presence of keywords in metadata (1/0). This binary approach is robust and fast for classifying large volumes of web pages.

See Also

• Mastering Maximum Arrangements in Python: Complete Guide and Tips for Developers
• Mastering Zebra Circles in Python: Complete Guide for Programming Enthusiasts