MLP (Multi-Layer Perceptron): Complete Guide — Principles, Examples and Python Implementation

Summary

The Multi-Layer Perceptron (MLP), or multilayer perceptron, is the most fundamental and widely used neural network architecture in machine learning. It is a feed-forward type model composed of an input layer, one or more hidden layers, and an output layer, where each neuron in a layer is fully connected to all neurons in the next layer. Thanks to the backpropagation algorithm, the MLP is capable of learning complex nonlinear relationships between inputs and outputs, making it a powerful tool for classification, regression, and many other tasks.

This guide will explore in detail the mathematical principles, deep intuition, practical Python implementation, and concrete use cases of the MLP.

Mathematical principle of the Multi-Layer Perceptron

General architecture

An MLP consists of three types of layers:

• Input layer: receives the raw data. Each neuron corresponds to a feature of the input vector.
• Hidden layers: one or more intermediate layers that progressively transform inputs into higher-level representations. Each layer applies a linear transformation followed by a nonlinear activation function.
• Output layer: produces the final result vector, whose dimension depends on the task (one neuron for regression, as many as classes for multi-class classification).

Forward Pass

The heart of the MLP is the forward pass, i.e., the sequential computation of the output from the input. For each layer l, the transformation is written as:

$$h_l = \sigma(W_l \cdot h_{l-1} + b_l)$$

where:

• $h_{l-1}$ is the activation vector of the previous layer ($h_0 = x$, the input vector)
• $W_l$ is the weight matrix of layer l, of dimension (number_of_neurons_l, number_of_neurons_l-1)
• $b_l$ is the bias vector of layer l
• $\sigma$ is the nonlinear activation function
• $h_l$ is the resulting activation vector of layer l

The most common activation functions are:

• Sigmoid: $\sigma(z) = \frac{1}{1 + e^{-z}}$ — compresses output between 0 and 1, ideal for binary classification
• Hyperbolic tangent (tanh): $\sigma(z) = \tanh(z)$ — compresses between -1 and 1, zero-centered, often preferred over sigmoid in hidden layers
• ReLU (Rectified Linear Unit): $\sigma(z) = \max(0, z)$ — the most used today, simple and efficient, avoids the vanishing gradient problem
• Leaky ReLU: $\sigma(z) = \max(0.01z, z)$ — a ReLU variant that allows a small negative gradient

Loss function

The loss function measures the gap between the network’s predictions and the target values. The choice depends on the task:

• Mean Squared Error (MSE): $L = \frac{1}{n} \sum_{i=1}^{n} (y_i – \hat{y}_i)^2$ — used for regression
• Binary Cross-Entropy: $L = -\frac{1}{n} \sum_{i=1}^{n} [y_i \log(\hat{y}_i) + (1 – y_i) \log(1 – \hat{y}_i)]$ — for two-class classification
• Categorical Cross-Entropy: $L = -\sum_{i=1}^{n} \sum_{c=1}^{C} y_{i,c} \log(\hat{y}_{i,c})$ — for multi-class classification with softmax at the output

Backward Pass (Backpropagation)

Backpropagation is the key algorithm that enables training the MLP. The principle consists of computing the gradient of the loss function with respect to each weight in the network using the chain rule of differential calculus, going from the output layer back to the input layer.

For the output layer L, the gradient is:

$$\frac{\partial L}{\partial W_L} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_L} \cdot \frac{\partial z_L}{\partial W_L} = \delta_L \cdot h_{L-1}^T$$

where $\delta_L = \frac{\partial L}{\partial z_L}$ is the error term of the output layer.

For a hidden layer l, the error is propagated backward:

$$\delta_l = (W_{l+1}^T \cdot \delta_{l+1}) \odot \sigma’(z_l)$$

where $\odot$ denotes the element-wise product (Hadamard product) and $\sigma’$ is the derivative of the activation function.

Weight update

Once the gradients are computed, the weights are updated via gradient descent:

$$w \leftarrow w – \eta \cdot \frac{\partial L}{\partial w}$$

where $\eta$ is the learning rate, a critical hyperparameter that controls the size of the optimization steps. A rate that is too large causes instability, a rate that is too small slows convergence.

Intuition: how the MLP learns representations

The MLP does not just memorize associations between inputs and outputs. It progressively builds increasingly abstract representations through its layers.

Imagine a child learning to recognize animals:

1. First layer (simple features): like the child’s retina that detects edges, colors, and elementary shapes, the first hidden layer of the MLP extracts low-level features — simple linear combinations of the inputs, filtered through the activation function.
2. Second layer (patterns): like the child who combines features to recognize pointed ears, a long tail, or stripes, the second layer combines simple features into more complex patterns. It is capable of detecting nonlinear interactions between the first layer’s features.
3. Third layer and beyond (concepts): like the child who assembles patterns to form the concept of “cat” or “dog”, the deep layers of the MLP combine patterns into high-level concepts, directly useful for the final task.

It is this hierarchy of representations — features → patterns → ideas — that gives the MLP its expressive power. The universal approximation theorem (Cybenko, 1989) formally demonstrates that an MLP with a single hidden layer and a sufficient number of neurons can approximate any continuous function on a compact set, to arbitrary precision.

Python implementation of the Multi-Layer Perceptron

1. With scikit-learn: MLPClassifier

Scikit-learn offers a robust and easy-to-use implementation:

from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

# Data generation
X, y = make_classification(
    n_samples=5000, n_features=20, n_informative=15,
    n_redundant=5, n_classes=3, random_state=42
)

# Train / test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Data normalization (essential for MLPs)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Creation and training of the MLP
mlp = MLPClassifier(
    hidden_layer_sizes=(128, 64, 32),  # three hidden layers
    activation='relu',                  # ReLU activation function
    solver='adam',                      # Adam optimizer
    alpha=0.001,                        # L2 regularization
    learning_rate_init=0.001,           # learning rate
    max_iter=500,                       # maximum iterations
    random_state=42,
    early_stopping=True,                # early stopping
    validation_fraction=0.15,           # validation fraction
    verbose=True
)

mlp.fit(X_train_scaled, y_train)

# Evaluation
y_pred = mlp.predict(X_test_scaled)
print(classification_report(y_test, y_pred))
print(f"Training accuracy: {mlp.score(X_train_scaled, y_train):.4f}")
print(f"Test accuracy: {mlp.score(X_test_scaled, y_test):.4f}")
print(f"Number of iterations: {mlp.n_iter_}")
print(f"Final loss: {mlp.loss_:.6f}")

2. With TensorFlow / Keras

TensorFlow offers finer control over architecture and training:

import tensorflow as tf
from tensorflow.keras import layers, models, callbacks
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Data preparation
X, y = make_classification(
    n_samples=5000, n_features=20, n_informative=15,
    n_redundant=5, n_classes=3, random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# One-hot encoding for multi-class classification
y_train_cat = tf.keras.utils.to_categorical(y_train, num_classes=3)
y_test_cat = tf.keras.utils.to_categorical(y_test, num_classes=3)

# Model building
model = models.Sequential([
    layers.Input(shape=(20,)),
    layers.Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dropout(0.3),                          # dropout regularization
    layers.Dense(64, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dropout(0.2),
    layers.Dense(32, activation='relu'),
    layers.Dense(3, activation='softmax')         # multi-class output
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

model.summary()

# Callbacks: early stopping and learning rate reduction
early_stop = callbacks.EarlyStopping(
    monitor='val_loss', patience=15, restore_best_weights=True
)
reduce_lr = callbacks.ReduceLROnPlateau(
    monitor='val_loss', factor=0.5, patience=5, min_lr=1e-6
)

# Training
history = model.fit(
    X_train_scaled, y_train_cat,
    validation_split=0.15,
    epochs=200,
    batch_size=32,
    callbacks=[early_stop, reduce_lr],
    verbose=1
)

# Evaluation
loss, accuracy = model.evaluate(X_test_scaled, y_test_cat)
print(f"Test loss: {loss:.4f}")
print(f"Test accuracy: {accuracy:.4f}")

3. Visualizing layer activations

It is instructive to visualize what each layer has learned:

import matplotlib.pyplot as plt
import numpy as np
from tensorflow.keras.models import Model

# Intermediate model to extract activations
layer_outputs = [layer.output for layer in model.layers if isinstance(layer, layers.Dense)]
activation_model = Model(inputs=model.input, outputs=layer_outputs)

# Extracting activations on a test sample
activations = activation_model.predict(X_test_scaled[:10])

# Visualizing activation distributions
fig, axes = plt.subplots(1, len(activations), figsize=(15, 4))
for i, act in enumerate(activations):
    axes[i].hist(act.flatten(), bins=50, alpha=0.7, edgecolor='black')
    axes[i].set_title(f"Layer {i+1}: dim={act.shape[1]}")
    axes[i].set_xlabel("Activation value")
    axes[i].set_ylabel("Frequency")
plt.tight_layout()
plt.show()

# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(history.history['loss'], label='Training')
ax1.plot(history.history['val_loss'], label='Validation')
ax1.set_title('Loss evolution')
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss')
ax1.legend()

ax2.plot(history.history['accuracy'], label='Training')
ax2.plot(history.history['val_accuracy'], label='Validation')
ax2.set_title('Accuracy evolution')
ax2.set_xlabel('Epoch')
ax2.set_ylabel('Accuracy')
ax2.legend()
plt.tight_layout()
plt.show()

Hyperparameter guide

The choice of hyperparameters is critical for the performance of the Multi-Layer Perceptron. Here is a detailed guide:

Regularization strategies

• Dropout: randomly deactivates a percentage of neurons at each iteration, forcing the network to not overly depend on individual neurons.
• Early Stopping: interrupts training when performance on the validation set stops improving for patience epochs.
• L2 Regularization: adds a penalty proportional to the square of the weights to the loss function, limiting their magnitude.
• Batch Normalization: normalizes the activations of each layer, accelerating convergence and reducing sensitivity to initialization.

Advantages and limitations of the MLP

Advantages

• Universality: thanks to the universal approximation theorem, the MLP can model any continuous functional relationship
• Flexibility: applicable to classification, regression, unsupervised learning (autoencoders) and even reinforcement learning
• Relative simplicity: architecture that is easy to understand and implement compared to specialized architectures (CNN, RNN)
• Inference speed: once trained, the MLP makes extremely fast predictions
• No strong assumptions: unlike linear regression, the MLP does not assume a linear relationship between variables

Limitations

• Black box: difficult to interpret; understanding why the network makes a particular decision remains a major challenge
• Tabular data: for structured data, decision trees and random forests often outperform the MLP
• Sequential or spatial data: RNN/LSTM/Transformers and CNNs are better suited respectively for time series and images
• Sensitivity to scaling: inputs must be normalized; without preprocessing, convergence is compromised
• Risk of overfitting: with too many neurons or too many epochs, the MLP memorizes noise instead of learning the signal
• Sensitive initialization: poor weight initialization can lead to the vanishing or exploding gradient problem

4 concrete use cases of the Multi-Layer Perceptron

1. Bank fraud detection

An MLP trained on transaction features (amount, location, time, customer history) can detect anomalous behavior in real time. The hidden layers learn complex combinations of weak signals that, taken individually, would not be suspicious but that together reveal fraud. The imbalanced class problem is handled through class weighting or oversampling techniques (SMOTE).

2. Assisted medical diagnosis

By combining blood test results, demographic data, and medical history, an MLP can help predict disease risk (diabetes, cardiovascular diseases). Each layer of the network progressively builds an increasingly rich representation of the patient’s health status. Interpretability remains a challenge, but techniques like SHAP or LIME allow explaining predictions.

3. Energy demand forecasting

An MLP can predict electricity consumption for a region based on temperature, day of the week, time of year, and economic activity. The nonlinear interactions between these variables (e.g., the impact of temperature differs on weekdays and weekends) are naturally captured by the network’s nonlinear hidden layers.

4. Basic recommendation system

By encoding user preferences and product features as dense vectors, an MLP can learn to predict the rating a user would give a product. The early layers capture simple affinities, while the deeper layers discover complex and contextual preferences. This is the basic principle behind modern recommendation systems, although current architectures use embeddings and attention layers.

See also

• Maîtrisez le Mahjong avec Python : Guide Complet pour Développeurs et Enthousiastes
• Maîtriser les Graphes en Grille avec Python : Guide Complet et Astuces d’Optimisation