PPO: Proximal Policy Optimization — Complete Guide

Summary

PPO (Proximal Policy Optimization) is the most widely adopted reinforcement learning algorithm in 2026, both in industry and academic research. Developed by John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov at OpenAI in 2017, PPO quickly supplanted previous methods thanks to its elegant combination of implementation simplicity and remarkable performance.

Unlike classical policy gradient methods such as TRPO (Trust Region Policy Optimization), PPO does not impose complex optimization constraints requiring costly Fisher matrix computations. Instead, it uses a clipped surrogate objective that implicitly limits policy updates. This approach is both easier to implement and more effective in practice.

PPO today serves as the backbone of many advanced AI systems, including conversational agents, autonomous robots, and industrial control systems. Its popularity is explained by a rare balance of robustness, ease of tuning, and computational efficiency.

Mathematical Principle of PPO

The Policy Ratio

At the heart of PPO lies the policy ratio, which measures how the new policy compares to the old policy for a given action:

$$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{old}}(a_t|s_t)}$$

This ratio quantifies the relative probability difference between the current policy $\pi_\theta$ and the old policy $\pi_{\text{old}}$ that generated the data. If $r_t(\theta) > 1$, the new policy assigns more probability to this action than the old one. If $r_t(\theta) < 1$, the opposite. When $r_t(\theta) = 1$, both policies are identical for this action.

The advantage of this ratio is that it is scale-invariant: it does not depend on the absolute magnitude of probabilities, only on their relative ratio. This makes it particularly robust to numerical variations during training.

The Clipped Surrogate Objective

The main contribution of PPO is its clipped surrogate objective. Instead of directly maximizing the expectation of advantages weighted by the ratio — which can lead to excessively large updates — PPO clips the ratio to an interval $[1-\varepsilon, 1+\varepsilon]$:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\left(r_t(\theta) \cdot \hat{A}_t,\; \text{clip}(r_t(\theta), 1-\varepsilon, 1+\varepsilon) \cdot \hat{A}_t\right) \right]$$

In this formula, $\hat{A}_t$ represents the estimated advantage at time $t$, and $\varepsilon$ is a hyperparameter (typically $0.2$) that determines the maximum allowed magnitude for the policy ratio.

The clipping mechanism works as follows: the minimum between two terms is selected. The first term is the standard policy gradient objective, while the second term is the same objective but with the clipped ratio. By taking the minimum, we ensure that the effective ratio can never stray too far from 1.

Concretely:

• If the advantage is positive (the action was good), clipping prevents $r_t(\theta)$ from increasing beyond $1+\varepsilon$. We therefore encourage the policy to choose this action more often, but without excess.
• If the advantage is negative (the action was bad), clipping prevents $r_t(\theta)$ from dropping below $1-\varepsilon$. We discourage the policy from choosing this action, but without eliminating it entirely.

This approach is remarkably elegant: it imposes no hard constraints on optimization, but implicitly penalizes overly aggressive updates.

Generalized Advantage Estimation (GAE)

To estimate the advantages $\hat{A}_t$, PPO generally uses Generalized Advantage Estimation (GAE), introduced by Schulman et al. in 2015. GAE combines several advantage estimates at different time horizons according to a parameter $\lambda$:

$$\hat{A}t^{\text{GAE}(\gamma,\lambda)} = \sum$$}^{\infty} (\gamma\lambda)^l \cdot \delta_{t+l

where $\delta_t = r_t + \gamma \cdot V(s_{t+1}) – V(s_t)$ is the temporal difference error (TD error).

The parameter $\lambda \in [0,1]$ controls the bias-variance tradeoff of the estimation:

• $\lambda$ close to 0: biased but low-variance estimate. Only the immediate TD error is considered, which is stable but may miss long-term rewards.
• $\lambda$ close to 1: low-bias but high-variance estimate. Nearly the entire trajectory is considered, which is precise but unstable.
• In practice, $\lambda = 0.95$ offers an excellent tradeoff.

The parameter $\gamma \in [0,1]$ is the standard discount factor in reinforcement learning, controlling the importance of future rewards relative to immediate rewards.

The Total Cost Function

The complete PPO objective combines three components:

$$L(\theta) = L^{\text{CLIP}}(\theta) + c_1 \cdot L^V(\theta) – c_2 \cdot S[\pi_\theta]$$

Where:

• $L^{\text{CLIP}}(\theta)$ is the clipped surrogate objective described above.
• $L^V(\theta)$ is the value function loss, typically a mean squared error between the predicted value $V(s_t)$ and the target $r_t + \gamma \cdot V(s_{t+1})$. The coefficient $c_1$ (usually $0.5$) controls the relative importance of this component.
• $S[\pi_\theta]$ is the policy entropy term. Maximizing entropy encourages exploration by keeping the action distribution sufficiently diverse. The coefficient $c_2$ (usually $0.01$) adjusts the strength of this regularization.

Intuition: why PPO works so well

Imagine a student learning a sports technique from a teacher. If the student tries to radically change their movement after a single lesson, they risk destroying everything they had previously learned. Conversely, if they never change their movement, they will not progress.

PPO finds exactly this middle ground. The clipping mechanism acts as a safeguard: the student can improve their technique, but not too abruptly. Each policy update stays within a “trust zone” around the previous policy.

By comparison, classical policy gradient methods (like REINFORCE) have no protection against excessive updates. A strongly positive advantage can cause the probability of an action to increase exponentially, leading to a premature deterministic policy that can no longer explore effectively. It’s as if the student, after succeeding at a single movement, decided to never do anything different — at the risk of missing out on superior techniques.

PPO avoids this pitfall through its clipping. Even if a action receives a very high advantage, the probability of that action can only increase by a factor of $(1+\varepsilon)$ at most per iteration. This guarantees a smooth and stable evolution of the policy, which is especially important in complex environments where reward surfaces are irregular and noisy.

Python Implementation with PyTorch

Here is a complete implementation of a PPO agent for the CartPole-v1 environment from Gymnasium:

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import gymnasium as gym
from torch.distributions import Categorical
import os

# ──────────────────────────────────────────────
# Configuration
# ──────────────────────────────────────────────
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
GAMMA = 0.99
GAE_LAMBDA = 0.95
CLIP_EPSILON = 0.2
VALUE_COEFF = 0.5
ENTROPY_COEFF = 0.01
LEARNING_RATE = 3e-4
N_STEPS = 2048
N_EPOCHS = 10
BATCH_SIZE = 64
MAX_EPISODES = 500


# ──────────────────────────────────────────────
# Actor-Critic Architecture
# ──────────────────────────────────────────────
class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.Tanh(),
            nn.Linear(128, 128),
            nn.Tanh(),
        )
        self.actor_head = nn.Linear(128, action_dim)
        self.critic_head = nn.Linear(128, 1)

    def forward(self, state):
        features = self.shared(state)
        action_logits = self.actor_head(features)
        state_value = self.critic_head(features)
        return action_logits, state_value

    def act(self, state):
        logits, value = self.forward(state)
        distribution = Categorical(logits=logits)
        action = distribution.sample()
        log_prob = distribution.log_prob(action)
        return action.item(), log_prob, value.squeeze(-1)

    def evaluate(self, state, action):
        logits, value = self.forward(state)
        distribution = Categorical(logits=logits)
        log_prob = distribution.log_prob(action)
        entropy = distribution.entropy()
        return log_prob, value.squeeze(-1), entropy


# ──────────────────────────────────────────────
# GAE Computation (Generalized Advantage Estimation)
# ──────────────────────────────────────────────
def compute_gae(rewards, values, dones, gamma=GAMMA, lam=GAE_LAMBDA):
    advantages = []
    gae = 0.0
    values = values + [0.0]  # Terminal state value

    for t in reversed(range(len(rewards))):
        delta = (rewards[t] + gamma * values[t + 1] * (1 - dones[t]) - values[t])
        gae = delta + gamma * lam * (1 - dones[t]) * gae
        advantages.insert(0, gae)

    returns = [adv + val for adv, val in zip(advantages, values[:-1])]
    return advantages, returns


# ──────────────────────────────────────────────
# PPO Training
# ──────────────────────────────────────────────
def train_ppo():
    env = gym.make("CartPole-v1")
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n
    env.close()

    policy = ActorCritic(state_dim, action_dim).to(DEVICE)
    optimizer = optim.Adam(policy.parameters(), lr=LEARNING_RATE)

    rewards_history = []
    best_mean_reward = -float("inf")

    global_step = 0

    for episode in range(MAX_EPISODES):
        env = gym.make("CartPole-v1")
        state, _ = env.reset()
        ep_reward = 0

        # Data collection
        roll_states, roll_actions, roll_log_probs = [], [], []
        roll_rewards, roll_dones, roll_values = [], [], []

        for step in range(N_STEPS):
            state_t = torch.tensor(state, dtype=torch.float32).to(DEVICE).unsqueeze(0)
            action, log_prob, value = policy.act(state_t)

            next_state, reward, terminated, truncated, _ = env.step(action)
            done = terminated or truncated

            roll_states.append(state)
            roll_actions.append(action)
            roll_log_probs.append(log_prob.item())
            roll_rewards.append(reward)
            roll_dones.append(float(done))
            roll_values.append(value.item())

            state = next_state
            ep_reward += reward
            global_step += 1

            if done:
                state, _ = env.reset()

        env.close()

        # GAE computation
        advantages, returns = compute_gae(
            roll_rewards, roll_values, roll_dones
        )

        advantages = torch.tensor(advantages, dtype=torch.float32, device=DEVICE)
        returns = torch.tensor(returns, dtype=torch.float32, device=DEVICE)
        states = torch.tensor(np.array(roll_states), dtype=torch.float32, device=DEVICE)
        actions = torch.tensor(np.array(roll_actions), device=DEVICE)
        old_log_probs = torch.tensor(roll_log_probs, dtype=torch.float32, device=DEVICE)

        # Advantage normalization
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

        # Policy update over N epochs
        dataset_size = len(states)
        indices = np.arange(dataset_size)

        for epoch in range(N_EPOCHS):
            np.random.shuffle(indices)

            for start in range(0, dataset_size, BATCH_SIZE):
                end = start + BATCH_SIZE
                batch_idx = indices[start:end]

                b_states = states[batch_idx]
                b_actions = actions[batch_idx]
                b_old_log_probs = old_log_probs[batch_idx]
                b_advantages = advantages[batch_idx]
                b_returns = returns[batch_idx]

                # Evaluation with current policy
                new_log_probs, values, entropies = policy.evaluate(b_states, b_actions)

                # Policy ratio
                ratios = torch.exp(new_log_probs - b_old_log_probs)

                # Clipped surrogate objective
                surr1 = ratios * b_advantages
                clipped_ratios = torch.clamp(ratios, 1 - CLIP_EPSILON, 1 + CLIP_EPSILON)
                surr2 = clipped_ratios * b_advantages
                policy_loss = -torch.min(surr1, surr2).mean()

                # Value function loss
                value_loss = nn.MSELoss()(values, b_returns)

                # Entropy term
                entropy_bonus = entropies.mean()

                # Total loss
                loss = policy_loss + VALUE_COEFF * value_loss - ENTROPY_COEFF * entropy_bonus

                optimizer.zero_grad()
                loss.backward()
                nn.utils.clip_grad_norm_(policy.parameters(), max_norm=0.5)
                optimizer.step()

        rewards_history.append(ep_reward)
        mean_reward = np.mean(rewards_history[-100:])

        if episode % 50 == 0:
            print(f"Episode {episode} | Reward: {ep_reward:.0f} | "
                  f"100-episode average: {mean_reward:.1f}")

        # Save best model
        if mean_reward > best_mean_reward:
            best_mean_reward = mean_reward
            save_path = os.path.join(
                os.path.dirname(os.path.abspath(__file__)), "best_ppo_cartpole.pt"
            )
            torch.save(policy.state_dict(), save_path)

        if mean_reward >= 490:
            print(f"\nConvergence reached! Average: {mean_reward:.1f}")
            break

    print(f"\nTraining complete. Best average: {best_mean_reward:.1f}")


if __name__ == "__main__":
    train_ppo()

This implementation illustrates the essential components of PPO:

1. Shared Actor-Critic network: the hidden layers are shared between the actor and the critic, enabling richer and more efficient representation learning.
2. GAE computation: the compute_gae function implements the recursive formula with the $\lambda$ parameter, providing stable advantage estimates.
3. Ratio clipping: the heart of PPO — torch.clamp(ratios, 1 - CLIP_EPSILON, 1 + CLIP_EPSILON) — ensures ratios stay within the interval $[0.8\ ;\ 1.2]$.
4. Advantage normalization: a standard practice that considerably stabilizes training by centering and scaling advantages.
5. Gradient clipping: clip_grad_norm_ protects against exploding gradients, a frequent problem in reinforcement learning.

Key PPO Hyperparameters

Hyperparameter tuning is crucial to getting good performance with PPO. Here are the most important parameters:

Practical advice: start with the default values. PPO is remarkably robust compared to other reinforcement learning algorithms. If performance plateaus, increase n_steps before touching other hyperparameters.

Advantages and Limitations of PPO

Advantages

• Simplicity of implementation: unlike TRPO which requires Fisher matrix inversions, PPO can be coded in a few dozen lines. This simplicity considerably reduces the risk of bugs.
• Remarkable stability: the clipping mechanism prevents destructive policy updates, leading to regular and predictable learning curves.
• Computational efficiency: PPO is significantly faster than TRPO in terms of training time. Updates are done via simple stochastic gradient descent.
• Robustness to hyperparameters: PPO works reasonably well with default values on a wide variety of environments, making it an excellent starting point.
• Support for continuous and discrete action spaces: the same algorithm applies to both cases, simplifying its use.

Limitations

• Sample inefficiency: like all online algorithms, PPO requires a large number of interactions with the environment. In real-world contexts (robotics, industrial control), this can be prohibitive in terms of time and resources.
• Sensitive to the choice of n_epochs: if the ratio between the amount of data collected and the number of update epochs is poorly calibrated, the algorithm can either under-utilize the data (insufficient learning) or use it too many times (overfitting on stale data).
• No strong theoretical guarantee: unlike TRPO which has monotonicity guarantees, PPO is a heuristic. Clipping does not formally guarantee that performance will not decrease at each step.
• May converge to local sub-optima: in environments with very sparse rewards or complex reward landscapes, PPO can get stuck in suboptimal policies.

Concrete Use Cases

1. Alignment of large language models (RLHF)

PPO is the foundational algorithm behind Reinforcement Learning from Human Feedback (RLHF), used to align language models with human preferences. In this context, PPO trains a language model to produce responses preferred by human annotators. The clipping mechanism is especially important here: LLMs are enormous policies, and overly aggressive updates could severely degrade linguistic quality. PPO was used by OpenAI to train ChatGPT, and remains the industry standard in 2026.

2. Humanoid robot control

Research teams such as those at Boston Dynamics and Berkeley use PPO to train locomotion policies for humanoid robots. PPO excels in these tasks because it can handle high-dimensional continuous action spaces — a humanoid robot often has more than 30 degrees of freedom, requiring fine and coordinated control. Clipping ensures that each update does not destroy already acquired skills (walking, standing upright).

3. Video games and game agents

PPO is used in classic benchmarks like Atari, MuJoCo, and more complex game environments like Dota 2 and StarCraft II (OpenAI Five). Even though AlphaStar used more sophisticated methods, PPO remains a reference approach for learning game strategies thanks to its ability to learn adaptive stochastic policies.

4. Algorithmic trading

Hedge funds and fintech companies are experimenting with PPO for automated trading strategies. PPO’s clipping is particularly well suited to this domain: financial markets being highly non-stationary, a policy that evolves too aggressively can quickly lose money. PPO’s progressive updates allow continuous and cautious adjustment of the trading strategy.

See also

• Trouvez le Plus Petit Multiple en Python : Guide Complet et Astuces d’Optimisation
• Implémentez l’algorithme de hachage SHA-1 en Python : Guide étape par étape