Deep Deterministic Policy Gradient: Complete Guide — Actor-Critic for Continuous Actions

Summary — DDPG (Deep Deterministic Policy Gradient) is an actor-critic reinforcement learning algorithm designed for continuous action spaces. Designed by Lillicrap et al. in 2015, it combines three fundamental ideas: a deterministic actor (which produces a precise action rather than a distribution), target networks to stabilize learning, and a replay buffer to decorrelate experiences. It is the natural extension of DQN to environments where actions are continuous (e.g., steering angle, engine torque) rather than discrete (left/right/jump).

Mathematical Principle

1. The problem of continuous actions

DQN works for discrete actions because it computes Q(s,a) for each possible action and takes the max. With continuous actions (e.g., an angle between -180° and +180°), the max over a continuous space is impossible to compute directly.

2. Deterministic policy

DDPG uses a deterministic actor mu(s|theta^mu) that directly produces the optimal action (not a probability distribution). The critic Q(s,a|theta^Q) evaluates this action:

Actor: a = mu(s | theta^mu)  # deterministic action
Critic: Q(s, a | theta^Q)   # Q-value of the state-action pair

3. Critic update

Target networks are used to stabilize, as in DQN:

y_i = r_i + gamma · Q'(s_{i+1}, mu'(s_{i+1} | theta^{mu'}) | theta^{Q'})
Critic loss: L = 1/N · sum (y_i - Q(s_i, a_i | theta^Q))^2

4. Actor update

The actor is updated via policy gradient ascent, using the gradient of the critic:

grad_theta_mu J = E[grad_a Q(s,a | theta^Q) · grad_{theta^mu} mu(s | theta^mu)]

Intuitively: we ask the critic “how to improve the action?” (grad_a Q) and propagate this signal through the actor to update its parameters.

5. Soft update of target networks

Instead of periodically copying weights (hard update), DDPG updates the targets continuously via moving average (Polyak averaging):

theta' <-- tau · theta + (1 - tau) · theta'

Where tau << 1 (typically 0.001). This ensures smooth evolution of the targets, avoiding oscillations of standard DQN.

6. Exploration noise: Ornstein-Uhlenbeck

For exploration in continuous spaces, DDPG adds temporally correlated noise instead of the independent noise of epsilon-greedy:

dx_t = theta_ou · (mu_ou - x_t) · dt + sigma_ou · dW_t

OU noise has memory: if an action was pulled in a direction, the noise tends to persist in that direction, which is useful for physical controls (inertia).

Intuition

DDPG is like an apprentice chef. The critic is the food critic who tastes each dish and rates its quality out of 10. The actor is the chef who proposes recipes with continuous actions: amount of salt (from 0 to 5g), cooking time (from 0 to 60 min), temperature (from 100 to 250°C).

The food critic doesn’t just say “good” or “bad” — they give a precise score and suggest: “a bit more salt would improve the dish.” The chef adjusts the recipe based on these ratings. Both improve together over many trials.

Python Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np


class Actor(nn.Module):
    """Deterministic network: state -> continuous action."""
    def __init__(self, state_dim, action_dim, max_action):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, action_dim), nn.Tanh()
        )
        self.max_action = max_action

    def forward(self, state):
        return self.net(state) * self.max_action


class Critic(nn.Module):
    """Q-network: (state, action) -> scalar value."""
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim + action_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1)
        )

    def forward(self, state, action):
        return self.net(torch.cat([state, action], dim=1))


class OUNoise:
    """Ornstein-Uhlenbeck noise for exploration."""
    def __init__(self, action_dim, mu=0.0, theta=0.15, sigma=0.2):
        self.action_dim = action_dim
        self.mu = mu
        self.theta = theta
        self.sigma = sigma
        self.state = np.ones(self.action_dim) * self.mu

    def sample(self):
        dx = self.theta * (self.mu - self.state)
        dx += self.sigma * np.random.randn(self.action_dim)
        self.state += dx
        return self.state.copy()


class ReplayBuffer:
    """Experience buffer to break temporal correlation."""
    def __init__(self, capacity):
        self.buffer = []
        self.capacity = capacity
        self.pos = 0

    def push(self, state, action, reward, next_state, done):
        if len(self.buffer) < self.capacity:
            self.buffer.append(None)
        self.buffer[self.pos] = (state, action, reward, next_state, done)
        self.pos = (self.pos + 1) % self.capacity

    def sample(self, batch_size):
        batch = np.random.choice(len(self.buffer), batch_size, replace=False)
        states = torch.FloatTensor(np.array([self.buffer[i][0] for i in batch]))
        actions = torch.FloatTensor(np.array([self.buffer[i][1] for i in batch]))
        rewards = torch.FloatTensor(np.array([self.buffer[i][2] for i in batch])).unsqueeze(1)
        next_states = torch.FloatTensor(np.array([self.buffer[i][3] for i in batch]))
        dones = torch.FloatTensor(np.array([self.buffer[i][4] for i in batch])).unsqueeze(1)
        return states, actions, rewards, next_states, dones


# DDPG Agent
class DDPG:
    def __init__(self, state_dim, action_dim, max_action, tau=0.005, gamma=0.99, lr=3e-4):
        self.actor = Actor(state_dim, action_dim, max_action)
        self.actor_target = Actor(state_dim, action_dim, max_action)
        self.critic = Critic(state_dim, action_dim)
        self.critic_target = Critic(state_dim, action_dim)

        # Copy initial weights
        self.actor_target.load_state_dict(self.actor.state_dict())
        self.critic_target.load_state_dict(self.critic.state_dict())

        self.actor_opt = torch.optim.Adam(self.actor.parameters(), lr=lr)
        self.critic_opt = torch.optim.Adam(self.critic.parameters(), lr=lr)

        self.tau = tau
        self.gamma = gamma

    def select_action(self, state, noise=None):
        state = torch.FloatTensor(state).unsqueeze(0)
        with torch.no_grad():
            action = self.actor(state).cpu().numpy()[0]
        if noise is not None:
            action += noise.sample()
        return np.clip(action, -self.actor.max_action, self.actor.max_action)

    def update(self, replay_buffer, batch_size=256):
        states, actions, rewards, next_states, dones = replay_buffer.sample(batch_size)

        # Critic loss
        with torch.no_grad():
            target_actions = self.actor_target(next_states)
            target_q = self.critic_target(next_states, target_actions)
            target_q = rewards + (1 - dones) * self.gamma * target_q
        current_q = self.critic(states, actions)
        critic_loss = F.mse_loss(current_q, target_q)

        self.critic_opt.zero_grad()
        critic_loss.backward()
        self.critic_opt.step()

        # Actor loss
        actor_loss = -self.critic(states, self.actor(states)).mean()
        self.actor_opt.zero_grad()
        actor_loss.backward()
        self.actor_opt.step()

        # Soft update of targets
        for param, target in zip(self.actor.parameters(), self.actor_target.parameters()):
            target.data.copy_(self.tau * param.data + (1 - self.tau) * target.data)
        for param, target in zip(self.critic.parameters(), self.critic_target.parameters()):
            target.data.copy_(self.tau * param.data + (1 - self.tau) * target.data)

# Test on Pendulum-v1 (continuous action environment)
try:
    import gymnasium as gym
except ImportError:
    import gym as gym

env = gym.make('Pendulum-v1')
agent = DDPG(env.observation_space.shape[0], env.action_space.shape[0], env.action_space.high[0])
noise = OUNoise(env.action_space.shape[0])
buffer = ReplayBuffer(100000)

for episode in range(100):
    state, _ = env.reset()
    episode_reward = 0
    for step in range(200):
        action = agent.select_action(state, noise)
        next_state, reward, done, _, _ = env.step(action)
        buffer.push(state, action, reward, next_state, done)

        if len(buffer.buffer) > 1000:
            agent.update(buffer)

        state = next_state
        episode_reward += reward
        if done:
            break
    print('Episode ' + str(episode) + ' | Reward: ' + str(round(episode_reward, 1)))

env.close()

Hyperparameters

Advantages

1. Continuous actions: Unlike DQN, DDPG naturally handles continuous action spaces.
2. Sample efficient: The replay buffer reuses past experiences, improving efficiency.
3. Stable: Target networks + soft updates + replay buffer = stable learning compared to naive policy gradient methods.

Limitations

1. Hypersensitive to hyperparameters: DDPG is known to be unstable if hyperparameters are not well tuned (especially exploration noise magnitude).
2. Critic overestimation: Like DQN, the critic tends to overestimate Q-values, which degrades performance. TD3 (Twin Delayed DDPG) fixes this problem with two critics and a delayed actor update.
3. Limited exploration: OU noise is a heuristic addition to the action, not a structural exploration of the action space.
4. Not applicable to discrete actions: DDPG is designed for continuous actions. For discrete actions, use DQN or PPO.

4 Concrete Use Cases

1. Robotic Arm Control

DDPG trains a robotic arm to grasp objects. The actions are joint angles (continuous), rewards are based on distance to the object and grasp success.

2. Algorithmic Trading

DDPG’s continuous actions (proportion of the portfolio to allocate to each asset) are well-suited to algorithmic trading where decisions are continuous and not discrete.

3. Autonomous Driving

Steering and acceleration/braking control are continuous actions. DDPG learns to maintain lane and safe speed in simulated environments.

4. Energy Consumption Optimization

In data centers, DDPG continuously optimizes ventilation and cooling (temperatures, fan speeds) to minimize energy consumption while respecting thermal constraints.

Conclusion

DDPG opened the door to reinforcement learning for continuous action spaces. Although surpassed by TD3 and SAC (Soft Actor-Critic) in terms of stability and performance, its fundamental principles (deterministic actor-critic, target networks, replay buffer) remain the foundation of most modern algorithms for continuous actions.

Its direct successor, TD3, fixes DDPG’s main problems (Q overestimation, hyperparameter sensitivity) with only a few elegant modifications: two critics to reduce overestimation, a delay to update the actor less frequently, and smooth noise on target actions.

See Also

• Démystifiez le Module ‘abc’ en Python : Guide Complet avec Exemples Pratiques
• Maîtriser le Jeu ‘Lights Out’ avec Python : Tutoriel Complet et Code Source