Policy Gradients & PPO

The objective is the expected return of trajectories $\tau = (s_0, a_0, s_1, a_1, \dots)$ generated by this policy:

$J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}}\left[\, R(\tau) \,\right], \qquad R(\tau) = \sum_{t=0}^{T} \gamma^{t}\, r_{t+1}.$

We want to climb $J$. The difficulty is that the distribution we sample from depends on the very parameters we are differentiating, the environment's dynamics sit between $\theta$ and the reward, and we cannot differentiate through them. The policy gradient theorem is the elegant way around this.

The policy-gradient theorem and REINFORCE

Using the "log-derivative trick" ($\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta$), the gradient of the objective takes a form that requires no gradient of the environment at all:

$\nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}}\left[\, \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t)\, R(\tau) \,\right].$

This is the REINFORCE estimator. Read it as a rule with a very intuitive shape:

1. 1Roll out the current policySample one or more full trajectories by acting in the environment.
2. 2Score each action's log-probability gradientCompute grad log pi(a_t | s_t), the direction in parameter space that makes that action more likely.
3. 3Weight by the returnMultiply each action's gradient by how much total reward the trajectory earned.
4. 4Average and stepTake the mean over samples and nudge theta in that direction (gradient ascent).

The high-variance problem

REINFORCE is unbiased but extremely noisy. Three forces conspire against it:

• Whole-trajectory credit. Every action in a trajectory is scaled by the same total return $R(\tau)$, even actions taken long after the reward was determined. Good actions get blamed for later bad luck and vice versa.
• Absolute-magnitude weighting. If all returns are large and positive (say between 990 and 1000), every action's probability is pushed up; the tiny differences that actually distinguish good from bad drown in the common offset.
• Monte Carlo sampling. Returns depend on many random transitions and action samples, so the estimate swings wildly from batch to batch.

The practical symptom is training that is slow, unstable, and hungry for enormous numbers of samples. Two ideas tame it.

Fix 1: causality (reward-to-go)

An action cannot affect rewards that already happened. So replace the full-trajectory return with the reward-to-go, only the rewards that came after the action:

$\nabla_{\theta} J(\theta) = \mathbb{E}\left[\, \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) \sum_{t'=t}^{T} \gamma^{t'-t} r_{t'+1} \,\right].$

This is still unbiased and strictly lower variance, because we stop crediting each action with rewards it could not have caused.

Fix 2: baselines and the advantage function

We can subtract any function of the state $b(s)$ (a baseline) from the weight without changing the gradient's expectation, because $\mathbb{E}[\nabla_\theta \log \pi_\theta(a\mid s)\, b(s)] = 0$. The variance-minimizing choice of baseline is (close to) the state value $V(s)$. Using reward-to-go with $V(s)$ as the baseline yields the advantage function:

$A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s).$

The advantage answers a sharper question: how much better than average is this action in this state? Positive advantage means "better than the policy's typical behavior here, do it more"; negative means "worse, do it less." Substituting it gives the modern actor–critic gradient:

$\nabla_{\theta} J(\theta) = \mathbb{E}\left[\, \sum_{t} \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t)\, A^{\pi}(s_t, a_t) \,\right].$

Here a learned value estimate $V(s)$ (the critic) supplies the baseline while the policy (the actor) is updated, hence "actor–critic." Centering the signal around zero is exactly what removes the "everything gets pushed up" pathology.

PPO: taking bigger steps without falling over

Even with advantages, plain policy gradients are fragile: one overly large update can push the policy into a bad region from which its own (now-bad) data cannot recover. We would like to reuse each batch of data for several gradient steps for efficiency, but naively doing so moves the policy far from the one that collected the data, and the gradient estimate becomes invalid.

PPO's answer is to optimize a surrogate objective built on the probability ratio between the new and old policies:

$r_t(\theta) = \frac{\pi_{\theta}(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}.$

A ratio above 1 means the new policy makes that action more likely than the data-collecting policy did. The clipped surrogate objective is:

$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\, \min\left( r_t(\theta)\, \hat{A}_t,\; \operatorname{clip}\!\big(r_t(\theta),\, 1-\epsilon,\, 1+\epsilon\big)\, \hat{A}_t \right) \right].$

The clip range $\epsilon$ is small (commonly $0.1$ or $0.2$). Here is why the $\min$-of-clipped construction stabilizes training:

1. 1When advantage is positiveWe want to raise the action's probability, so the ratio grows above 1. Clipping caps the reward of the objective at ratio = 1 + epsilon, so once the policy has moved 'enough' there is no further incentive to push harder.
2. 2When advantage is negativeWe want to lower the probability, so the ratio drops below 1. Clipping floors it at 1 - epsilon, again removing the incentive for an enormous corrective step.
3. 3The min() keeps it pessimisticBy taking the minimum of the clipped and unclipped terms, PPO never lets a large ratio inflate the objective; it only ever discards improvement that would come from moving too far.
4. 4ResultYou can safely run several epochs of minibatch updates on the same rollout data, because the objective refuses to reward straying far from theta_old.

A tiny numeric feel for clipping

Suppose an action has positive advantage $\hat{A}_t = +2$, the clip parameter is $\epsilon = 0.2$, and after some updates the ratio has grown to $r_t(\theta) = 1.5$ (the new policy makes this action 50% more likely than the old one). The unclipped term is $1.5 \times 2 = 3.0$, but the clipped term is $\operatorname{clip}(1.5, 0.8, 1.2)\times 2 = 1.2 \times 2 = 2.4$. PPO takes the minimum, $2.4$, so the gradient contribution behaves as if the ratio were pinned at $1.2$, the policy gets no extra credit for having moved past the trust band, and the incentive to move even further vanishes.

Further reading

• Schulman et al., "Proximal Policy Optimization Algorithms" (2017), the PPO paper and the clipped objective: https://arxiv.org/abs/1707.06347
• OpenAI Spinning Up, "Vanilla Policy Gradient" and "PPO", derivations plus reference implementations: https://spinningup.openai.com/en/latest/algorithms/ppo.html
• Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), Chapter 13, policy-gradient methods and the policy-gradient theorem: http://incompleteideas.net/book/the-book.html
• Schulman et al., "High-Dimensional Continuous Control Using Generalized Advantage Estimation" (GAE), the advantage estimator PPO usually pairs with: https://arxiv.org/abs/1506.02438
• UC Berkeley CS285, lectures on policy gradients and actor–critic methods: https://rail.eecs.berkeley.edu/deeprlcourse/