On-Policy vs Off-Policy RL

What does on-policy vs off-policy mean?

Every reinforcement learning agent does two things at once: it acts in the world (choosing actions to gather experience) and it learns (updating its estimate of how good actions are). The on-policy / off-policy distinction is about a single question — is the policy you are learning about the same as the policy you are using to act?

• On-policy methods evaluate and improve the same policy that generates the data. The agent learns the value of the policy it is actually following, exploration and all.
• Off-policy methods learn about a target policy $\pi$ (the one you ultimately care about, usually the greedy/optimal one) using data drawn from a different behavior policy $b$ (the one that actually picks actions, usually something more exploratory).

That second policy is the whole trick. Off-policy learning decouples acting from learning, which is why an off-policy agent can explore wildly while still learning the value of behaving optimally — and why it can learn from old data, demonstrations, or another agent entirely.

The canonical pair: SARSA vs Q-learning

The cleanest way to feel the difference is to compare the two most famous tabular control algorithms. They look almost identical — both are temporal-difference methods that bootstrap a Q-value toward a one-step target — but they differ in a single term: what they assume happens next.

That one change — $Q(s',a')$ versus $\max_{a'} Q(s',a')$ — is the entire on-policy / off-policy distinction in tabular form. See Q-learning for the full treatment.

Cliff Walking: where the difference becomes visible

The textbook demonstration is Cliff Walking (Sutton & Barto). An agent must cross a grid to a goal. Each step costs $-1$; stepping off the cliff edge costs $-100$ and resets the episode. The shortest path runs right along the cliff edge.

Why off-policy is powerful: the payoffs

Decoupling behavior from target unlocks capabilities on-policy methods simply cannot have:

The catch: importance sampling and instability

If off-policy is so capable, why use on-policy at all? Because the freedom comes with a bill.

When you learn about $\pi$ from data drawn under $b$, the experience is sampled from the wrong distribution. To get an unbiased estimate you must reweight each sample by how much more (or less) likely the target policy was to take that action — the importance sampling ratio:

For a multi-step return you multiply the per-step ratios, $\rho_{t:T} = \prod_{k=t}^{T} \rho_k$. This is unbiased, but the product can explode or vanish, giving these estimates enormous variance — a long trajectory where $\pi$ and $b$ disagree even mildly produces a wildly swinging weight.

The deadly triad

Variance is not the only hazard. Sutton & Barto name the deadly triad — the three ingredients that, combined, can make value estimates diverge to infinity:

1. Function approximation (e.g. a neural network instead of a table),
2. Bootstrapping (updating an estimate from another estimate, as in TD),
3. Off-policy learning (training on data from a different policy).

Any two are usually fine. All three together can blow up. This is exactly the regime Deep Q-Networks operate in — function approximation + bootstrapping + off-policy replay — which is why DQN needs stabilizers like a frozen target network and experience replay to tame the triad. On-policy methods sidestep the off-policy leg entirely, which is a large part of why they tend to be more stable, if less sample-efficient.

The full spectrum: how common algorithms line up

In practice “on-policy vs off-policy” is less a binary and more a spectrum of how far the data-generating policy is allowed to drift from the one being optimized.

PPO is the interesting middle case. It is usually called on-policy, yet it does reuse each batch of trajectories for several gradient epochs — technically off-policy. It gets away with this by forming the importance ratio $\rho_t = \pi_\theta(a_t\mid s_t)/\pi_{\text{old}}(a_t\mid s_t)$ and clipping it so the new policy can never stray far from the one that collected the data. That keeps $\pi$ and $b$ close enough that variance stays manageable — “near-on-policy” is the honest label.

On-policy vs off-policy in LLM post-training

The distinction has moved to the center of frontier AI. Reinforcement learning for language models — RLHF, RLVR, GRPO — is dominated by on-policy methods: PPO and GRPO both sample fresh completions from the current model, score them, and update. On-policy is favored here because it is stable and because the “environment” (a reward model or verifier) is cheap to query relative to a real robot.

But the same sample-efficiency pressure that drives control research is now appearing in LLMs. Generating completions from a giant model is expensive, so reusing samples is tempting. Recent work shows off-policy RL can match or beat on-policy GRPO for LLM reasoning at much higher sample efficiency, and a wave of methods (e.g. clipped/balanced off-policy objectives) aim to stabilize off-policy LLM training. DPO, notably, learns directly from a fixed offline dataset of preferences — squarely off-policy / offline. See RL for reasoning and agentic RL for where this is heading.

A short history

How to choose

Frequently asked questions

Key references

• Reinforcement Learning: An Introduction — Sutton & Barto — Chapters 5–7 define on/off-policy, importance sampling, SARSA, Q-learning, and the deadly triad.
• Deep Reinforcement Learning and the Deadly Triad — van Hasselt et al., 2018 — when off-policy deep RL actually diverges.
• Q-learning — Watkins & Dayan, 1992 — convergence of the canonical off-policy method.
• Proximal Policy Optimization — Schulman et al., 2017 — the near-on-policy workhorse.
• Soft Actor-Critic — Haarnoja et al., 2018 — sample-efficient off-policy continuous control.

Related

Q-learning · Deep Q-Networks · Temporal-difference learning · Policy gradients · PPO · Offline RL · Exploration vs exploitation · Continuous control