GRPO: Group Relative Policy Optimization

What is GRPO?

Group Relative Policy Optimization (GRPO) is a reinforcement-learning algorithm for fine-tuning large language models. Its one big idea is a study group that grades itself: for each prompt, the model generates a group of answers (say 8 or 16), each answer is scored, and an answer is rewarded for beating the group average — not for hitting some absolute target. Answers above the group’s mean get pushed up; answers below it get pushed down.

That sounds small, but it removes the most expensive, most finicky component of the classic PPO recipe: the value network (the “critic”). PPO trains a second neural network, nearly as large as the policy itself, just to estimate how good a state is so it can compute an advantage. GRPO replaces that learned estimate with a statistic computed from the group of samples — the mean reward becomes the baseline, for free. No critic to train, store, or get wrong.

Where GRPO came from

GRPO was introduced by DeepSeek in the DeepSeekMath paper (Feb 2024), as “a variant of PPO that enhances mathematical reasoning while concurrently optimizing the memory usage of PPO.” It then went mainstream a year later when DeepSeek-R1 (Jan 2025) used it at scale to elicit emergent reasoning — including the now-famous R1-Zero, which applied GRPO with pure RL and no supervised fine-tuning and watched chain-of-thought, self-verification, and “aha moments” emerge on their own.

The core idea: group-relative advantage

How a group of samples replaces the critic

In any policy-gradient method you update the policy in proportion to an advantage — “how much better than expected was this action?” PPO learns a value function $V(s)$ to define “expected,” then sets advantage $\approx r - V(s)$. GRPO instead samples a group of $G$ completions $\{o_1, \dots, o_G\}$ for the same prompt, scores each, and normalizes the rewards within the group:

Every token of completion $o_i$ shares the same advantage $\hat{A}_{i,t}$ (in the outcome-supervised case). The mean is the baseline; the standard deviation rescales updates so a group where answers barely differ doesn’t dominate one with a clear winner.

A worked numeric example

Suppose for one math prompt the model samples four answers and a rule-based checker scores them (1 = correct, 0 = wrong):

Here mean = 0.5 and std = 0.5, so the correct answers get $\hat A = +1$ and the wrong ones $\hat A = -1$. The gradient pushes the policy up on tokens that produced the correct chains and down on the failed ones — no critic, no learned value, just the group comparing itself. If all four answers were correct (or all wrong), the std is 0 and the advantages are undefined/zero — that prompt teaches nothing, which is itself a useful signal (TRL logs it as frac_reward_zero_std).

GRPO vs PPO: what’s actually different

No value network, lower memory

Dropping the critic isn’t just a memory win — it removes a whole class of bugs. A miscalibrated value net silently corrupts every advantage; GRPO’s baseline is a transparent arithmetic mean you can inspect.

Why this fits reasoning + verifiable rewards (RLVR)

GRPO pairs naturally with Reinforcement Learning with Verifiable Rewards (RLVR): tasks where a program can check correctness — does the math answer match? do the unit tests pass? does the output match the required format? You write a cheap rule-based reward function instead of training a reward model. That sidesteps reward hacking of a learned RM, and the binary/sparse reward is exactly what group-relative normalization handles gracefully — you don’t need fine-grained per-token value estimates, just “which of these whole answers were right.”

The math (practitioner depth)

The clipped surrogate objective

GRPO keeps PPO’s clipped surrogate so each update stays in a trust region, then sums over the group and its tokens:

where $r_{i,t}(\theta) = \dfrac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q, o_{i,<t})}$ is the per-token probability ratio. The $\min$/$\operatorname{clip}$ pair caps how far the policy can move in a single step, just like PPO.

The KL penalty to the reference policy

The $\beta\,\mathbb{D}_{\text{KL}}[\pi_\theta \| \pi_{\text{ref}}]$ term keeps the policy near a frozen reference model — the same “safety belt” idea as RLHF. DeepSeekMath used an unbiased $k_3$ KL estimator added directly to the loss (not folded into the reward as in classic PPO-RLHF).

Outcome vs process supervision

• Outcome supervision (the common case): one reward for the whole answer; every token in $o_i$ gets the same advantage. Cheap, and all you need with a final-answer checker.
• Process supervision: a process reward model scores each reasoning step; the advantage at token $t$ is the normalized sum of step rewards from $t$ onward. Denser signal, better credit assignment on long chains — but you need step-level labels or a PRM.

Known biases and failure modes

GRPO works, but its exact normalization introduces subtle optimization biases that 2025 papers diagnosed and fixed.

Length bias and difficulty bias

• Length bias. The loss normalizes by token count in a way that, combined with the std term, systematically rewards longer responses — the policy learns to ramble. Dr. GRPO (“Understanding R1-Zero-Like Training”) traced this to the per-response length normalization and the std scaling.
• Difficulty bias. Dividing by $\operatorname{std}(\mathbf{r})$ over-weights questions that are very easy or very hard (low-variance groups) and under-weights medium-difficulty ones, distorting the curriculum. TRL exposes scale_rewards=False to disable std scaling for exactly this reason.

Token-level vs sequence-level aggregation

How you average the loss matters. Averaging per-response (then across responses) lets long answers dominate; DAPO’s token-level loss averages over all tokens in the batch so every token counts equally regardless of which answer it came from. At the other extreme, GSPO argues the per-token importance ratio is itself the problem — for long sequences the product of token ratios is high-variance — and moves clipping/weighting to the sequence level, which stabilizes training dramatically for Mixture-of-Experts models.

GRPO variants (2025)

Each major variant maps to a specific bias in vanilla GRPO. This is the table most ranking pages predate:

Implementing GRPO in practice

The fastest path is Hugging Face TRL’s GRPOTrainer. You provide a model, a dataset of prompts, and one or more reward functions (plain Python that returns a score per completion).

When to use GRPO vs PPO vs RLOO

GRPO, RLOO and DAPO all sit in the same family: on-policy, critic-free, group-baselined policy gradients. They differ mainly in how the baseline and the loss are normalized. PPO is the heavyweight with a critic; DPO leaves the RL loop behind altogether.

Building the environments, verifiers and reward pipelines that GRPO depends on is increasingly its own discipline — see the RL environment and eval providers and our RL environments page.

Researcher takes

Lambert summarizes the now-famous length-normalization bias in DeepSeek’s GRPO.

A contrarian framing arguing GRPO’s length bias might be intentional rather than a defect.

Frequently asked questions

Key papers

• DeepSeekMath — Shao et al., 2024 — introduces GRPO.
• DeepSeek-R1 — DeepSeek-AI, 2025 — GRPO at scale; R1-Zero’s pure-RL reasoning (Nature version).
• Understanding R1-Zero-Like Training (Dr. GRPO) — Liu et al., 2025 — diagnoses length/difficulty bias.
• DAPO — Yu et al., 2025 — open large-scale recipe, 50 on AIME 2024.
• GSPO — Zheng et al., 2025 — sequence-level optimization for MoE RL.
• A Technical Survey of RL for LLMs — 2025 — situates GRPO among PPO and newer methods.

Related

PPO · RLHF · RLVR · DPO & preference optimization · Reward models · RL for reasoning · What is reinforcement learning?