DPO and Preference Optimization

What is preference optimization?

Preference optimization is a family of methods for fine-tuning a language model to behave the way humans prefer by training directly on comparisons — “response A is better than response B” — instead of a hand-written numeric score. It’s the same goal as RLHF: turn cheap human judgments of which answer is better into a training signal. The difference is mechanism.

Direct Preference Optimization (DPO) is the technique that made this dramatically simpler. Classic RLHF needs two heavy stages: train a separate reward model, then optimize the policy against it with reinforcement learning (usually PPO). DPO showed that both stages can be folded into a single supervised loss on the policy — no reward model, no sampling, no RL loop. IPO, KTO, ORPO and SimPO are popular descendants that each tweak DPO’s loss to fix a specific weakness.

Where it fits: SFT, then alignment

Post-training a base model is a pipeline. Pretraining gives broad knowledge. Supervised fine-tuning (SFT) teaches the model to follow instructions by imitating good demonstrations. But SFT can only copy answers — it can’t express that a helpful, honest answer should beat a plausible-but-useless one. That’s a preference, not a label, and it’s what the alignment stage handles.

DPO and its variants live in that alignment stage, almost always after SFT. The SFT model becomes the starting point — and, for most variants, also the reference model that keeps training from drifting too far.

The problem DPO solves: classic RLHF is heavy

The original RLHF recipe is powerful but fiddly. To optimize a policy you juggle four models in memory — the policy, a frozen reference, the reward model, and PPO’s value/critic network — and run an on-policy RL loop that samples fresh generations every step.

That machinery brings three pains: it’s expensive (more GPUs, more memory), it’s unstable (PPO is notoriously sensitive to hyperparameters), and it’s complex to implement correctly. The reward model can also be reward-hacked — the policy learns to exploit quirks the RM loves but humans don’t. DPO’s pitch: get most of the benefit while deleting the reward model and the RL loop entirely.

How DPO works in plain English

DPO starts from the same data RLHF uses — (prompt, chosen, rejected) triples — but skips straight to tuning the policy. The intuition: make the chosen answer more likely and the rejected answer less likely, but only relative to where the reference model started.

A subtle but important detail from practice: DPO often achieves the margin mostly by lowering the probability of the rejected response, not by raising the chosen one. That’s fine for ranking, but it’s the seed of the failure modes we’ll see later.

DPO never sees a numeric score for either answer — only the ordering. From thousands of such orderings it learns the policy directly.

Under the hood: Bradley-Terry and the implicit reward

The statistical foundation is the Bradley-Terry model, the same one RLHF reward models use. It says the probability that response $y_1$ beats $y_2$ is a sigmoid of their reward difference:

RLHF fits a reward model $r$ to this, then optimizes the KL-regularized objective

The DPO insight is that this objective has a known closed-form solution. The optimal policy is

Rearranging for $r$ shows the reward is implied by the policy itself:

This is the “your language model is secretly a reward model” line. Substitute this expression back into Bradley-Terry and the intractable partition function $Z(x)$ cancels (it’s the same for both responses to a prompt). You’re left with a loss in the policy’s parameters only — no reward model to train.

The DPO loss, the reference model, and β

Plugging the implicit reward into Bradley-Terry gives the DPO loss in its canonical form (this is exactly what Hugging Face TRL implements as the default sigmoid loss):

where $y_w$ is the chosen (winning) response and $y_l$ the rejected (losing) one.

Two pieces carry the weight of the whole method:

What the data looks like

DPO’s appeal is partly that its data format is trivially simple — a prompt plus two completions, one marked chosen and one rejected:

# Standard preference example (HF TRL format)
{
  "prompt":   "Explain photosynthesis to a 6-year-old.",
  "chosen":   "Plants are like tiny chefs. They take in sunlight...",
  "rejected": "Photosynthesis is the biochemical process by which...",
}

Public preference datasets in this shape power most open models: UltraFeedback (binarized into chosen/rejected), Anthropic’s HH-RLHF (helpful/harmless), and many task-specific sets. The comparisons can come from humans or, increasingly, from a strong “judge” model (RLAIF style). Quality matters more than quantity — a few tens of thousands of clean, on-distribution pairs typically beat a noisy hundred thousand.

The variants and what each fixes

DPO works, but it has sharp edges: it can overfit when preferences are near-deterministic, it requires paired data, and it requires a reference model (extra memory). Each major variant targets one of these. The cleanest way to see the differences is to put the four losses side by side — notice they only change what goes inside the loss, not the data philosophy.

IPO — stopping DPO from overfitting

IPO (Identity Preference Optimization), from Azar et al., diagnoses a real DPO weakness: when a preference is near-deterministic (chosen always beats rejected), the sigmoid loss keeps pushing the margin toward infinity, ignoring the KL constraint and overfitting. IPO swaps the logsigmoid for a bounded regression target — it drives the margin toward a finite value $\tfrac{1}{2\beta}$ instead of $\infty$ — so the implicit KL term can’t be steamrolled. It’s available in TRL as loss_type="ipo".

KTO — aligning from thumbs-up / thumbs-down

KTO (Kahneman-Tversky Optimization), from Ethayarajh et al., removes DPO’s hardest data requirement: paired comparisons. In production, paired data is scarce, but a stream of binary signals — a thumbs-up or thumbs-down per response — is cheap and abundant. KTO frames alignment as a HALO (Human-Aware Loss) using a Kahneman-Tversky prospect-theory utility, weighting losses against gains asymmetrically (humans feel losses more sharply). It still uses a reference model, but each example only needs a desirable/undesirable label, no partner response.

ORPO — one stage, no reference model

ORPO (Odds-Ratio Preference Optimization), from Hong et al., is the most radical simplification: it fuses SFT and alignment into a single stage and drops the reference model entirely. It appends a small log-odds-ratio penalty — which directly discourages the rejected response — to the ordinary negative-log-likelihood SFT loss. No separate alignment phase, no frozen reference in memory, and no SFT→DPO distribution shift between stages.

SimPO and cDPO — reference-free and noise-robust

SimPO (Meng et al., NeurIPS 2024) also goes reference-free: it defines the implicit reward as the length-normalized average log-probability of a sequence and adds a target reward margin γ. Length-normalization is deliberate — it directly attacks DPO’s length bias (below). SimPO reported strong AlpacaEval 2 results and is exposed in TRL as loss_type="sigmoid_norm". cDPO (conservative DPO) takes a different angle: it assumes preference labels are noisy (some fraction are flipped) and applies label smoothing so the loss can’t be dominated by mislabeled pairs — TRL implements the related idea via loss_type="robust" with a label_smoothing parameter.

When to use which

There is no universal winner, and benchmark leaderboards are noisy. The honest 2026 picture: a well-tuned DPO (or IPO when overfitting bites) remains a very strong default, with the variants winning in specific situations rather than across the board.

Failure modes to watch

DPO’s simplicity hides some subtle traps that most introductory explainers skip.

• Likelihood displacement. DPO can lower the probability of the preferred response while widening the margin — and in the worst case shift probability mass to opposite-meaning or even unsafe outputs. Razin et al.’s Unintentional Unalignment (ICLR 2025) documents this; the fix is careful data curation (avoid chosen/rejected pairs that are too similar) and watching the logps/chosen metric, not just the margin.
• Length / verbosity bias. Because longer sequences accumulate more log-prob terms, DPO tends to reward longer answers regardless of quality. LD-DPO and SimPO’s length-normalization both target this directly.
• Over-optimization. With too small a β the policy drifts far from the reference and quality degrades — the offline cousin of RLHF reward over-optimization.

Online vs offline, and iterative DPO

The deepest difference between DPO and PPO isn’t the loss — it’s where the data comes from.

• Offline (vanilla DPO): trains on a fixed dataset of pairs. The model never generates fresh samples during training. Simple and cheap, but the model can only learn from a static snapshot of preferences.
• Online (PPO-style): the policy generates new responses each step, which are scored and learned from. More expensive, but the signal tracks the model’s current behavior — and it’s harder to over-optimize a stale dataset.

Iterative DPO is the bridge: alternate between generating fresh responses from the current policy, labeling them (by humans or a judge/reward model) into new pairs, and running another DPO round. This recovers much of online RL’s benefit while keeping DPO’s simple loss — and it’s how several open models were tuned. Pushed further, you arrive back at full on-policy RL like PPO or GRPO.

How to run it in practice

The standard toolkit is Hugging Face TRL, whose DPOTrainer covers DPO and — via loss_type — IPO, SimPO, robust/cDPO and a dozen more, plus dedicated KTOTrainer and ORPOTrainer. A minimal DPO run is genuinely a few lines:

from trl import DPOConfig, DPOTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",          # your SFT model
    args=DPOConfig(beta=0.1, loss_type="sigmoid"),
    train_dataset=dataset,            # prompt / chosen / rejected
)
trainer.train()

If ref_model is left out, TRL automatically uses the initial state of model as the frozen reference. Switching to IPO is just loss_type="ipo"; SimPO is loss_type="sigmoid_norm". The Hugging Face Alignment Handbook packages full SFT→DPO recipes (the Zephyr models came out of it), and TRL’s DPO docs list every loss variant.

Building the preference data and reward pipelines behind this — at production scale, with clean comparisons and good coverage — is its own industry; see the preference-data and RL environment companies.

A short history

DPO vs RLHF/PPO at a glance

Researcher takes

A clear correction of the most common misconception about how DPO actually works under the hood.

A co-author of the original DPO paper stakes out the pro-DPO position and challenges PPO advocates to justify its extra cost.

Frequently asked questions

Key papers

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., NeurIPS 2023 — the seminal DPO paper.
• A General Theoretical Paradigm to Understand Learning from Human Preferences — Azar et al., 2023 — introduces IPO.
• KTO: Model Alignment as Prospect Theoretic Optimization — Ethayarajh et al., 2024.
• ORPO: Monolithic Preference Optimization without Reference Model — Hong et al., EMNLP 2024.
• SimPO: Simple Preference Optimization with a Reference-Free Reward — Meng et al., NeurIPS 2024.
• Unintentional Unalignment: Likelihood Displacement in DPO — Razin et al., ICLR 2025.
• A Comprehensive Survey of Direct Preference Optimization — 2024 — the variant landscape.

Related

RLHF · Reward models · PPO · GRPO · RLVR · RL for reasoning · What is reinforcement learning?