From RLHF to DPO: the maths of preference training

A pre-trained language model has read most of the internet and wants to continue text. A useful assistant has to do something narrower: answer the question, follow instructions, say when it doesn’t know. Nobody can write down a loss function for “helpful”, but people can compare two answers and say which is better. Preference training turns those comparisons into a better model.

This article follows the maths from the original RLHF recipe to DPO, which removes the reward model and the reinforcement learning with one algebraic step, and on to GRPO, the method behind recent reasoning models. The figures compute the KL-regularised optimum exactly, so you can watch reward hacking happen.

Why preferences

Supervised fine-tuning (SFT) on demonstrations teaches the format of a good answer, but demonstrations are expensive and imitate their writers, mistakes included. Comparisons are cheaper, more consistent across annotators, and can rank answers better than anything the annotator would have written. InstructGPT (Ouyang et al., 2022) showed the payoff: a 1.3B-parameter model trained with human feedback was preferred to the 175B GPT-3 it started from.

Bradley–Terry reward models

The first step is to turn comparisons into a scalar score. The Bradley–Terry model, originally for ranking chess players and sports teams, assumes each answer y has a latent reward r(x, y) and that the probability a person prefers y_w over y_l is a sigmoid of the difference:

A reward model r_φ, usually the SFT model with its output layer replaced by a single scalar head, is trained by maximum likelihood on the comparison data, minimising −log σ(r_φ(y_w) − r_φ(y_l)). Only differences matter, so the reward has no absolute scale, a detail that becomes important below.

RLHF with PPO

The policy is then optimised to score highly under the reward model, without straying too far from where it started:

π_ref is the frozen SFT model. The KL term is the leash: it stops the policy from drifting into text the reward model has never seen and therefore scores unreliably. InstructGPT optimised this with PPO (Schulman et al., 2017), which means four models in memory: the policy, the reference, the reward model, and a value model that estimates expected future reward so that advantages, how much better an action was than expected, can be computed with GAE. PPO then maximises a clipped surrogate, min(ρA, clip(ρ, 1 − ε, 1 + ε)A) with ρ = π_new/π_old, which stops any single update from moving the policy too far. It works, but it is expensive, has many hyperparameters, and is famously sensitive to them.

The KL leash and its optimum

The KL-regularised objective has a closed-form solution. For each prompt, the optimal policy reweights the reference by the exponentiated reward:

Z(x) sums over every possible response, which is why you can’t just compute π* for a language model. But on a small set of candidate answers you can, and it shows exactly what β does:

DPO: the reward model disappears

Direct Preference Optimisation (Rafailov et al., 2023) reads the optimum backwards. Take logs and solve for the reward:

Any reward is expressible through the policy that is optimal for it. Now substitute into Bradley–Terry. The preference probability depends only on the difference of rewards for the same prompt, so the intractable β log Z(x) cancels:

That is the whole method: a binary classification loss on preference pairs, using log-probabilities from the policy and the frozen reference. There is no reward model, no sampling and no value function. The policy is the reward model, with implicit reward r̂_θ = β log π_θ/π_ref. Its gradient shows what it does:

Raise the likelihood of the chosen answer, lower the rejected one, and weight each pair by how wrong the implicit reward currently is. Pairs the model already ranks correctly contribute little.

Reward hacking

A reward model is a learned approximation trained on finite data. Optimise against it hard enough and the policy finds its blind spots: length, confident tone, flattery, formatting that annotators happened to like. Gao et al. (2022) measured this systematically. As the policy moves away from the reference (KL grows), the learned reward keeps rising while the “gold” reward, from a much larger reference reward model, rises and then falls, following predictable curves. Fig. 1 reproduces that shape in miniature.

The defences are the ones you’d expect: the KL penalty, early stopping on held-out gold evaluations, reward model ensembles, regularly refreshing the reward model on outputs from the current policy, and, where possible, rewards that can’t be fooled at all.

GRPO and verifiable rewards

For maths, code and other tasks with checkable answers, the reward model can be replaced by a verifier: is the final answer correct, do the tests pass, is the output in the required format? Such rewards can’t be flattered. Reinforcement learning with verifiable rewards (RLVR) is how recent reasoning models were trained.

GRPO, Group Relative Policy Optimisation (Shao et al., 2024, introduced with DeepSeekMath), is the algorithm most associated with it. It keeps PPO’s clipped objective and KL penalty but drops the value model. For each prompt it samples a group of G answers, scores them, and uses each answer’s standardised reward within the group as its advantage:

The group mean plays the role the value model played in PPO, an estimate of how good a typical answer to this prompt is, at the cost of sampling several answers instead of training a second large network. DeepSeek-R1-Zero applied GRPO directly to a base model with only rule-based rewards for correctness and format, and long chains of thought, including checking and backtracking, emerged without being demonstrated. Follow-up work (for example Dr. GRPO) showed that the per-group std normalisation and per-sequence length averaging introduce biases, such as rewarding longer wrong answers less harshly, and proposed simple corrections.

The wider family

• RLAIF / Constitutional AI (Bai et al., 2022): the comparisons come from a model judging against written principles rather than from people.
• IPO: a squared loss on the log-ratio gap that doesn’t push it to infinity on deterministic preferences.
• KTO (Ethayarajh et al., 2024): learns from single thumbs-up or thumbs-down labels instead of pairs, using a prospect-theory value function.
• Best-of-N and rejection sampling fine-tuning: sample many answers, keep the best by reward, fine-tune on them. It is crude but strong, and a useful baseline for any RL method.

The losses in code

// Log-probabilities are summed over the response tokens, for the policy and the frozen reference.
type Pair = { logpChosen: number; logpRejected: number; refChosen: number; refRejected: number };

const logSigmoid = (x: number) => (x >= 0 ? -Math.log1p(Math.exp(-x)) : x - Math.log1p(Math.exp(x)));

// DPO: a logistic loss on the difference of implicit rewards beta * log(pi / pi_ref).
export function dpoLoss(batch: Pair[], beta = 0.1) {
  let loss = 0;
  for (const p of batch) {
    const rChosen = beta * (p.logpChosen - p.refChosen);
    const rRejected = beta * (p.logpRejected - p.refRejected);
    loss += -logSigmoid(rChosen - rRejected);
  }
  return loss / batch.length;
}

// GRPO: advantages are z-scores of rewards within the group sampled for one prompt.
export function groupAdvantages(rewards: number[], eps = 1e-6) {
  const mean = rewards.reduce((a, b) => a + b, 0) / rewards.length;
  const std = Math.sqrt(rewards.reduce((s, r) => s + (r - mean) ** 2, 0) / rewards.length);
  return rewards.map((r) => (r - mean) / (std + eps));
}

// The clipped surrogate both PPO and GRPO maximise, per token, with ratio = pi_new / pi_old.
export function clippedObjective(ratio: number, advantage: number, clip = 0.2) {
  return Math.min(ratio * advantage, Math.min(Math.max(ratio, 1 - clip), 1 + clip) * advantage);
}

References

1. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155
2. Christiano, P. et al. (2017). Deep reinforcement learning from human preferences. arXiv:1706.03741
3. Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347
4. Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290
5. Gao, L., Schulman, J. & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760
6. Azar, M. G. et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv:2310.12036
7. Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300
8. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948
9. Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783
10. Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073
11. Ethayarajh, K. et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. arXiv:2402.01306
12. Bradley, R. A. & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika. doi:10.2307/2334029