Chapter 35 Preference Optimization: RLHF, DPO, and the Limits of Alignment

35.1 Which answer deserves to be copied?

Our fictional Beijing policy gives a current lodging limit of 750 yuan per room-night; the archived version gives 600. A user asks for the current amount. One answer is “750 yuan under the current policy.” Another is “Certainly! Your limit is 600 yuan.” The second is friendly and confident. Copying it as an SFT demonstration would nevertheless teach the wrong answer.

Suppose comparing responses is easier than writing a perfect response ourselves. Instead of supplying one target, we record which of two candidates is preferable for this request and evidence. The preferred response is conventionally called chosen and the other rejected. These names record a judgment, not a fact about correctness. A mislabeled chosen answer remains wrong.

Chapter 34 made demonstrations and their loss explicit. Here we audit comparisons, then use the probabilities of Chapter 6 to learn from them. Success means improvement under an explicit rubric and retained evaluation contract. It does not mean satisfying everybody’s preferences or proving every claim true.

35.2 The label is only as good as its question

A rubric specifies criteria for making and checking a judgment. Ours first requires the requested version, support from the supplied record, and respect for action boundaries. Only after these conditions pass do we compare clarity and relevant detail. Friendliness cannot compensate for a wrong amount. Missing evidence is grounds to withhold a training label, not guess.

Example 35.1 — Audit four pairs. The evidence and requested scope belong to each pair. These are original bilingual teaching data, not a human preference study. They are retained in data/part-vi/preferences-v1.json.

Table 35.1 — Similar labels need different treatment.

Request and supplied evidence	Recorded chosen	Recorded rejected	Audit
Current; old=600, new=750	Current cap: 750	Current cap: 600	Keep: chosen uses the new version
Historical; old=600, new=750	Old cap: 750	Old cap: 600	Reverse: requested scope changes the answer
Current; policy record missing	Current cap: 750	Current cap: 600	Quarantine: neither amount is supported
Current; new=750	750 yuan	Current cap: 750 yuan	Tie: both meet this rubric

The digits that make the first answer good make the second row’s chosen answer bad. In the third row, a reviewer who remembers our earlier example cannot act as an undisclosed source for a missing-record task. In the fourth, forcing a preference would invent an unjustified distinction. This small collection admits only strict preferences: ties remain in the audit but are omitted from its pair loss. Other objectives can model ties, with their own specification.

Two strict comparisons remain usable: one kept and one reversed. Preserve the original labels, rationale, rubric version and correction. For actual annotation, separate candidate identity from presentation order, record relevant disagreement, and keep related prompts in the same split. Ten pairs made from a few answers are not ten independent requests.

35.3 From preferences to a reward, then to a policy

To make better answers more likely, we need a distribution that can change. A policy here is the language model’s conditional distribution over responses. Choosing the next token is an action; a response is a sequence of actions. A reward is scalar feedback assigned to a response. It is a training signal, not money, a calibrated truth probability or permission to act.

In one common reinforcement learning from human feedback (RLHF) pipeline, comparisons train a separate reward model. It reads a prompt and response and produces a score. Suppose chosen receives score 2 and rejected score 1. A standard preference model maps their difference to a predicted probability of preferring chosen:

$$\sigma(1)=\frac{1}{1+\exp(-1)}=0.731059.$$

The function $\sigma(z)=1/(1+\exp(-z))$ maps a real score difference $z$ to a value between zero and one. Equal scores give $\sigma(0)=0.5$; a larger chosen-minus-rejected difference gives a larger predicted preference probability. For a label saying chosen wins, the cost $-\ln\sigma(z)$ penalizes a low probability. At $z=1$, this cost is 0.313262. Adding 100 to both scores changes neither their difference nor the cost. Comparisons alone do not establish an absolute reward zero.

After fitting this scorer, sample responses from the policy, score them, and update the policy to encourage higher-reward behavior. This use of sampled actions and feedback is reinforcement learning (RL). InstructGPT uses PPO for these updates; its optimizer details are outside this chapter’s scope. During the policy-update stage described here, the reward model and a reference policy remain fixed. The reference is a saved distribution used to measure departure, often the starting SFT checkpoint. It is neither a human label nor another voter.

Reward model → RL policy update

1. Outside the modelPair A > B → fit reward scores
2. Stored model numbersReward model: trained first, fixed in this RL stage
3. An operationSample policy outputs → reward − β log ratio
4. Stored model numbersUpdate policy; reference remains fixed

Comparison labels and sampled actions have different jobs. The expected log ratio supplies the KL penalty.

Original offline DPO

1. Outside the modelSame prompt, chosen A, rejected B
2. Stored model numbersPolicy (0.75, 0.25); fixed reference (0.5, 0.5)
3. An operationβ = 1: margin ln 3 → loss 0.287682
4. Stored model numbersUpdate policy from pair loss

No standalone reward-model fit or on-policy rollout inside this update; labels and reference probabilities are still required.

Figure 35.1 — Two paths from the same preference pair to a changed policy. Original diagram of the declared training stages · CC BY-SA 4.0

Figure 35.1 distinguishes two updates in this route. Comparison labels first change reward-model parameters. Newly sampled responses and their scores then guide policy changes. The reference evaluates the same responses without following those updates. Later collection and retraining rounds are possible; “fixed” refers to this declared stage.

35.4 How much reward is worth a large change?

A reward model has evidence of reliability where its training comparisons provide coverage. A policy pushed only toward high scores may discover outputs rewarded for unintended reasons. A penalty based on Kullback–Leibler divergence (KL) relative to the reference can discourage excessive departure. Here divergence measures a difference between distributions; it is not a distance in physical space.

Example 35.2 — A complete two-answer comparison. For one prompt, restrict the toy answer space to A and B, in that order, with assigned rewards 1 and 0. The reference probabilities are (0.5, 0.5). A proposed policy uses (0.75, 0.25). These are complete-response probabilities in an authored two-answer world, not measured probabilities of real texts.

The reference’s expected reward is $0.5\times1+0.5\times0=0.5$; the proposal’s is $0.75$. Compute the log ratio for each answer and average under the proposed policy:

$$D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{ref}}) =0.75\ln\frac{0.75}{0.5}+0.25\ln\frac{0.25}{0.5} =0.130812.$$

Here $\pi$ is the proposed policy and $\pi_{\mathrm{ref}}$ the reference. The terms are approximately 0.304099 and −0.173287 nats. Individual terms can be negative although the complete KL is nonnegative. Reversing the averaging distribution gives 0.143841, a different number. If the reference assigns zero probability to something the policy can produce, this direction of KL is infinite. Compatible support matters.

For arbitrary complete responses $y$ under a fixed prompt $x$, the same operation is

$$D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{ref}}) =\sum_y \pi(y\mid x)\ln\frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}.$$

In an autoregressive model, a sequence log probability is a sum of conditional token log probabilities. Practical RL training estimates the expectation with sampled responses; it does not enumerate all possible text.

Let $\beta$ be a nonnegative penalty weight, in reward units per nat. Compare expected reward minus $\beta$ times KL.

Table 35.2 — A penalty changes the tradeoff.

Policy	Expected reward	KL, nats	Reward − KL, $\beta=1$	Reward − 2 KL, $\beta=2$
Reference (0.5, 0.5)	0.500000	0.000000	0.500000	0.500000
Proposed (0.75, 0.25)	0.750000	0.130812	0.619188	0.488376

Among these two choices, the proposal wins at $\beta=1$ but loses at $\beta=2$. We evaluated assigned distributions, not an optimizer’s converged result. The penalty does not certify the reference as correct. A soft penalty also cannot guarantee a hard quality or safety threshold.

35.5 Learning directly from the pair

The reward-model route fits a scorer and then optimizes the policy using sampled responses. Direct Preference Optimization (DPO) offers another route in its original fixed-reference, offline setting: train the policy from pairs without fitting a standalone reward model or producing fresh on-policy rollouts inside each update.

DPO still evaluates both responses under the current policy and reference. It asks whether the policy favors chosen relative to what the reference favored.

Example 35.3 — Four probabilities make one DPO loss. Let A be chosen and B rejected. Reuse reference (0.5, 0.5) and policy (0.75, 0.25). A’s log ratio is $\ln(1.5)$; B’s is $\ln(0.5)$. Subtract them to get $\ln(3)=1.098612$. With DPO’s coefficient $\beta=1$:

$$\sigma(\ln 3)=\frac{1}{1+1/3}=0.75, \qquad L_{\mathrm{DPO}}=-\ln(0.75)=0.287682.$$

If policy and reference coincide initially, both log ratios are zero and loss is $\ln 2=0.693147$. The proposal therefore gives this preference label a higher probability under the DPO model.

Write $y_c$ for chosen and $y_r$ for rejected. One pair’s general expression is

$$z=\beta\left[ \ln\frac{\pi_\theta(y_c\mid x)}{\pi_{\mathrm{ref}}(y_c\mid x)} -\ln\frac{\pi_\theta(y_r\mid x)}{\pi_{\mathrm{ref}}(y_r\mid x)} \right], \qquad L_{\mathrm{DPO}}=-\ln\sigma(z).$$

The subscript $\theta$ marks trainable policy parameters. All four probabilities use the same prompt and compatible serialization. Substitution recovers Example 35.3. The original objective averages over pairs. Response probabilities include the declared response tokens and termination; dividing their log probabilities by answer length would change the objective.

The DPO derivation connects a particular preference model to a KL-regularized reward objective. Its $\beta$ scales the log-ratio margin; it is not another human rating or a “probability of safety.” Finite data, imperfect labels and restricted model capacity separate real training from the ideal optimum. Improving a relative margin need not increase chosen’s absolute probability on every update: other answers share mass and parameters.

35.6 A better score can hide a worse answer

Our deliberately defective scorer awards 1 whenever a response contains 750. It therefore rewards “The cap is not 750; use 600.” A search or training loop can exploit this by inserting the expected digits into wrong responses. This is reward hacking: raising measured reward by exploiting its mismatch with the intended objective.

A learned scorer can use subtler proxies: length, confidence, agreement with the user, or familiar formatting. DPO still needs the pair audit. A reversed historical label encourages the wrong preference even when the probabilities are calculated perfectly. Eliminating a standalone reward model does not eliminate defective data or optimization against an imperfect signal.

Preference, truth, fairness and safety need distinct evidence. An answer can win when both candidates are false. A rubric can overlook affected groups. Describing an approval does not establish that approval occurred. Test the relevant facts, affected cases and operational boundaries directly. InstructGPT’s limitations discuss its labeler group and remaining failures; DPO’s objective is a learning rule, not an alignment certificate.

Tasks with explicit, checkable answers can reduce some ambiguity in the source of feedback. Chapter 36 examines that possibility. The checker itself will still need inspection.

35.7 Summary

Preference optimization starts with a context-dependent judgment. A common RLHF route fits a reward model and then uses sampled rewards to update a policy relative to a fixed reference. Original offline DPO trains directly from pairs and reference log ratios. Both depend on what their labels reward. A preference win or small KL is not proof of truth, fairness, safety or authorization.

35.8 Exercises

1. In Table 35.2, use $\beta=0.5$. Calculate the proposed policy’s penalized score and compare it with the reference.
2. Reference probabilities are (0.8, 0.2) for chosen/rejected, but policy probabilities remain (0.75, 0.25). With DPO $\beta=1$, compute the margin’s exponential, preference probability and loss. Why is “chosen has probability above one half” insufficient?
3. A reviewer favors an answer confidently repeating a user’s stale 600-yuan premise. Identify the rubric defect and the evidence needed.
4. A scorer accepts every answer ending in “verified.” Explain how reward could increase without accuracy improving. Design a diagnostic whose gold label does not use that same word.
5. A report says “DPO needs no reward model, so neither a preference audit nor a reference checkpoint is needed.” Correct both implications. What would a successful local preference test still not prove?

35.9 Reference answers

1. The proposed score is $0.75-0.5\times0.130812=0.684594$, above the reference’s 0.500000. The coefficient changed; the assigned rewards and policy did not.
2. The ratio of ratios is $(0.75/0.8)/(0.25/0.2)=0.75$. Thus $\exp(z)=0.75$, $\sigma(z)=0.75/1.75=3/7=0.428571$, and loss is $-\ln(3/7)=0.847298$. Chosen receives less relative preference than in the reference, despite still getting more mass than rejected.
3. Agreement and confidence have displaced evidence and version scope. Inspect the current policy and question. If current=750 and current is requested, 600 fails. If the record is absent, withhold the factual preference rather than assume an unrecorded source.
4. A policy can append the word to a wrong answer. Pair correct and incorrect responses with the same suffix, including contradictions and unsupported action claims. Grade against supplied facts and boundaries; keep false acceptances as scorer failures.
5. Original DPO still needs sound comparisons and fixed-reference probabilities, which may be cached when checkpoint, tokenizer and template are fixed. A local preference pass does not establish all facts, affected groups’ interests, safety conditions or execution permissions.

35.10 References

• Ouyang et al., Training language models to follow instructions with human feedback, 2022, sections 3.5 and 5. Reward-model/PPO/reference training and the limits of preference alignment.
• Rafailov et al., Direct Preference Optimization, 2023, sections 3–4, equations 3–7. Derives the direct pair objective under a preference model and KL regularization; supports the four-probability example.