RLHF: From Human Preferences to KL-Controlled PPO

Good to see you again. In the last lesson, you built the central artifact used by classical RLHF: a pairwise preference record $(x,y_w,y_l)$, and you saw how a reward model learns to assign a higher scalar score to the response a labeler preferred. The important caveat was that this score is a learned proxy for a particular rubric and labeling process, not an intrinsic measure of truth or quality.

This lesson completes the classical RLHF story. You will trace how human comparisons become a reward model, how an SFT model is optimized with PPO against that model, and why KL control is essential both for training stability and for preventing the policy from exploiting an imperfect reward model.

---

RLHF as a controlled optimization system

Classical RLHF has three stages, each producing an artifact consumed by the next:

1. Supervised fine-tuning (SFT): human-written demonstrations turn a pretrained model into a capable instruction-following policy.
2. Reward modeling: humans compare several candidate responses to the same prompt; a reward model learns to predict those relative preferences.
3. Reinforcement-learning optimization: an evolving policy generates new responses, receives scores from the frozen reward model, and is updated with PPO while being constrained to remain close to a frozen reference policy.

The diagram’s first two stages occur before the inner RL loop. In the traditional offline setup, humans do not need to rate every response generated during PPO. That would make the optimization loop constrained by annotation latency and cost. Instead, human judgments train a reward model once, or periodically, and the reward model supplies scalable feedback during many PPO iterations.

The key components during PPO are:

Component	Role during RLHF	Updated during PPO?
Policy ${\pi }_{\theta }$	Generates the response; this is the model being improved.	Yes
Reference policy ${\pi }_{\mathrm{ref}}$	A frozen copy of the SFT policy used to measure behavioral drift.	No
Reward model $r_{\varphi }$	Scores a completed prompt-response pair according to learned preferences.	No
Value model $V_{\psi }$	Predicts expected future return from a partially generated response; reduces variance in PPO updates.	Yes
Prompt dataset	Supplies prompts for policy rollouts. It need not contain reference answers.	N/A

At deployment time, normally only the final policy is required for text generation. The reward model, reference model, and critic are training-time machinery.

---

From preference collection to a reward model

Start with prompts drawn from the intended product distribution. For each prompt, candidate responses are generated from an SFT model, model checkpoints, or a mixture of systems. Labelers compare responses under a written rubric that might include helpfulness, factuality, policy compliance, style, or task completion.

For a prompt $x$, suppose a labeler prefers $y_w$ over $y_l$. The reward model is trained so that:

$$r_{\varphi }(x,y_w)>r_{\varphi }(x,y_l)$$

using the pairwise ranking loss from the previous lesson:

$${\mathcal{L}}_{\mathrm{RM}}(\varphi )=-\mathbb{E}\left[\log \sigma \left(r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)\right)\right]$$

After training, freeze the reward model. During PPO, it receives a newly generated complete response and emits one scalar. It does not generate text itself and does not receive PPO gradients.

A practical consequence follows directly from this boundary: the reward model is now a learned specification that the policy will optimize aggressively. A reward-model validation accuracy that looks reasonable is not sufficient evidence that it is safe to optimize against. The policy will search for unusual outputs that score well, including regions of response space that human labelers never saw.

---

Recasting language generation as reinforcement learning

For an LLM, the reinforcement-learning vocabulary maps cleanly onto autoregressive generation:

• State $s_t$: the prompt plus all tokens generated so far.
• Action $a_t$: the next token selected by the policy.
• Policy ${\pi }_{\theta }(a_t\mid s_t)$: the LLM’s probability distribution over the vocabulary.
• Trajectory $\tau$: the whole generated response.
• Terminal reward: the reward model’s scalar score for the completed response.

If the prompt is $x$ and generated tokens are $y_1,\dots ,y_T$, then the policy probability of the full completion factors token by token:

$${\pi }_{\theta }(y\mid x)=\prod _{t=1}^T{\pi }_{\theta }(y_t\mid x,y_{<t})$$

The conceptual objective is to maximize expected reward over responses sampled from the policy:

$$J(\theta )={\mathbb{E}}_{y\sim {\pi }_{\theta }(\cdot \mid x)}\left[R(x,y)\right]$$

Unlike SFT, PPO has no gold token sequence to imitate. The model first generates an answer, receives a scalar evaluation for the trajectory, then adjusts the probabilities of its sampled tokens according to whether that trajectory did better or worse than expected.

That delayed, noisy signal is why PPO needs additional machinery: a critic for estimating expected outcomes, bounded updates, and a reference-policy constraint.

---

One PPO iteration, end to end

A practical PPO iteration can be traced in six steps.

1. Sample prompts.
   Draw a batch of prompts from the RL prompt distribution. These should represent actual intended usage and important safety or quality slices.

2. Generate rollouts.
   The current policy generates one or more sampled completions for each prompt. Sampling rather than greedy decoding matters: the policy must explore alternatives in order to learn which kinds of responses earn better reward.

3. Record training signals.
   For every generated token, store:

   • the policy’s old log probability;
   • the reference policy’s log probability;
   • the critic’s estimated value;
   • masks for valid completion tokens.

   For the complete response, call the reward model and record $r_{\varphi }(x,y)$.

4. Build a KL-regularized reward.
   Combine reward-model quality with a penalty for moving away from the frozen SFT reference. The resulting returns become the learning target for the critic and the policy.

5. Estimate advantages.
   Compare what actually happened to what the critic predicted. Positive advantage means the sampled decision sequence was better than expected; negative advantage means it was worse.

6. Run several PPO minibatch updates.
   Reuse the rollout batch for a small number of optimization epochs. PPO clipping limits how much the policy is allowed to change relative to the policy that produced those rollouts. Then discard the batch and generate fresh on-policy samples.

This workflow makes rollout generation a major systems cost. A training stack must serve the policy autoregressively, run reward and reference-model forward passes, retain per-token rollout metadata, and then run learner updates. With full parameter training, several large model states may coexist. With LoRA or another PEFT method, the frozen reference and trainable policy can often share a base model while using different adapters, reducing the memory burden.

---

Reward, value, and advantage are different quantities

Three scalar quantities are commonly conflated:

Quantity	Meaning	Source
Reward-model score $r_{\varphi }(x,y)$	Predicted human preference for the completed answer.	Frozen reward model
Value $V_{\psi }(s_t)$	Predicted expected future regularized reward from an incomplete response.	Trainable critic
Advantage ${\hat{A}}_t$	How much better the sampled continuation was than expected.	Returns minus critic prediction

The critic is not another name for the reward model. The reward model answers, “How preferred is this finished response?” The critic answers, “Given the prompt and partial response so far, what final regularized return should I expect from here?”

A simple intuition for the advantage is:

$${\hat{A}}_t\approx {\hat{G}}_t-V_{\psi }(s_t)$$

where ${\hat{G}}_t$ is the observed return from time $t$ onward. In practice, PPO commonly uses generalized advantage estimation, which blends observed rewards and value predictions to reduce the high variance caused by a mostly terminal reward.

If an answer receives a high reward-model score but the critic already expected that score, the advantage may be small. PPO therefore concentrates learning pressure on outcomes that are unexpectedly good or unexpectedly bad, not merely on responses with large raw scores.

---

PPO’s proximal update

When rollouts are generated, save the behavior policy as ${\pi }_{\mathrm{old}}$. During later learner updates, the trainable policy has changed to ${\pi }_{\theta }$. PPO compares their probability for each token that was actually sampled:

$${\rho }_t(\theta )=\frac{{\pi }_{\theta }(y_t\mid s_t)}{{\pi }_{\mathrm{old}}(y_t\mid s_t)}$$

If ${\rho }_t(\theta )=1$, the new policy assigns the sampled token the same probability as the rollout policy. If it is greater than one, the new policy favors that token more strongly; if it is less than one, it favors it less.

The clipped PPO policy objective is:

$$J_{\mathrm{PPO}}(\theta )={\mathbb{E}}_t\left[\min \left({\rho }_t(\theta ){\hat{A}}_t,\mathrm{clip}\left({\rho }_t(\theta ),1-ϵ,1+ϵ\right){\hat{A}}_t\right)\right]$$

where $ϵ$ is a small update-bound hyperparameter.

The clipping has an asymmetric practical effect:

• With positive advantage, PPO wants to increase the sampled token’s probability, but stops granting additional objective benefit once the increase is too large.
• With negative advantage, PPO wants to decrease that probability, but likewise limits an excessive decrease.

This makes several learner updates on the same rollout batch safer than unconstrained policy-gradient updates. The total implementation loss usually also includes a critic regression loss and may include an entropy term to discourage premature collapse of output diversity.

Crucially, clipping constrains each update relative to the immediately preceding rollout policy. It does not, by itself, ensure that the policy remains close to the original SFT model after hundreds of iterations. That is the job of reference KL control.

---

KL control: preserve useful behavior while optimizing reward

Let ${\pi }_{\mathrm{ref}}$ be the frozen SFT policy. For a sampled response, a common KL-shaped reward is:

$$\hat{R}(x,y)=r_{\varphi }(x,y)-\beta \sum _{t=1}^T\left[\log {\pi }_{\mathrm{old}}(y_t\mid s_t)-\log {\pi }_{\mathrm{ref}}(y_t\mid s_t)\right]$$

The reward model provides the incentive to improve according to learned preferences. The second term charges the policy for placing high probability on tokens that the reference considers unlikely. In expectation over policy samples, this log-probability difference estimates a reverse KL divergence:

$$D_{\mathrm{KL}}\left({\pi }_{\theta }\Vert {\pi }_{\mathrm{ref}}\right)$$

The intended optimization is therefore approximately:

$$\underset{{\pi }_{\theta }}{\max }\mathbb{E}\left[r_{\varphi }(x,y)\right]-\beta D_{\mathrm{KL}}\left({\pi }_{\theta }\Vert {\pi }_{\mathrm{ref}}\right)$$

The coefficient $\beta$ controls the quality-versus-drift trade-off:

KL coefficient choice	Likely behavior
Too small	Higher apparent reward, faster behavioral drift, increased reward-hacking risk.
Too large	Outputs stay close to SFT behavior, but preference improvements can be weak.
Adapted toward a target KL	Keeps training within a chosen average drift range despite changing reward scales.

A sampled token-level log-ratio can be negative. That does not mean the true KL divergence is negative; KL is nonnegative in expectation over the relevant distribution. It means that one particular sampled token happened to be more probable under the reference than under the rollout policy. Systems should monitor aggregate KL estimates, not react to an individual negative contribution as though it were necessarily an error.

Do not confuse the two constraints

The term “KL control” can refer to two related but distinct mechanisms:

Mechanism	Compares	Main purpose
PPO clipping or a PPO trust-region penalty	${\pi }_{\theta }$ against ${\pi }_{\mathrm{old}}$	Keeps individual learner updates stable while reusing rollout data.
Reference drift penalty	${\pi }_{\theta }$ against ${\pi }_{\mathrm{ref}}$	Keeps the final policy grounded in the capable SFT model and limits reward-model exploitation.

A model can satisfy PPO clipping at every small step and still drift far from its original SFT behavior over a long run. Conversely, a reference KL penalty does not eliminate the need for conservative optimizer updates. Production RLHF needs both forms of control, whether they appear explicitly in the same loss or are implemented through reward shaping and PPO clipping.

---

What to monitor in an RLHF run

A Staff-level explanation of RLHF should include the feedback loops, not just the equations. Reward-model reward increasing is necessary but not sufficient. Monitor at least:

• Reward-model score and regularized reward: distinguish raw improvement from improvement after KL cost.
• Average KL from the reference: detect uncontrolled behavioral drift.
• PPO clip fraction and approximate update KL: identify overly aggressive learner steps.
• Value loss and explained variance: detect a critic that cannot predict returns reliably.
• Response length, repetition rate, format markers, refusal rate, and entropy: detect proxies being exploited.
• Held-out reward-model ranking accuracy: verify the reward model remains meaningful on fresh preference pairs.
• Human and adversarial evaluations: the final authority on task quality, factuality, safety, and regressions.

For example, if reward rises sharply while responses become unusually long, repetitive, or filled with a superficial style feature correlated with labels, suspect reward hacking. Increasing $\beta$ may contain the immediate drift, but the durable fix is often data and reward-model work: improve the rubric, collect adversarial comparisons, and retrain the reward model on the failure mode.

---

Key takeaways

Classical RLHF first uses human preference comparisons to train a frozen reward model, then uses that model to score new responses generated by a trainable policy.

PPO treats next-token prediction as sequential decision-making. It gathers on-policy completions, turns reward and critic estimates into advantages, and makes bounded policy updates using the ratio between the current policy and the rollout policy.

KL control is not an implementation detail. A reference-policy penalty constrains long-run drift from the SFT model and reduces reward-model exploitation, while PPO clipping constrains individual learner updates. Rising reward-model scores alone do not establish real product quality.

Next, you will compare RLHF with Direct Preference Optimization (DPO) and decide when avoiding the reward-model-and-PPO training stack is the better engineering choice.