Preference Data and Reward Model Objectives

Good to see you again. In the previous lesson, you compared full fine-tuning, adapters, LoRA, and QLoRA as choices shaped by adaptation capacity, training memory, and deployment topology. All of those methods optimize against an explicit training target. Preference-based post-training changes the kind of target: instead of asking for the correct continuation, we ask people which of several model responses is better.

In this lesson, you will construct a production-usable preference-data record and explain exactly what a reward model learns from it. The central idea is simple but important: a reward model does not learn an objective, universally meaningful “quality score.” It learns a scalar scoring function whose differences predict which response a labeler would prefer for a particular prompt and policy rubric. That distinction matters when the model later becomes an optimization target.

---

From demonstrations to comparisons

A supervised fine-tuning example has one desired completion:

$$(x,y^{\ast })$$

where $x$ is the prompt and $y^{\ast }$ is a human-written or otherwise approved target response. The SFT loss increases the probability of the tokens in $y^{\ast }$.

A preference example instead contains one prompt and at least two candidate responses:

$$(x,y_w,y_l)$$

where $y_w$ is the response a labeler chose and $y_l$ is the response they considered less preferred. The candidates must answer the same prompt under the same system instructions and available context. Otherwise, the label mixes response quality with a changed task.

The diagram’s “human scoring” stage can use direct pairwise comparisons, a ranking of several candidates, or ratings along dimensions such as accuracy and coherence. For standard reward modeling, these signals are usually converted into pairwise statements:

Given this prompt and context, response $A$ is preferred to response $B$.

Pairwise judging is often cheaper and faster than asking evaluators to write a high-quality response from scratch. It also captures choices that are difficult to turn into a single reference answer: whether a refusal is appropriately scoped, whether an explanation is misleading, whether a response is clear without being unnecessarily long, or whether it handles uncertainty honestly.

A preference label is only as meaningful as its rubric. “Better” must be operationally defined for the product. For a general assistant, a rubric might prioritize:

1. factual grounding and calibrated uncertainty;
2. safety and policy compliance;
3. task completion and relevance;
4. clarity, tone, and concision.

The ordering and trade-offs should be explicit. A coding copilot, medical-information assistant, and creative-writing tool should not silently optimize the same notion of preference.

---

Construct a preference-data example

Consider an internal IT-support assistant. The application has retrieved an approved policy excerpt, and the user asks:

“Can I share a customer export with a contractor helping on the project?”

The context supplied to the model says: Customer exports may be shared only with contractors who are approved vendors, have a current data-processing agreement, and are granted least-privilege access through the approved workspace. Public-link sharing is prohibited.

Two model completions are sampled under identical system instructions and retrieved context:

• Candidate A: “Yes. Send the export to the contractor if they need it for the project. A password-protected ZIP is a good extra precaution.”
• Candidate B: “Only if the contractor is an approved vendor with a current data-processing agreement. Share it through the approved workspace with least-privilege access; do not use a public link. If you cannot verify those conditions, ask the project owner or security team before sharing.”

A labeler should choose Candidate B. It directly answers the question, uses the provided policy, gives an actionable path, and avoids inventing an unsupported workaround. Candidate A is superficially helpful, but it omits the governing conditions and recommends a mechanism that does not satisfy the stated policy.

A compact pairwise record could look like this:

{
  "example_id": "it-support-004281",
  "messages": [
    {
      "role": "system",
      "content": "You are an internal IT-support assistant. Use supplied policy context. Do not invent policy."
    },
    {
      "role": "context",
      "content": "Customer exports may be shared only with approved vendors that have a current data-processing agreement and least-privilege workspace access. Public-link sharing is prohibited."
    },
    {
      "role": "user",
      "content": "Can I share a customer export with a contractor helping on the project?"
    }
  ],
  "chosen": "Only if the contractor is an approved vendor with a current data-processing agreement. Share it through the approved workspace with least-privilege access; do not use a public link. If you cannot verify those conditions, ask the project owner or security team before sharing.",
  "rejected": "Yes. Send the export to the contractor if they need it for the project. A password-protected ZIP is a good extra precaution.",
  "rubric_version": "security-support-v3",
  "label_type": "pairwise_preference",
  "label_confidence": "high",
  "source_model": "assistant-sft-2025-02",
  "source_model_version": "17",
  "split": "train"
}

The core training fields are the shared prompt/context, chosen, and rejected. The surrounding fields are what make the record governable and debuggable in a real platform.

Field or practice	Why it matters
Full message sequence	A response can be appropriate only relative to the system prompt, conversation history, retrieved evidence, and user request.
Chosen and rejected responses	Defines the pairwise training signal. Preserve the raw text exactly as judged.
Rubric and rubric version	Makes the preference semantics auditable when guidelines evolve.
Candidate provenance	Lets you detect whether labels are biased toward a particular model, decoding configuration, or response length.
Labeler metadata or cohort	Supports quality audits, disagreement analysis, and representation monitoring, subject to privacy controls.
Confidence, rationale, or error tags	Helps prioritize review and identify systematic failure modes such as hallucination or excessive refusal.
Dataset split	Prevents prompt-level leakage from training into validation and test sets.

Two data-design rules are especially important.

Keep the comparison fair

Both candidates must be judged against exactly the same task state. If Candidate A sees a retrieved policy document and Candidate B does not, a preference label no longer measures which response is better; it partly measures retrieval availability.

Similarly, randomize display order. If the first visible response is systematically chosen more often, a reward model can learn a positional artifact rather than your intended quality criteria.

Preserve ambiguity instead of manufacturing certainty

Not every pair has a clear winner. When candidates make different trade-offs that the rubric does not resolve, collect a tie, an “insufficient context” label, or send the case to adjudication. Forcing a binary label adds noise precisely at the product boundary where policy is unclear.

The same principle applies to examples with low annotator agreement. They are useful signals that the rubric, context, candidate quality, or task definition needs attention; they are not merely inconvenient rows to discard without analysis.

For $K$ candidate responses ranked for one prompt, you can derive several pairwise labels. A ranking such as:

$$y_1\succ y_2\succ y_3\succ y_4$$

implies that $y_1$ is preferred over each of $y_2,y_3,y_4$, and so on. This can increase label efficiency, but those derived pairs are correlated: they arise from the same prompt, candidates, and judgment event. Training and evaluation should keep all candidates for a prompt in the same dataset split, rather than letting related pairs leak across splits.

---

What a reward model is trained to predict

A reward model takes the complete conditioning context $x$ and a candidate response $y$, then produces one scalar:

$$r_{\theta }(x,y)\in \mathbb{R}$$

Architecturally, it is often a transformer initialized from a language model or SFT model, with the normal next-token prediction head replaced by a scalar-value head. The exact architecture can vary, but its interface is stable:

Input	Output
System instructions, conversation, relevant retrieved context, and one candidate response	One scalar reward score

For one labeled pair, define:

• $y_w$: human-preferred, or winning, response;
• $y_l$: less-preferred, or losing, response.

The model converts its reward difference into a predicted preference probability:

$$P_{\theta }(y_w\succ y_l\mid x)=\sigma \left(r_{\theta }(x,y_w)-r_{\theta }(x,y_l)\right)$$

where $\sigma$ is the sigmoid function. Training minimizes the negative log-likelihood of the observed human choice:

$${\mathcal{L}}_{\text{RM}}(\theta )=-{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \sigma \left(r_{\theta }(x,y_w)-r_{\theta }(x,y_l)\right)\right]$$

This is a pairwise logistic, or Bradley-Terry-style, objective.

The mechanism is worth reading directly from the primary source.

Read the loss as a ranking objective

Suppose the reward model assigns:

$$r_{\theta }(x,y_w)=1.2$$

and

$$r_{\theta }(x,y_l)=0.4$$

The reward difference is:

$$1.2-0.4=0.8$$

The model’s predicted probability that the chosen response wins is:

$$\sigma (0.8)\approx 0.69$$

and the loss for this one example is:

$$-\log (0.69)\approx 0.37$$

To reduce the loss, optimization can raise the winner’s score, lower the loser’s score, or do some combination of both. A large positive reward gap means the model is confident in the observed preference. If the rejected response receives the higher reward, the gap is negative, the predicted preference probability is low, and the loss is large.

The objective has several consequences that are easy to miss.

Reward scores are relative, not absolute

The loss cares about:

$$r_{\theta }(x,y_w)-r_{\theta }(x,y_l)$$

not about either score alone. Adding the same prompt-dependent constant $c(x)$ to both rewards changes nothing:

$$\left[r_{\theta }(x,y_w)+c(x)\right]-\left[r_{\theta }(x,y_l)+c(x)\right]=r_{\theta }(x,y_w)-r_{\theta }(x,y_l)$$

So it is incorrect to interpret a score of $4.0$ as inherently “twice as good” as a score of $2.0$, or to assume chosen responses must receive positive scores while rejected ones must receive negative scores. The meaningful claim is comparative:

For this prompt and these two possible responses, the reward model assigns a higher score to the one humans preferred.

That is why score-scale shifts across reward-model versions are not automatically meaningful. For monitoring, use held-out pairwise accuracy, log loss, calibration checks, and targeted slices—not a raw-score threshold copied from an earlier model.

The reward model learns the rubric embedded in data

A reward model does not discover universal human values. It approximates the preferences expressed by your prompt distribution, candidate distribution, rubric, labelers, and adjudication process.

For example, if labelers consistently prefer longer answers because length is mistaken for helpfulness, the reward model may learn a length bias. If candidates from one model family are easy to recognize stylistically, it may learn a provenance cue. If preference data lacks difficult adversarial prompts, the model can look strong in aggregate while failing exactly where safety or factuality matters most.

This creates a feedback-design responsibility:

• sample prompts from intended production use, including high-risk and edge-case slices;
• generate candidates with diversity in model version, decoding method, and quality level;
• make rubric trade-offs explicit;
• retain disagreement and provenance data;
• evaluate the reward model on held-out comparisons and targeted red-team cases;
• refresh the dataset as the policy model changes.

The “reward” in reward model should therefore be understood as a learned proxy for the label process, not as a ground-truth utility function.

---

Why this artifact matters downstream

A preference dataset provides a compact human signal, but its operational value comes from turning that signal into a model that can score many newly generated responses. Rather than requiring a person to judge every candidate during post-training, the reward model generalizes from labeled comparisons to produce scalar feedback at scale.

That scalability is also its risk. Any systematic reward-model error can be amplified when a later optimization process searches for high-scoring outputs. A response that looks unusually good to the reward model while violating the intended rubric is a form of reward hacking. Strong reward-model validation, held-out human evaluation, and conservative optimization are therefore not optional infrastructure details.

For now, keep the boundaries clear:

• Preference data records what a labeler chose among alternatives.
• The reward model maps one prompt-response pair to a scalar.
• The reward-model loss makes the chosen response score higher than the rejected response for the same context.
• A later post-training method can use that learned score as feedback—but the reward model itself is not the policy and does not generate the final answer.

---

Key takeaways

A robust preference example contains a shared prompt and context, two or more candidate responses, an explicit human choice or ranking, and sufficient metadata to audit rubric, provenance, confidence, and dataset splits.

The central pairwise record is:

$$(x,y_w,y_l)$$

where $y_w$ is preferred over $y_l$ for the same conditioning context.

A reward model outputs a scalar $r_{\theta }(x,y)$. It is trained so that the difference between the chosen and rejected rewards predicts the probability of the observed human preference:

$$\sigma \left(r_{\theta }(x,y_w)-r_{\theta }(x,y_l)\right)$$

The model learns relative ranking, not a calibrated universal quality score. Its behavior reflects the feedback rubric and data-generation process, which makes data governance and reward-model evaluation central engineering concerns.

Next, you will trace the full RLHF pipeline: preference collection, reward-model training, policy optimization with PPO, and the role of KL control in preventing the optimized policy from drifting too far from a capable reference model.