Preparing Supervised Fine-Tuning Examples: Prompt Formatting, Tokenization, and Loss Masking

Welcome. This module moves from what post-training is meant to achieve to the concrete artifacts a training system consumes. In this first lesson, you will prepare a single supervised fine-tuning (SFT) example correctly: render the conversation in the model’s native prompt format, convert it to token IDs, and ensure that loss is charged only for the desired assistant response.

This is a small piece of code with outsized systems consequences. A formatting error can train a model to emit malformed chat headers; an off-by-one error can eliminate the first response token from training; a masking error can spend substantial GPU time teaching the model to reproduce user prompts rather than answer them. By the end, you should be able to inspect an SFT batch and explain exactly which model prediction receives gradient at each position.

---

The SFT artifact: one token sequence, selective supervision

A causal language model accepts one sequence of token IDs:

$$x_0,x_1,\dots ,x_{T-1}$$

At each position, it predicts the next token. SFT does not replace that mechanism. It supplies carefully chosen demonstrations of desired behavior, then limits the loss to tokens representing the target answer.

For an instruction-following example, the conceptual data item is:

messages = [
    {
        "role": "system",
        "content": "Answer concisely. State uncertainty when relevant."
    },
    {
        "role": "user",
        "content": "What is the capital of France?"
    },
    {
        "role": "assistant",
        "content": "Paris."
    },
]

The model cannot consume this Python structure directly. It needs a serialized token sequence, conceptually resembling:

<begin>
<system-header>
Answer concisely. State uncertainty when relevant.
<end-turn>
<user-header>
What is the capital of France?
<end-turn>
<assistant-header>
Paris.
<end-turn>

The exact special tokens, spacing, newline rules, and end-of-turn markers differ by model family. That is why the tokenizer’s chat template is part of the model interface, not a cosmetic formatting preference.

The desired training objective is:

$${\mathcal{L}}_{\mathrm{SFT}}=-\frac{1}{|A|}\sum _{t\in A}\log p_{\theta }(x_t\mid x_{<t})$$

Here, $A$ is the set of assistant target token positions, usually the assistant’s response content plus its termination token. The model can attend to the system instruction, user request, and assistant header as context, but those tokens do not contribute directly to the loss.

For a focused walkthrough of this idea and a small batch-preparation routine, watch:

Three masks that should not be conflated

Training pipelines often use the word “mask” for different mechanisms. They answer different questions:

Mechanism	Question answered	Typical representation
Causal masking	Which earlier tokens may a position attend to?	Built into a causal decoder; future positions are unavailable
Attention mask	Which tensor positions are real tokens rather than padding?	1 for real tokens, 0 for padding
Loss mask / labels	Which target tokens should affect gradients?	Token ID for supervised targets; -100 for ignored positions

An attention_mask prevents the model from treating padding as meaningful context. It does not, by itself, prevent padded or prompt tokens from contributing to cross-entropy. That job belongs to labels.

---

Prompt formatting is behavioral compatibility

A chat-tuned model has learned an interaction grammar during its earlier instruction tuning: roles, separators, turn endings, and often a distinct marker that says, “the assistant’s response starts here.” SFT should preserve that grammar.

The primary rule is:

Use the same chat template at training and inference.

Do not manually write strings such as "User: ... Assistant: ..." unless that is precisely the template expected by the checkpoint. A model trained with one format and served with another may still produce text, but it has been asked to generalize across an unnecessary distribution shift.

For a single-turn prompt-completion example, a robust conceptual procedure is:

1. Place the system and user turns into the prompt.
2. Render them with the tokenizer’s apply_chat_template, adding the assistant-generation header.
3. Tokenize the expected assistant answer without adding a second beginning-of-sequence token.
4. Append an appropriate end-of-sequence or end-of-turn token.
5. Form labels that ignore the entire prompt and retain the answer targets.

The diagram below distinguishes this SFT objective from training on every token, and also previews an efficiency concern: batches contain conversations of unequal lengths.

Training on all tokens is not inherently “wrong.” It is appropriate for continued pretraining or tasks where the whole sequence is target text. But for ordinary instruction SFT, it changes the objective: the model is rewarded for predicting system and user wording, even though the product behavior you want is high-quality assistant output conditioned on that wording.

The Hugging Face TRL documentation formalizes the same practical model: examples are processed through preprocessing and tokenization, then trained with token-level negative log-likelihood. Read the selected sections now as a reference for the framework conventions that will recur throughout this course.

---

Build a correct single-turn example

Assume the final message is the gold assistant response. The following implementation expresses the essential logic without tying it to a specific model’s special-token strings.

IGNORE_INDEX = -100

def prepare_single_turn_sft_example(messages, tokenizer):
    """
    messages ends with the desired assistant response.
    Returns unpadded input_ids, attention_mask, and labels.
    """

    assert messages[-1]["role"] == "assistant"

    prompt_messages = messages[:-1]
    target_text = messages[-1]["content"]

    # Renders system/user turns plus the model-specific assistant header.
    # The assistant answer itself is not included yet.
    prompt_ids = tokenizer.apply_chat_template(
        prompt_messages,
        tokenize=True,
        add_generation_prompt=True,
    )

    # Encode only answer content. Do not add BOS or other sequence-start tokens.
    target_ids = tokenizer(
        target_text,
        add_special_tokens=False,
    )["input_ids"]

    # Teach the model to stop after completing this answer.
    if tokenizer.eos_token_id is None:
        raise ValueError("Choose an explicit end-of-turn policy for this tokenizer.")

    target_ids = target_ids + [tokenizer.eos_token_id]

    input_ids = prompt_ids + target_ids

    # Same length as input_ids. Prompt positions are not supervised.
    labels = (
        [IGNORE_INDEX] * len(prompt_ids)
        + target_ids.copy()
    )

    attention_mask = [1] * len(input_ids)

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels,
    }

This construction makes the boundary explicit:

input_ids:  [ prompt tokens including assistant header | Paris . EOS ]
labels:     [ -100       -100       -100             | Paris . EOS ]

The assistant header belongs in input_ids, because it tells the model what behavior is expected next. It is usually ignored in labels, because the primary target is response content. The final end token is typically supervised so the model learns when to stop.

Why labels are not manually shifted in most training APIs

A common source of bugs is applying the next-token shift twice.

At a high level, the token at position $t-1$ is used to predict the token at position $t$. Hugging Face causal language-model implementations usually perform that alignment internally when you call:

outputs = model(
    input_ids=input_ids,
    attention_mask=attention_mask,
    labels=labels,
)
loss = outputs.loss

So labels should ordinarily be aligned with the unshifted input_ids, as in the function above.

Suppose the assistant-header token is at position $h$, and the first answer token "Paris" is at position $h+1$:

Position	Input token	Label stored at this position	Prediction scored
$h$	assistant header	-100	No direct target at this location
$h+1$	Paris	ID for Paris	The logit produced at $h$ is scored against Paris
$h+2$	.	ID for .	The logit produced at $h+1$ is scored against .
$h+3$	end token	end-token ID	The logit produced at $h+2$ is scored against the end token

Internally, the effective custom-loss computation looks like this:

shift_logits = logits[:, :-1, :]
shift_labels = labels[:, 1:]

loss = cross_entropy(
    shift_logits.reshape(-1, shift_logits.size(-1)),
    shift_labels.reshape(-1),
    ignore_index=-100,
)

The crucial implication is that the final prompt token, usually the assistant header, is precisely where the model learns to predict the first assistant content token. Masking the header’s own label does not suppress learning of the first answer token.

If you build a separate Boolean role mask and then manually shift tensors, be explicit about which convention it represents:

• a mask aligned to token identities says whether each token is assistant content;
• a mask aligned to logit positions says whether the next-token prediction made at that position is scored.

These conventions differ by one position. Frameworks that expose a “mask rolling” step are resolving this alignment issue.

---

Tokenization and batching details that determine correctness

Tokenize the rendered conversation, not independent natural-language fragments

The safe single-turn pattern above renders the prompt with the model’s template, then appends target IDs encoded with add_special_tokens=False. This avoids accidental duplication of a beginning-of-sequence token or an assistant header.

Avoid these failure modes:

• Handwritten control tokens. A token such as <|assistant|> may be wrong for the selected checkpoint, may require a newline, or may not tokenize as intended.
• Duplicate generation headers. If add_generation_prompt=True already adds the assistant header, do not add another one manually.
• Missing stop token. Without a supervised end token, the model receives no explicit signal to end the response.
• Tokenizing answer text with default special tokens. This can insert a second beginning-of-sequence marker between prompt and answer.
• Truncating away all target tokens. An example with only -100 labels has no learning signal and should be rejected.

For multi-turn conversations, assistant-only masking is more subtle. You may want the model to learn from every assistant turn, while system and user turns remain context-only. In that case, derive the mask from the rendered template rather than guessing boundaries from decoded text. Many modern chat templates can expose assistant-generation spans; TRL’s assistant_only_loss=True relies on the template having appropriate generation markers.

A conceptual multi-turn representation is:

tokenized = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_assistant_tokens_mask=True,
)

input_ids = tokenized["input_ids"]
assistant_mask = tokenized["assistant_masks"]

labels = input_ids.clone()
labels[assistant_mask == 0] = IGNORE_INDEX

The exact returned field names and supported options depend on tokenizer and library versions. The invariant does not: inspect the final labels and verify that their non-ignored tokens decode to precisely the assistant targets you intended.

Padding: attention and loss must agree

Within a batch, examples have different lengths. A conventional right-padded batch might look like:

Example 1 input:  [prompt ... answer EOS PAD PAD]
Example 1 labels: [-100  ... answer EOS -100 -100]

Example 2 input:  [prompt ... answer EOS PAD]
Example 2 labels: [-100  ... answer EOS -100]

Padding is necessary to form a rectangular tensor, but it must never become a target. Even when a tokenizer reuses the EOS token as its padding token, padded positions in labels still need to be -100.

A simple right-padding collation pattern is:

import torch

def collate_sft_examples(examples, tokenizer):
    tokenizer.padding_side = "right"

    model_inputs = [
        {
            "input_ids": ex["input_ids"],
            "attention_mask": ex["attention_mask"],
        }
        for ex in examples
    ]

    batch = tokenizer.pad(
        model_inputs,
        padding=True,
        return_tensors="pt",
    )

    batch_size, max_length = batch["input_ids"].shape
    labels = torch.full(
        (batch_size, max_length),
        fill_value=IGNORE_INDEX,
        dtype=torch.long,
    )

    for row, ex in enumerate(examples):
        length = len(ex["labels"])
        labels[row, :length] = torch.tensor(ex["labels"])

    batch["labels"] = labels
    return batch

Right padding is often convenient during training because original tokens retain the same leading positions. Left padding can also work, but label alignment and position handling must be implemented consistently.

Packing is an efficiency optimization, not a formatting repair

Padding can waste a great deal of compute when examples vary widely in length. Packing concatenates multiple short examples into a larger fixed-length sequence. It can raise token throughput substantially, especially for instruction data containing many short responses.

But packing creates an isolation requirement: tokens in one conversation must not attend to prior, unrelated conversations as though they were part of the same dialogue. A correct packed implementation therefore tracks sequence boundaries and applies variable-length or block-structured attention behavior. It also preserves each example’s loss mask.

For now, treat packing as a second-stage optimization:

1. First verify that each unpacked example has correct template rendering, token IDs, labels, and termination.
2. Then pack examples only if the training stack preserves attention boundaries, position semantics, and per-token loss masks.

This ordering is useful in production debugging: bad labels are a correctness bug; padding overhead is a performance problem. Solving the latter cannot compensate for the former.

---

A preflight inspection checklist

Before launching a costly job, run this inspection on several randomly selected examples from every dataset shard:

• Rendered transcript: decode input_ids with special tokens visible. Confirm the exact role delimiters and turn endings expected by the model.
• Target-only decode: collect labels not equal to -100 and decode them. It should contain the intended assistant answer and its desired end token, nothing from system or user content.
• Boundary check: ensure the first supervised token is the first assistant-content token, not the user’s last token or a duplicated role marker.
• Padding check: every padded position has attention_mask = 0 and labels = -100.
• Signal check: every retained training example has at least one supervised label; in practice, it should usually contain a meaningful number of target tokens.
• Serving compatibility: format a held-out prompt with the same chat template used in training. The prefix should match the prefix that precedes target responses in the SFT data.

These checks are inexpensive and highly diagnostic. They convert “training loss is strange” into a concrete question about a decoded training record.

---

Key takeaways

SFT uses the same causal next-token objective as pretraining, but applies it to curated demonstrations of assistant behavior. The training example must be one model-specific, chat-templated token sequence; the model sees the full context, while labels select only assistant response tokens for loss.

Use -100 for ignored labels, including prompt and padding positions. Keep labels aligned with input_ids when using standard causal-LM APIs, because those APIs usually perform the one-token shift internally. Finally, treat the chat template and end-of-turn policy as compatibility contracts between your data pipeline and serving system.

Next, we will examine LoRA adapter parameter counts and see how adapter rank controls the trainable footprint of an SFT run.