Fine-Tuning and Preference-Based Post-Training

Prepare a supervised fine-tuning example with correct prompt formatting, tokenization, and loss masking.

Calculate LoRA trainable-parameter counts and explain the effect of adapter rank.

Compare full fine-tuning, adapter-based PEFT, LoRA, and QLoRA under memory, quality, and deployment constraints.

Construct a preference-data example and explain the objective of a reward model.

Trace the RLHF pipeline from preference collection through reward modeling, PPO updates, and KL control.

Compare RLHF and direct preference optimization and select between them for a stated product constraint.