Llama Local Models and Training

Distinguish pretraining, inference, prompt engineering, retrieval-augmented generation, and fine-tuning by the role each plays in a local LLM system.

Create and transform PyTorch tensors while tracking their shapes, dtypes, and devices.

Use embedding and linear layers to map token IDs to logits over a vocabulary.

Calculate cross-entropy loss and perplexity from a small set of next-token predictions.

Use automatic differentiation and an optimizer to train a small PyTorch model.