RAG Foundations and a First Working Pipeline

Explain how tokens, context windows, embeddings, and probabilistic next-token generation constrain an LLM application's behavior.

Use embeddings and similarity measures to explain semantic retrieval.

Construct a prompt with explicit context, instructions, constraints, and output format.

Run and modify a scaffolded Python notebook that uses functions, dictionaries, environment variables, and embedding and language-model client calls without exposing credentials.

Prepare source documents for retrieval by applying cleaning, chunking, overlap, and metadata choices.

Create embeddings and store them in a simple vector index.

Implement semantic retrieval and inspect ranked results for relevance.

Construct a grounded generation prompt that uses retrieved context and produces citations.