LLM Inference Mechanics and Optimizations

Distinguish the computational and memory-access behavior of prefill and autoregressive decode.

Compute time to first token, inter-token latency, end-to-end latency, and token throughput from an inference timeline.

Calculate KV-cache memory from batch size, sequence length, layer count, head dimensions, and precision.

Evaluate how weight and KV-cache quantization affect capacity, bandwidth demand, latency, and model quality.

Explain how FlashAttention reduces high-bandwidth-memory traffic without approximating attention.

Calculate when speculative decoding improves performance from acceptance rate, draft-model cost, and verification cost.