LLM Inference Workloads and a Local Baseline

Trace a prompt through tokenization, model loading, prefill, autoregressive decoding, and detokenization.

Calculate model-weight memory requirements from parameter count and numerical precision.

Calculate approximate KV-cache memory from model dimensions, context length, batch size, and precision.

Distinguish time to first token, inter-token latency, end-to-end latency, throughput, and concurrency.

Select an open-weight model and quantization that fit within a specified memory budget.

Serve a small quantized language model on Apple Silicon with llama.cpp or MLX and query it from Python.