AI Systems Engineering for LLMs

Trace the full lifecycle of an HTTP request to token output by instrumenting a live LLM serving stack.

Map the conceptual stages of inference (tokenization, prefill, decode, detokenization) to the corresponding function calls in a Hugging Face Transformers generate pipeline.

Measure wall-clock time for each inference stage using Python profiling tools on a small model.

Analyze profiling data to characterize the distinct compute and memory profiles of the prefill vs. autoregressive decode phases.

Measure and compare time-to-first-token (TTFT) vs. inter-token latency (ITL) on a running model to diagnose performance characteristics.

Set up a local inference environment with PyTorch and CUDA on a single consumer GPU.

Load and serve a small model (e.g., Phi-3-mini) using Hugging Face Transformers.

Profile GPU memory usage and utilization during inference using nvidia-smi and PyTorch CUDA utilities.

Identify performance bottlenecks in a naive single-GPU serving setup by analyzing GPU utilization and memory traces.

Serve the model behind a FastAPI endpoint and benchmark end-to-end request latency.

Implement a minimal decoder-only transformer from scratch in PyTorch, including embedding, attention, FFN, and output layers.

Write a function to programmatically compute the total parameter count of a transformer from its architectural hyperparameters.

Calculate the VRAM required to store model weights for a given parameter count and data type (fp32, fp16, bf16).

Calculate the activation memory consumed during a forward pass for a given batch size and sequence length.

Analyze the forward pass of a decoder block, mapping operations to their respective compute (MatMul) and memory (data movement) costs.

Calculate the total KV cache size for a given model architecture, batch size, and sequence length.

Apply VRAM and KV cache calculations to a 70B model to explain its large memory footprint.

Distinguish between compute-bound and memory-bandwidth-bound operations in transformer inference.

Calculate the arithmetic intensity for key operations (e.g., matrix multiplications) in attention and FFN layers.

Apply the roofline model concept to determine if an operation is compute- or memory-bound on a specific GPU.

Implement operator fusion (e.g., for QKV projection) to reduce kernel launch overhead and improve memory access patterns.

Apply torch.compile to the model and measure its effect on inference latency and GPU kernel execution.

Profile inference with Nsight Systems to pinpoint compute bottlenecks at the CUDA kernel level.

Profile standard scaled dot-product attention to identify the memory-access bottleneck caused by HBM reads/writes.

Explain the tiling and online softmax algorithm used by FlashAttention to optimize memory access.

Benchmark standard attention vs. a FlashAttention implementation, measuring speedup and memory savings across varying sequence lengths.

Implement Multi-Query Attention (MQA) and measure the reduction in KV cache size compared to standard Multi-Head Attention.

Generalize the MQA implementation to Grouped-Query Attention (GQA) and analyze its trade-off between memory footprint and model quality.

Implement a basic KV cache for autoregressive decoding and measure its memory growth over the generation sequence.

Implement and compare different KV cache eviction strategies (e.g., sliding window, token dropping) and evaluate their impact on generation quality.

Implement prefix caching to reuse KV cache entries for requests that share a common prompt.

Explain the memory fragmentation problem in naive KV cache allocation and the core concept of paged memory management.

Implement a paged KV cache allocator that manages non-contiguous physical memory blocks for storing logical token sequences.

Integrate the paged KV cache into an inference loop and benchmark the reduction in memory waste under a simulated concurrent workload.

Quantize a small model to INT8 using a library like bitsandbytes and compare its memory footprint and inference latency against the fp16 baseline.

Quantize a model to INT4 using an algorithm like GPTQ and measure the resulting VRAM savings and latency change.

Compare the perplexity of INT8 and INT4 quantized models against the original to evaluate the quality vs. memory trade-off.

Explain the core principles of Activation-aware Weight Quantization (AWQ) and how it differs from post-training quantization methods like GPTQ.

Benchmark a weight-only quantized model against one with both weight and activation quantization to compare their performance profiles.

Combine quantization with paged KV cache optimization and measure the cumulative memory savings on a single GPU.

Implement static batching for LLM inference and identify its inefficiency with variable-length inputs and outputs.

Design and implement the core logic for a continuous batching scheduler that manages a queue of incoming requests.

Integrate the scheduler into the inference loop, dynamically composing a batch from multiple requests at each iteration.

Design the data structures and interface for communication between the scheduler and the paged KV cache manager.

Benchmark the throughput (tokens/sec) of continuous batching against static batching under a mixed workload.

Implement and compare scheduling policies (e.g., FCFS, preemption) and analyze their effect on TTFT and ITL.

Set up a PyTorch distributed process group using the NCCL backend for multi-GPU communication.

Explain tensor parallelism and how it shards weight matrices across GPUs to run a single layer in parallel.

Implement tensor-parallel inference for a transformer block across 2+ GPUs, including the necessary all-reduce communication steps.

Explain pipeline parallelism and how it partitions model layers across multiple GPUs.

Implement a forward pass for a pipeline-parallel model, managing micro-batches and data transfers between stages.

Visualize the GPU utilization timeline for a pipeline-parallel implementation to identify and measure the 'pipeline bubble'.

Analyze the communication overhead (all-reduce vs. point-to-point) of tensor vs. pipeline parallelism.

Determine an optimal parallelism configuration (e.g., hybrid TP/PP) for serving a 70B model on a 4xH100 cluster based on memory and communication trade-offs.

Deploy a 7B model using vLLM and benchmark its throughput, latency, and GPU utilization.

Deploy the same model using SGLang and benchmark its baseline performance against vLLM.

Compare the structured generation capabilities (e.g., for JSON output) of SGLang vs. vLLM.

Build and deploy a model using TensorRT-LLM by compiling it into an optimized engine, then benchmark its performance against vLLM and SGLang.

Explain the disaggregated serving architecture of systems like LLM-d and their target use cases for extreme scale.

Create a decision matrix for selecting between vLLM, SGLang, and TensorRT-LLM based on workload characteristics (e.g., latency vs. throughput, structured vs. open-ended generation).

Deploy a 70B+ model on a multi-GPU setup using a chosen framework and load-test it with 500 concurrent users.

Design the high-level component architecture (API server, scheduler, executor, cache manager) and define their interfaces.

Implement the core model executor, capable of running prefill and decode steps for a continuously changing batch of requests.

Integrate the continuous batching scheduler with the model executor.

Integrate the paged KV cache manager into the engine's memory management and execution loop.

Integrate quantized model loading and execution into the engine.

Add tensor-parallel execution capabilities to the engine for multi-GPU deployment.

Implement request ingestion and response streaming layers for an end-to-end test.

Benchmark the custom engine against vLLM under a realistic traffic pattern and analyze its architectural strengths and weaknesses.

Implement an OpenAI-compatible chat completions API with streaming support on top of the custom inference engine.

Design and implement an in-memory session store for managing concurrent, multi-turn conversation states.

Implement stateful multi-turn tool-calling, focusing on KV cache reuse across tool-call rounds and its impact on the scheduler.

Implement token-bucket rate limiting at the API gateway level.

Implement a request queue with a defined capacity and a graceful degradation policy (e.g., 503 Service Unavailable) for overload scenarios.

Profile the full end-to-end API request path to identify and optimize non-inference latency bottlenecks.

Containerize the complete inference service and deploy it.

Load-test the deployed service at high concurrency, measuring performance against defined SLA targets for TTFT and throughput.