Impact of Weight and KV-Cache Quantization on Capacity, Bandwidth, Latency, and Quality

Welcome back. Last time, you derived KV-cache memory from active token positions and saw why long contexts and concurrent decoding can consume more GPU memory than the model weights themselves. You also established an important baseline: the cache’s storage precision is an independent multiplier, even when weights use a different precision.

This lesson turns that observation into a serving decision. You will evaluate weight quantization and KV-cache quantization separately, then together, in terms of GPU capacity, high-bandwidth-memory traffic, latency, and model quality. The central systems question is not “what is the lowest bit width?” It is: which tensor is limiting this workload, and what quality risk is acceptable to relieve that limit?

---

Quantization: deliberate numerical compression

A trained model is mostly numerical tensors. Quantization stores or computes some of those tensors using fewer bits than their original representation. Instead of preserving every FP16 or BF16 value exactly, the runtime maps values to a limited set of representable levels and later reconstructs an approximation.

For an integer quantizer, a simplified affine formulation is:

$$q=\mathrm{clip}\left(\mathrm{round}\left(\frac{x}{s}+z\right),q_{\min },q_{\max }\right)$$ $$\hat{x}=s(q-z)$$

Here, $x$ is the original value, $q$ is the low-bit stored value, $s$ is a scale, $z$ is a zero point, and $\hat{x}$ is the reconstructed approximation. Two errors are unavoidable:

• Rounding error: a value falls between available levels.
• Clipping error: a value lies outside the chosen representable range.

The practical benefit is straightforward: fewer bits usually means less device memory, fewer bytes moved through HBM, and potentially faster matrix operations on hardware with suitable low-precision kernels.

FP8 and INT8 each occupy one byte per scalar, but they are not interchangeable:

• INT8 has a fixed integer range, usually paired with explicit scale factors.
• FP8 has exponent and mantissa fields, giving it a larger dynamic range but less uniform resolution.
• INT4 and FP4 can reduce storage to roughly half a byte per value before accounting for metadata and packing.
• FP16 and BF16 consume two bytes per value.

Read the following sections for the essential distinction among weights, activations, and KV cache, then the role of scaling and quantization granularity.

A scale is itself metadata that must be stored and read. Therefore, “4-bit quantization” is a useful shorthand, not an exact end-to-end memory guarantee. Small quantization groups generally improve fidelity because an outlier affects fewer values, but they add scale metadata and can complicate kernels.

---

Two targets, two bottlenecks

For inference, separate the relevant tensors before considering a bit width.

Target	Lifetime	Main serving effect	Most relevant phase
Weights	Loaded while the model replica is resident	Model-fit capacity and repeated HBM reads	Prefill and decode
Activations	Temporary, input-dependent intermediate tensors	Compute-kernel efficiency and temporary memory	Primarily prefill
KV cache	Per active request, growing with retained tokens	Long-context capacity and attention reads	Mostly decode

This lesson concentrates on weights and KV cache. The labels commonly used in serving configurations make the separation explicit:

• Weight-only quantization, such as W4A16, stores weights at 4 bits while keeping activations at 16 bits.
• Weight-and-activation quantization, such as W8A8, uses low precision for both inputs to major matrix multiplications.
• KV-cache quantization stores cached keys and values, often in FP8 or an integer format, while the attention computation may reconstruct them in a higher precision.

The following attention diagram shows where KV-cache quantization occurs. At each new token, the model computes new $K_t$ and $V_t$, quantizes them for storage, then dequantizes prior cached keys and values when calculating attention scores and the weighted value sum.

The important implication is that weight and KV-cache compression solve different problems:

• If the model cannot fit on one GPU before serving starts, weight quantization is the immediate lever.
• If the model fits but long prompts or active concurrency exhaust memory, KV-cache quantization is often the more targeted lever.
• If decode is limited by both repeatedly loading weights and reading a large cache, using both may be justified.

---

Capacity: calculate the memory released by each choice

Weight capacity

If a model has $N$ parameters, a first-order weight-storage estimate is:

$$M_{\mathrm{W}}\approx N\times p_{\mathrm{W}}+M_{\mathrm{metadata}}$$

where $p_{\mathrm{W}}$ is bytes per stored weight and $M_{\mathrm{metadata}}$ includes scales, zero points, packed-tensor alignment, and format-specific overhead.

Ignoring metadata temporarily:

Weight format	Bytes per parameter	Storage relative to FP16
FP16 or BF16	2	$1$
FP8 or INT8	1	$1/2$
FP4 or INT4	$0.5$	$1/4$

For example, 70 billion FP16 parameters require approximately:

$$70\times 10^9\times 2=140\times 10^9\text{ bytes}$$

or about $130$ GiB before runtime overhead. At nominal 4-bit weight storage, the same parameters require about one quarter as much payload storage:

$$140\text{ GB}\times \frac{4}{16}=35\text{ GB}$$

This can change the deployment topology entirely: a model that previously required weight sharding across multiple GPUs may fit in one GPU’s memory. But its actual serving performance still depends on whether the inference engine has an efficient kernel for that model architecture and quantization format.

KV-cache capacity

From the previous lesson, KV-cache payload for a uniform batch is:

$$M_{\mathrm{KV}}=2\times B\times S\times L\times H_{\mathrm{KV}}\times d_h\times p_{\mathrm{KV}}$$

Quantizing the cache changes $p_{\mathrm{KV}}$, while the number of layers, KV heads, and cached token positions stays the same.

If the baseline cache uses FP16 and the quantized cache uses FP8:

$$\frac{M_{\mathrm{KV},FP8}}{M_{\mathrm{KV},FP16}}=\frac{1}{2}$$

Nominally, FP8 KV cache halves cache payload. A 4-bit cache reduces it to one quarter of FP16 payload, subject to scale metadata and implementation constraints.

For the prior GQA example, 16 active requests at 8,192 tokens required 16 GiB of FP16 KV cache. With FP8 cache storage:

$$M_{\mathrm{KV},FP8}\approx 16\text{ GiB}\times \frac{1}{2}=8\text{ GiB}$$

The 8 GiB saved can support more active cached tokens, a longer maximum context, or a larger safety margin against allocator fragmentation.

Combined capacity calculation

Consider an 80 GiB GPU with this measured serving budget:

Component	FP16-cache baseline
Model weights	30 GiB
Runtime, workspaces, safety margin	10 GiB
Memory available to KV cache	40 GiB

Suppose each request at its target context length needs 1 GiB of FP16 cache. The idealized capacity limit is:

$$B_{\max ,FP16}=\lfloor \frac{40\text{ GiB}}{1\text{ GiB per request}}\rfloor =40$$

Now evaluate two changes.

Case 1: FP8 KV cache only. Each request needs roughly 0.5 GiB of cache:

$$B_{\max ,FP8KV}=\lfloor \frac{40}{0.5}\rfloor =80$$

Case 2: 4-bit weights plus FP8 KV cache. Assume actual quantized weights plus their metadata consume 8 GiB rather than the idealized 7.5 GiB. Then cache-available memory becomes:

$$80-8-10=62\text{ GiB}$$

and idealized request capacity becomes:

$$B_{\max ,W4,FP8KV}=\lfloor \frac{62}{0.5}\rfloor =124$$

These are memory bounds, not throughput guarantees. They assume all requests reach the target length, ignore block-allocation slack, and say nothing about whether the GPU can produce tokens fast enough to meet an inter-token latency objective. Still, the calculation reveals the division of labor: weight quantization released 22 GiB of fixed replica memory, while KV-cache quantization halved the variable per-request cost.

---

Bandwidth and latency: why lower precision can make decoding faster

Capacity determines how much work can be admitted. Bandwidth strongly influences how fast decode proceeds once that work is admitted.

During autoregressive decode, the GPU processes few new tokens at a time. The arithmetic per token is relatively modest compared with the large amount of state read from HBM:

1. Model weights must be accessed across transformer layers.
2. Attention reads the accumulated keys and values for every active sequence.
3. The current token’s new key and value must be appended to the cache.

A useful lower-bound model is:

$$T_{\mathrm{decode}}\gtrsim \max \left(\frac{D_{\mathrm{W}}+D_{\mathrm{KV}}}{B{W}_{\mathrm{effective}}},\frac{F}{P_{\mathrm{effective}}}\right)+T_{\mathrm{quant}\mathrm{overhead}}$$

where:

• $D_{\mathrm{W}}$ is weight data movement;
• $D_{\mathrm{KV}}$ is KV-cache data movement;
• $B{W}_{\mathrm{effective}}$ is achieved memory bandwidth;
• $F$ is required arithmetic;
• $P_{\mathrm{effective}}$ is achieved compute throughput;
• $T_{\mathrm{quant}\mathrm{overhead}}$ represents packing, dequantization, scale handling, and other overhead not completely hidden by the kernel.

This is a diagnostic model, not a promise of an exact latency. It emphasizes a systems fact: a bit-width reduction helps only if it reduces the actual limiting cost more than it adds conversion and kernel overhead.

Watch this short explanation of why decode tends to be memory-bound and why compression can improve token generation speed.

Weight quantization and latency

Weight quantization can improve both prefill and decode, but the mechanism differs by configuration.

• In weight-only quantization, the engine loads compact weights and often dequantizes them within a fused matrix-multiply kernel. This can reduce HBM traffic substantially, especially in decode.
• In weight-and-activation quantization, compatible tensor cores can execute low-precision matrix multiplications at higher throughput. This offers a larger potential speedup, but requires careful activation handling and hardware support.
• In prefill, large matrix multiplications have enough work to use compute hardware efficiently. If prefill is compute-bound, merely reducing weight storage may not improve TTFT much.
• In decode, lower arithmetic intensity makes reduced weight traffic especially valuable. However, batching helps reuse weights across several requests, so the per-request weight cost is not simply one full model read per request.

KV-cache quantization and latency

KV-cache quantization has its largest effect when contexts are long or many cached sequences are active.

Each decode step must attend over prior positions. As retained context grows, attention must read more K and V data. Halving cache precision approximately halves the cache payload that must be read, assuming the attention kernel can use the compressed cache efficiently.

Its impact differs by user-visible metric:

Metric	Likely effect of KV quantization
TTFT	Often limited for short prompts; potentially helpful for long-prompt attention and memory pressure
Inter-token latency	Can improve materially at long contexts, where reading cached K and V is expensive
Throughput	Can rise by allowing more active tokens and reducing cache bandwidth demand
Queueing delay	May fall if freed capacity prevents admission throttling; may rise if the new capacity is used to form excessively large batches

The final row matters in a production design. A service can use freed memory to admit more requests and increase aggregate tokens per second, yet produce worse tail latency if its scheduler over-batches decode work. Quantization enlarges the feasible operating region; it does not choose the scheduling policy for you.

---

A bandwidth thought experiment

Suppose one batched decode step has an approximate FP16 data-movement budget of:

Data read or written	Baseline traffic
Weights	30 GiB
KV cache across active sequences	16 GiB
Total, ignoring smaller tensors	46 GiB

Now compare formats using a deliberately simple payload model:

Configuration	Approximate traffic	Ideal bandwidth-limited speedup
FP16 weights, FP16 KV	46 GiB	$1.0$
4-bit weights, FP16 KV	$8+16=24$ GiB	$46/24\approx 1.9$
FP16 weights, FP8 KV	$30+8=38$ GiB	$46/38\approx 1.2$
4-bit weights, FP8 KV	$8+8=16$ GiB	$46/16\approx 2.9$

The 8 GiB estimate for 4-bit weights intentionally includes some allowance beyond the ideal 7.5 GiB for metadata and packing. The speedups are upper-bound intuition, not benchmark results. Real performance can be lower because:

• data may be reused in caches rather than fetched from HBM;
• dequantization uses instructions and registers;
• low-bit kernels may have lower occupancy or less favorable tensor shapes;
• some operations remain FP16 or BF16;
• prefill and decode have different bottlenecks.

Still, the calculation guides the right profiling question: is the dominant byte volume weights, KV cache, or neither?

---

Quality: what precision reduction changes in the model

Quantization error is not uniformly harmful. Some layers and channels tolerate approximation well; others can be unusually sensitive because they contain outliers or make a disproportionate contribution to model outputs.

For weights, the error appears in projections and feed-forward layers throughout the network. For the KV cache, the error enters attention in two ways:

• Quantized keys perturb attention scores before softmax, potentially changing which positions receive attention.
• Quantized values perturb the vectors combined after attention weights are calculated.

These errors may be more visible in workloads that require exact retrieval from a long context, multi-step reasoning, code generation, structured output, or strict safety behavior. A generic chat benchmark can look stable while a product-critical long-context or tool-use workflow regresses.

The most reliable ways to improve quality at a given bit width are not simply “use a larger model.” They are quantization-method choices:

• Finer granularity: use separate scales per channel or block rather than one scale for a whole tensor.
• Calibration: collect representative activation statistics and choose scales that reflect production inputs.
• Outlier-aware methods: preserve or rescale particularly important channels more carefully.
• Mixed precision: leave sensitive layers, tensors, or operations at a higher precision.
• Quantization-aware training (QAT): simulate quantization effects during fine-tuning when post-training quantization does not meet the quality bar.

Continue NVIDIA’s article for its overview of quality-preserving algorithms and the distinction between post-training quantization and QAT.

A practical inference policy is usually conservative at first:

1. Establish the BF16 or FP16 baseline on representative traffic.
2. Test a moderate format such as FP8 or INT8 for KV cache and weights where supported.
3. Measure quality, TTFT, inter-token latency, throughput, HBM use, and failure modes.
4. Move to 4-bit weights or lower-bit cache storage only when the capacity or bandwidth gain addresses a measured constraint.
5. Keep a higher-precision fallback for sensitive models, tenants, or request classes if the product can support it.

Do not infer quality from a few appealing chat responses. Treat quantization as a model variant with its own release evaluation and regression gates.

---

Choosing the right lever

Use this decision guide when discussing serving design in an interview or architecture review.

Observed constraint	First quantization candidate	Why
Weights do not fit on the target GPU	Weight-only 8-bit or 4-bit	Reduces fixed per-replica memory
Model fits, but long-context requests cause OOM	FP8 or lower-bit KV cache	Reduces memory per active cached token
Decode inter-token latency is bandwidth-bound and weight reads dominate	Weight quantization with optimized decode kernels	Reduces repeated weight movement
Decode is bandwidth-bound and context is very long	KV-cache quantization	Reduces attention reads of stored K and V
Prefill is compute-bound	Weight-and-activation low precision, if hardware supports it	Can increase matrix-multiply throughput
Quality-sensitive long-context retrieval regresses	Higher-precision KV cache, finer scales, or selective mixed precision	Keys and values directly affect attention behavior
Both fixed weights and variable cache exhaust memory	Combine weight and KV-cache quantization	Releases fixed memory and lowers per-request memory

There is one final operational check: make sure the serving runtime actually supports the chosen format efficiently on the deployed accelerator. A mathematically attractive W4 or FP8-KV configuration can underperform if it causes unfused dequantization, unsupported attention kernels, excessive scale overhead, or an unfavorable parallelism layout.

---

Key takeaways

Quantization is not one switch. Weights, activations, and KV cache are different tensors with different lifetimes and serving consequences.

• Weight quantization reduces fixed model-residency memory and can reduce repeated HBM traffic during both prefill and decode.
• KV-cache quantization reduces memory per active retained token and can lower long-context attention bandwidth demand during decode.
• Relative to FP16, FP8 or INT8 has roughly half the nominal payload size; 4-bit formats have roughly one quarter, before metadata and runtime effects.
• Decode frequently benefits from quantization because it is often memory-bandwidth-bound, but conversion overhead and kernel quality determine realized speedup.
• Lower precision can degrade model quality through rounding and clipping errors. Long-context retrieval, structured output, reasoning, and safety evaluations are essential validation targets.
• The correct choice follows the bottleneck: quantify weights, cache, bandwidth, and workload shape before selecting a format.

Next, you will examine FlashAttention: an optimization that reduces high-bandwidth-memory traffic for exact attention without approximating the attention result.