Estimating KV-Cache Memory Requirements

Good to see you again. In the previous lesson, you estimated the memory occupied by model weights and treated that value as a lower bound for serving. This lesson adds the largest request-dependent term in that budget: the key-value (KV) cache.

By the end, you will be able to estimate KV-cache memory from a model configuration, a context length, a batch size, and the cache precision. This is the calculation that turns a statement such as “the weights fit on this GPU” into the more useful question: “How many long, active requests can this system actually serve?”

---

Why inference needs a KV cache

An autoregressive language model generates one token at a time. When it generates the next token, self-attention needs access to the representations of prior tokens in the sequence.

Without a cache, generating token $t$ would repeatedly recompute attention-related information for tokens $1$ through $t-1$. The KV cache avoids that repetition. For every processed token and every transformer layer, the engine stores:

• the key vector, $K$;
• the value vector, $V$.

For the next generated token, the model computes a new query, key, and value for that token, appends the new key and value to the cache, and reuses all earlier cached keys and values.

The cache is therefore a deliberate trade-off:

• Benefit: much less repeated computation during decoding.
• Cost: memory grows as tokens accumulate.

During prefill, the model processes the prompt and writes K and V entries for all prompt tokens. During decode, it adds entries one generated token at a time. Thus, a sequence of 3,000 prompt tokens followed by 1,000 generated tokens requires cache capacity for roughly 4,000 token positions.

---

The memory formula

For a typical decoder-only transformer, the approximate KV-cache memory is:

$$M_{\mathrm{KV}}=B\times T\times 2\times L\times H_{\mathrm{KV}}\times D_{\mathrm{head}}\times S$$

where:

Symbol	Meaning
$B$	Batch size: number of active sequence slots
$T$	Tokens stored per sequence, including prompt and generated tokens
$2$	One tensor for keys and one for values
$L$	Number of transformer layers
$H_{\mathrm{KV}}$	Number of key-value heads
$D_{\mathrm{head}}$	Dimension of each attention head
$S$	Bytes per cached element

The result is in bytes.

The factor of two is easy to forget: a cache contains both K and V, not merely one attention tensor. The layer count is equally important: each layer needs its own cache, because each layer computes its own attention representation.

For a model with standard multi-head attention, the number of KV heads equals the number of query heads:

$$H_{\mathrm{KV}}=H_{\mathrm{Q}}$$

In that common case:

$$H_{\mathrm{KV}}\times D_{\mathrm{head}}=D_{\mathrm{model}}$$

So the formula can be written more compactly as:

$$M_{\mathrm{KV}}=B\times T\times 2\times L\times D_{\mathrm{model}}\times S$$

This is the form you will often see in introductory sizing guides.

The important modern-model detail: KV heads, not always hidden size

Many newer models use grouped-query attention (GQA) or multi-query attention (MQA). These architectures use many query heads but fewer KV heads. They reduce the cache footprint because fewer distinct key and value vectors are stored per token.

For these models, use:

$$H_{\mathrm{KV}}\times D_{\mathrm{head}}$$

not the full model hidden size.

For example, a model might have:

$$H_{\mathrm{Q}}=32$$

but only:

$$H_{\mathrm{KV}}=8$$

With the same head dimension, its cache requires one quarter as much memory as a standard 32-head KV design. This is one reason architecture details in a model’s configuration file matter for inference capacity planning.

---

Precision: cache memory is separate from weight memory

The last term, $S$, is the number of bytes stored for each key or value element.

KV-cache representation	$S$, bytes per element
FP32	$4$
FP16 or BF16	$2$
INT8	$1$

A serving engine may store weights in 4-bit form while using FP16 or BF16 for the KV cache. Do not assume that the cache has the same precision as the weights.

For a baseline estimate, FP16 or BF16 KV cache means:

$$S=2$$

Cache quantization can reduce this number, but exact memory use can then include quantization metadata, temporary buffers, or a small higher-precision residual cache. For initial capacity estimates, use the format configured by the inference engine and treat the result as approximate.

---

A reliable calculation workflow

When inspecting a model configuration, collect these values:

1. Number of layers, $L$.
2. Number of KV heads, $H_{\mathrm{KV}}$.
3. Head dimension, $D_{\mathrm{head}}$.
4. Maximum active tokens per sequence, $T$.
5. Batch size or number of active sequence slots, $B$.
6. KV-cache bytes per element, $S$.

Then calculate in two stages.

First, find cache bytes per token, per sequence:

$$m_{\mathrm{token}}=2\times L\times H_{\mathrm{KV}}\times D_{\mathrm{head}}\times S$$

Then multiply by sequence length and batch size:

$$M_{\mathrm{KV}}=B\times T\times m_{\mathrm{token}}$$

This two-stage method makes it easier to sanity-check your arithmetic. The per-token value tells you the memory cost of extending one active request by one additional token.

---

Worked example: a standard-attention 7B-style model

Consider a model with the following architecture:

Configuration item	Value
Layers, $L$	$32$
KV heads, $H_{\mathrm{KV}}$	$32$
Head dimension, $D_{\mathrm{head}}$	$128$
Cache precision	FP16
Bytes per element, $S$	$2$
Batch size, $B$	$1$
Sequence length, $T$	$4096$

First calculate per-token cache storage:

$$m_{\mathrm{token}}=2\times 32\times 32\times 128\times 2$$ $$m_{\mathrm{token}}=524,288\text{ bytes}$$

That is:

$$524,288\text{ bytes}=512\text{ KiB}$$

Now multiply by the context length:

$$M_{\mathrm{KV}}=1\times 4096\times 524,288$$ $$M_{\mathrm{KV}}=2,147,483,648\text{ bytes}$$

This is approximately:

$$2.15\text{ GB}$$

or:

$$2\text{ GiB}$$

So this one request, at 4,096 total cached tokens, uses roughly 2 GB of FP16 KV cache. That matches the common Llama 2 7B example used in inference-sizing references.

The model’s FP16 weights might be about 14 GB. On a 24 GB GPU, a rough initial budget is therefore:

Memory category	Approximate amount
FP16 model weights	14 GB
One 4,096-token KV cache	2 GB
Remaining capacity	8 GB

That apparent 8 GB is not all safely available for additional requests: the engine also needs runtime workspaces, framework allocations, and some practical safety margin. Nevertheless, the calculation immediately shows why “the weights fit” is only the first screen.

---

Scaling the same workload

The formula is linear in both batch size and sequence length. For the same model and precision:

• doubling $T$ doubles KV memory;
• doubling $B$ doubles KV memory;
• doubling both multiplies KV memory by four.

Suppose the model above serves a batch of four sequences, each with capacity for 8,192 cached tokens:

$$M_{\mathrm{KV}}=4\times 8192\times 524,288$$ $$M_{\mathrm{KV}}=17,179,869,184\text{ bytes}$$

That is approximately:

$$16\text{ GiB}$$

The cache has grown from 2 GiB to 16 GiB. The model weights did not change; only the workload configuration changed.

This is a central inference-engineering insight: long contexts and concurrency consume memory together. A configuration that is comfortable for one interactive request can become impossible when several long requests are active.

---

Worked example: grouped-query attention

Now consider a GQA model with:

Configuration item	Value
Layers, $L$	$32$
Query heads	$32$
KV heads, $H_{\mathrm{KV}}$	$8$
Head dimension, $D_{\mathrm{head}}$	$128$
Cache precision	FP16
Bytes per element, $S$	$2$
Batch size, $B$	$8$
Sequence length, $T$	$8192$

Use the eight KV heads, not 32 query heads:

$$m_{\mathrm{token}}=2\times 32\times 8\times 128\times 2$$ $$m_{\mathrm{token}}=131,072\text{ bytes}$$

That is $128$ KiB per token per sequence. The total is:

$$M_{\mathrm{KV}}=8\times 8192\times 131,072$$ $$M_{\mathrm{KV}}=8,589,934,592\text{ bytes}$$

So the total is:

$$8\text{ GiB}$$

Had you incorrectly used all 32 query heads as KV heads, you would estimate 32 GiB instead. That error could cause you to reject a configuration that is actually feasible, or misunderstand why a model can support a much larger context than a similarly sized older architecture.

---

What exactly counts as sequence length?

For KV-cache sizing, $T$ is the number of tokens that must remain available in the cache. In the usual full-context case:

$$T=T_{\mathrm{prompt}}+T_{\mathrm{generated}}$$

If a request has a 6,000-token prompt and the server permits up to 2,000 output tokens, provision for:

$$T=8000$$

not merely the prompt length.

There are two operational interpretations:

• Actual usage: cache grows as tokens are processed. A request that stops after 200 output tokens uses less than its maximum allowance.
• Reserved capacity: many engines allocate or reserve memory based on a configured maximum sequence length or cache pool. In that case, a conservative plan should use the reservation policy, not the average observed output length.

For a simple local baseline, calculate from the maximum prompt plus maximum generation length you intend to support. Later, when benchmarking real workloads, you can compare this safe upper bound with measured active-cache use.

---

Sanity checks before trusting an estimate

Use these checks to catch common mistakes:

Check	What it prevents
Include the factor $2$	Forgetting either keys or values
Multiply by every layer	Treating the cache as if it existed once for the whole model
Use $H_{\mathrm{KV}}$ for GQA or MQA	Overestimating based on query heads or hidden size
Include prompt plus generation tokens	Underestimating the final cache size
Multiply by active sequences	Mistaking a single-request estimate for a concurrent-serving estimate
Use KV precision, not weight precision	Assuming 4-bit weights imply a 4-bit cache
Keep units explicit	Mixing bytes, GB, GiB, token counts, and parameter counts

The estimate is intentionally approximate. Real usage can differ because of memory alignment, engine-managed cache blocks, allocator behavior, metadata, and temporary tensors. But the architecture-level formula is the right first calculation for comparing configurations and rejecting impossible ones early.

---

Key takeaways

• The KV cache stores prior keys and values for every processed token and every transformer layer, avoiding repeated attention computation during generation.
• Approximate KV-cache memory is:

$$M_{\mathrm{KV}}=B\times T\times 2\times L\times H_{\mathrm{KV}}\times D_{\mathrm{head}}\times S$$

• In standard multi-head attention, $H_{\mathrm{KV}}\times D_{\mathrm{head}}$ is typically the model hidden size. In GQA and MQA, use the smaller KV-head count directly.
• Cache memory grows linearly with active sequence length, batch size, layer count, KV dimension, and bytes per cache element.
• Weight quantization and KV-cache precision are separate choices. A model can have compact weights while still using an FP16 or BF16 cache.
• For capacity planning, calculate the cache from the maximum supported prompt plus generated tokens, then leave headroom for engine overhead.

Next, you will shift from memory capacity to user-visible performance: distinguishing time to first token, inter-token latency, end-to-end latency, throughput, and concurrency.