Explainer

KV caches: the memory behind a faster conversation

Why a language model stores keys and values, how the cache grows, and what a simple memory calculation can tell you.

By Saud AlotaibiPublished 21 October 20263 min readSources checked 20 October 2026
The takeaway

A KV cache saves repeated computation by keeping attention state. Its memory cost grows with the stored context, so a smaller model is not the only way to serve more conversations.

What the cache remembers

An autoregressive language model generates a sequence one token at a time. In causal attention, earlier tokens cannot read future tokens. Their previously computed keys and values can therefore be reused while the model processes a new token. The KV cache stores those per-layer tensors instead of rebuilding them for the entire prefix at every generation step. [S1]

This is temporary inference state, separate from the model's learned weights. It is also different from a database of saved chat messages. The same text can exist in a conversation log while its attention cache has already been discarded.

Earlier tokens supply stored keys and values. A new token adds its own state, then attends to the available history.
Earlier tokens supply stored keys and values. A new token adds its own state, then attends to the available history. Open full size

A small memory calculation

Here is a hypothetical full-attention decoder, not a measurement of a named model. Assume a single sequence, 32 attention layers, eight KV heads per layer, a head dimension of 128, and two bytes per stored number.

The raw cache payload is approximately:

bytes = 2 × layers × tokens × KV heads × head dimension × bytes per number
      = 2 × 32 × 4096 × 8 × 128 × 2
      = 536,870,912 bytes = 512 MiB

The first factor of two accounts for keys and values. Doubling the stored token count doubles this payload under these assumptions. Four independent sequences of that size require about 2 GiB for their raw KV tensors before sharing or other implementation effects.

This estimate is not total GPU memory. Weights, activations, allocation overhead, serving buffers, and temporary workspaces also matter. Sliding-window layers, compressed state, different numerical formats, and reserved cache capacity change the calculation.

What grouped-query attention changes

The number of query heads and KV heads need not be equal. Grouped-query attention lets several query heads share key/value heads; multi-query attention uses one shared KV head. The GQA paper studies the quality and efficiency tradeoff between these designs. [S2]

For the toy configuration above, changing eight KV heads to four would halve the calculated KV payload. That arithmetic does not establish a latency improvement or guarantee unchanged model quality. Compare trained checkpoints and the actual serving implementation, not just tensor shapes.

Paging solves a different problem

Even when the necessary tensor payload is understood, allocating it efficiently is another challenge. Conversations have different lengths, arrive at different times, and may share a prefix. PagedAttention addresses cache allocation and sharing through a block-based design inspired by virtual memory. [S3]

Think of an illustrative server with three requests that each need a different amount of history. Reserving every request's maximum possible length can leave substantial space unused. Allocating smaller blocks as the requests grow can reduce that waste. It does not make the mathematical state for a unique stored token disappear.

What I would measure first

For a capacity test, I would record the model and cache format, input and output lengths, concurrent requests, allocated cache memory, and peak device memory. Then I would measure both time to the first token and the delay between subsequent tokens. These are my suggested checks, not a result from a benchmark run.

Hugging Face documents dynamic, static, offloaded, and quantized cache strategies with different memory and performance tradeoffs. [S4] A useful comparison holds the workload fixed and explains which bottleneck changed. If a report says only that the model fits on a device, it has not yet answered how many long conversations the application can support.

Sources & further reading