KV cache explained: why a long context window eats your VRAM

You load a model and it takes 6 GB, which is fine. You paste in a long document and the runtime stops with an out-of-memory error, though the weights are the same size they were a minute ago. What grew is the KV cache, a block of memory that scales with the length of the conversation and that most size guides never mention.

What the cache holds

Every time a transformer reads a token, each layer computes a key and a value vector for it. Later tokens need those vectors again, and recomputing them would be ruinous, so the runtime keeps them. That store is the KV cache. It holds one key and one value per token, per KV head, per layer, for the whole window, and its size follows from the model's architecture and your context length:

cache = 2 x layers x kv_heads x head_dim x tokens x bytes_per_element

For Llama 3.1 8B that is 2 x 32 x 8 x 128 x 2 bytes, which is 128 KB per token. It sounds trivial and is not.

Context	KV cache	Total memory
4k	0.50 GB	5.13 GB
8k	1.00 GB	5.63 GB
16k	2.00 GB	6.63 GB
32k	4.00 GB	8.63 GB
64k	8.00 GB	12.6 GB
128k	16.0 GB	20.6 GB

We checked the formula against a real run rather than trusting it. On an RTX 3050 with Qwen2.5 7B, going from a 2k to a 32k window grew the model's resident memory by 1.69 GB, and the formula predicts 1.64 GB. Everything else in the run was constant. The cache is the part that moves, and it moves exactly as the arithmetic says.

Why two models of the same size differ so much

The kv_heads term in that formula is a design choice, and it varies by a factor of 32 across models that look alike on a download page. The original transformer gave every attention head its own key-value pair. Grouped-query attention shares one pair across a group of query heads, commonly 4 or 8 of them, and the quality loss is small. The cache shrinks by the group size.

Almost every model released since 2024 uses it, which is why old advice about context memory overstates the cost. Older and smaller models often do not. DeepSeek Coder 6.7B keeps all 32 KV heads and needs 16.0 GB of cache at 32k, against 4.00 GB for Llama 3.1 8B, which is 4.0x the memory for a model with fewer parameters. This is the gap a rule of thumb cannot express, and the reason every model page here shows the cache separately from the weights.

Four designs that keep it small

Grouped-query attention is the baseline. Past it, models cut the cache in three more ways, and reading the real layout of each one is the part we think matters most.

Sliding-window attention caps most layers at a fixed window, so their cache stops growing however long the conversation gets. Gemma 3 27B does this in 51 of its 62 layers with a 1,024-token window, and only the other 11 keep growing. At 128k its cache is 11.4 GB, where treating every layer as full would give 62.0 GB. gpt-oss alternates full and windowed layers one for one, which halves its cache at long context.

Hybrid linear attention goes further. Qwen3.5 and Qwen3-Next replace 3 layers in 4 with a linear layer that keeps a small fixed state instead of a cache, so only 1 layer in 4 grows. Qwen3.5 9B needs 4.05 GB at 128k. Latent compression, used by DeepSeek, projects keys and values into one small vector per layer, so DeepSeek R1 holds 8.58 GB at 128k despite having 685 billion parameters.

None of this is visible from the parameter count. A calculator that treats every layer as full overstates the cache of a Qwen3.5 model by about 4 times, and one that treats an alternating model as fully windowed understates it badly. Our own catalog had both mistakes until we started reading the per-layer types from each model's configuration, which is a reminder of how easy they are to make.

What to do about it

Set the window to what you need rather than the maximum. Ollama chooses a default from your VRAM: 4k below 24 GiB, 32k between 24 and 48, 256k above that, and you change it with OLLAMA_CONTEXT_LENGTH. In llama.cpp it is -c. A model loaded at 128k reserves that memory whether you use it or not, and setting it to the 8k your conversations actually take frees gigabytes at once.

Then consider storing the cache at 8 bits. In Ollama that is OLLAMA_KV_CACHE_TYPE=q8_0, about half the memory, and we regard it as the best first move when you are a gigabyte short. Watch parallelism as well. Ollama's documentation says required memory scales with OLLAMA_NUM_PARALLEL times the context length, so a second parallel slot doubles the cache.

When you read a review that says a model "runs comfortably in 8 GB", ask what window it was measured at. Short windows are not dishonest, they just answer a different question from the one you have if you plan to paste in documents. The guide to total VRAM puts the cache together with the other two terms.