Why a long context window eats your VRAM

You load a model, it takes six gigabytes, everything is fine. You paste in a long document and the runtime falls over with an out-of-memory error. The weights did not change. What grew is the cache.

What is being stored

When a transformer reads a token, every layer computes a key and a value vector for it. Those vectors are needed again for every token that comes after, so rather than recompute them the runtime keeps them. That store is the KV cache, and it holds one key and one value per token, per attention head, per layer, for the whole conversation.

The size follows directly:

cache = 2 × layers × kv_heads × head_dim × tokens × bytes_per_element

The two is for keys and values. Everything else is the model's architecture except tokens, which is your conversation, and that is the whole problem: the model's contribution is fixed and yours grows without limit.

For an 8B model with 32 layers and 8 KV heads of 128 dimensions, at 16-bit precision, each token costs 128 kilobytes. Sounds trivial. A 32,000-token context is four gigabytes — comparable to the weights themselves at four-bit precision.

Why two models of the same size differ so much

The kv_heads term is a design decision, and it varies by a factor of thirty-two across models that look otherwise identical.

The original transformer gave every attention head its own key-value pair — multi-head attention. Grouped-query attention shares one key-value pair across a group of query heads, commonly eight to one. The model loses very little quality and the cache shrinks eightfold.

Almost every model released since 2024 uses it, which is why old advice about context memory overstates the cost badly. But older and smaller models often do not, and that is where people get caught: a 1.5B model with full multi-head attention can want more cache at long context than an 8B model with grouped queries.

Three ways to make it smaller

Sliding-window attention. Instead of every layer attending to the whole history, most layers only look back a fixed distance — often a few thousand tokens — and one layer in every five or six attends globally. The local layers stop accumulating past their window, so the cache flattens instead of growing. Gemma models are built this way, which is why their long-context memory cost is far lower than their size suggests.

Latent cache compression. Rather than storing full keys and values, project them into one small shared vector per layer and reconstruct on the fly. This is what multi-head latent attention does in the DeepSeek line, and the effect is dramatic: a model with hundreds of billions of parameters can carry a smaller cache than a conventional 70B.

Quantising the cache. The cheapest fix, available on any model. Store the cache at eight bits instead of sixteen and it halves, with a quality cost close to zero. Four bits quarters it, at a real cost to long-range recall. If you are a couple of gigabytes short of the window you want, this is the first thing to try — it is almost always a better trade than dropping the weights another level.

Practical consequences

Set the context you need, not the maximum. Runtimes allocate the cache for the full declared window up front. Loading a 128k-capable model with the window left at 128k reserves that memory whether you use it or not. Setting it to 8k because that is what your conversations look like frees gigabytes instantly. This one setting is behind a large share of "my model does not fit" reports.

Long-context work needs a different card than chat. Two people with the same model and the same GPU can have completely different experiences depending on how they use it. Chat in short bursts and the cache is a rounding error. Feed the model whole codebases and the cache becomes the dominant cost, larger than the weights.

Watch for it in benchmarks. A review saying "runs comfortably in 8 GB" almost always measured at a short context. That is not dishonest, it is just a different question from the one you are asking if you plan to paste in documents.

Every model page here shows the cache separately from the weights, and lists what it costs at four, eight, sixteen, thirty-two, sixty-four and a hundred and twenty-eight thousand tokens, so you can see where your own use lands before you download anything. Start with the calculator or browse the model catalogue.