How much VRAM do I need for a local AI model?

Someone asks this in every local-AI forum every day, and the replies are always a number: "8 GB for a 7B model." That number is right often enough to be dangerous. It is right at a 2k context window on Linux with nothing else running, and wrong by four gigabytes the moment you open a long conversation on a Windows desktop.

The memory a model needs is three things added together, and only the first one depends on the model's size.

The three terms

Weights. The parameters themselves, compressed. At four-bit precision this is roughly 0.6 GB per billion parameters — an 8B model is about 4.8 GB, a 70B is about 42 GB. This is the number everyone quotes, and it is the number on the download page.

The KV cache. Every token in the conversation leaves behind a key and a value vector in every layer, and they stay in memory until the conversation ends. This grows in a straight line with context length. It is why the same model that loaded fine yesterday runs out of memory today when you paste in a long document.

Overhead. Compute buffers, the logits buffer, and the slice the graphics driver keeps for itself. Call it 0.6 to 1.5 GB depending on the backend and the context size.

Then subtract what the operating system already took: Windows keeps roughly 0.8 GB of the display adapter's memory for the desktop, Linux with a desktop about 0.3 GB, a headless server almost nothing. On a 8 GB card that reserve is a tenth of your budget.

Where the rule of thumb breaks

The cache is the interesting term because its size depends on a design choice, not on the parameter count.

An older model with plain multi-head attention stores one key and one value per attention head. A model with grouped-query attention shares each key-value pair across a group of heads — commonly eight heads to one KV head, sometimes thirty-two to one. The cache shrinks by exactly that ratio.

The practical effect is large. Take two models both in the seven-to-nine billion range: Mistral 7B v0.3 with eight KV heads, and an older 7B with all thirty-two. At a 32k context the first spends about 4 GB on cache and the second about 16 GB. Same class of model, same download size, twelve gigabytes apart in what your card actually needs. No single number can express that, which is why every model page here shows the cache separately from the weights.

Two further wrinkles cut the other way. Some models — the Gemma family among them — use sliding-window attention, where most layers only look back a fixed distance and stop accumulating cache past it. Others, notably the DeepSeek line, compress the cache into a small latent vector per layer, which is how a model with hundreds of billions of parameters ends up with a smaller cache than a 70B.

What that means for real cards

Rounding to what these work out to in practice, at a 4-bit quantisation and an 8k window:

Card	Usable	Comfortable model size	With a long context
6 GB	~5.2 GB	Up to 4B	3B and below
8 GB	~7.2 GB	7B to 9B	4B to 7B
12 GB	~11.2 GB	12B to 14B	8B to 12B
16 GB	~15.2 GB	20B to 24B	14B
24 GB	~23.2 GB	30B to 32B	24B
48 GB	~47.2 GB	70B	32B to 49B

The right-hand column is the one people forget when they buy. If you intend to feed the model whole files, budget one size class down from what the weights alone would allow.

When it does not fit

Not fitting is not a wall. Every serious runtime will put as many layers as fit on the GPU and run the remainder on the CPU. The model works; it just reads part of itself out of system RAM, which is roughly ten times slower than GPU memory.

The arithmetic is unforgiving. Put half the layers on the CPU and you do not get half the speed, you get something closer to a fifth of it, because the slow half dominates every token. A 70B model with forty of its eighty layers offloaded lands somewhere around one to two tokens per second on a fast desktop. That is unusable for chat and perfectly fine for a batch job you leave running.

The lesson is that memory capacity decides whether, and memory bandwidth decides how fast. They are separate purchases. A 16 GB card with slow memory will load models that a 12 GB card cannot, and then answer more slowly than the 12 GB card does on the models they share.

The shortest useful answer

If you want one sentence to give someone: take the model's parameter count in billions, multiply by 0.6 for a four-bit download, add a gigabyte of overhead, then add the cache — which you have to look up, because it depends on the model's attention design and on how long you want the conversation to be.

Or put your card into the calculator and read the number.