How much VRAM do you need for a local LLM?

A rule you will see everywhere says an 8B model needs 8 GB of VRAM, and it is wrong in both directions. At 4-bit and an 8k context window, Llama 3.1 8B needs 5.63 GB in all, and the same model at 32k needs 8.63 GB. Only one of the three terms behind that number is the size of the model.

Three terms make up the bill

The first term is the weights, which is parameters times bits per weight, divided by 8. A Q4_K_M file is not 4 bits per weight but about 4.83, because the quantization keeps block scales and a few tensors at higher precision. That puts Llama 3.1 8B at 4.52 GB on disk, and it is why a "4-bit 8B model" never downloads as 4 GB.

The second is the KV cache, the memory that holds a key and a value for every token already in the conversation, in every layer. For Llama 3.1 8B that is 2 x 32 layers x 8 KV heads x 128 dimensions x 2 bytes, or 128 KB per token. At 8k tokens that is 1.00 GB, and at 32k it is 4.00 GB. This is the term that grows while you use the model, and the one most size guides leave out. We wrote a separate guide to the KV cache because it deserves one.

The third is everything else: compute buffers, the runtime's own context, a little scratch space. We measured it on a real run. Qwen2.5 7B on an RTX 3050 held between 0.27 and 0.32 GB of memory that was neither weights nor cache, at every context from 2k to 32k. It stays flat because Ollama and llama.cpp both turn on flash attention by default, which never builds the full table of attention scores. Older advice that makes overhead grow with context describes a runtime without it.

Model	Weights	Cache at 8k	Total at 8k	Cache at 32k	Total at 32k
Llama 3.1 8B Instruct 8.03B	4.52 GB	1.00 GB	5.63 GB	4.00 GB	8.63 GB
Qwen3 14B 14.77B	8.30 GB	1.25 GB	9.56 GB	5.00 GB	13.3 GB
Qwen3 32B 32.76B	18.4 GB	2.00 GB	20.4 GB	8.00 GB	26.4 GB
Llama 3.3 70B Instruct 70.55B	39.7 GB	2.50 GB	42.1 GB	10.0 GB	49.6 GB

What the system takes before you start

Your operating system uses the card too. On the PC we test on, the Windows desktop and a browser already held 1155 MiB of a 6 GB card before any model loaded, which is a sixth of the memory gone. The calculator assumes 0.8 GB on a Windows desktop, 0.3 GB on Linux with a desktop and 0.1 GB on a headless box.

There is a cheap fix if your CPU has integrated graphics: plug the monitor into the motherboard instead of the card, and the desktop moves off the GPU. On a 6 or 8 GB card that is the difference between a model fitting and not. Closing browsers with hardware acceleration helps for the same reason.

On a Mac the rule is different. The GPU shares one pool with the CPU, and macOS lets it wire down roughly two thirds of a small machine and three quarters of a large one by default. A 48 GB Mac gives a model about 36 GB, not 48.

Rules of thumb that hold, and ones that do not

The weights rule holds: about 0.6 GB per billion parameters at Q4_K_M. If you only need to know whether a model downloads onto your disk, that is enough. It stops being enough the moment you care about context, and there are three ways it breaks.

Older models without grouped-query attention pay a cache several times larger. DeepSeek Coder 6.7B needs 16.0 GB of cache at 32k, against 4.00 GB for Llama 3.1 8B, a gap of 4.0x between two models of nearly the same size. Mixture-of-experts models need memory for every expert but only read a few of them per token, so Qwen3 30B-A3B has a 30B memory bill and a 3B speed. And hybrid models like Qwen3.5 keep a real cache in only a quarter of their layers, which is why Qwen3.5 9B needs 4.05 GB at 128k where a conventional model of that depth would want 4 times as much.

When it does not fit

Not fitting is not a wall. Runtimes put as many layers as they can on the GPU and run the rest on the CPU, and we measured what that costs. On the RTX 3050 test machine, Qwen2.5 7B ran at 29.5 tokens per second with every layer on the GPU, at 9.6 with 17 of 29 layers there, and at 4.76 with none. The damage is not proportional to how much you move. The first layers you offload cost little and the last ones cost a lot, because the slow memory ends up setting the pace for every token.

One result surprised us. At a 4k window Ollama kept only 84% of the model on the GPU and ran at 19.4 tokens per second. Forcing every layer onto the card with num_gpu 29 worked, left 0.63 GB free, and ran at 29.5, a gain of 52%. The automatic placement is cautious, and on a tight card the caution costs real speed. llama.cpp's -ngl flag does the same job. We go through both in the guide to offload flags.

Choosing a card for the model you want

Work backwards from the model. Take the total for your model at the window you actually use, add the system's share, and you have the card size. For an 8B model at 8k that lands just under 8 GB on Windows, which is why 8 GB cards are the entry point for this class and why 12 GB is more comfortable. A 14B model wants 12 GB at a short window and 16 GB if you push the context. A 32B wants 24 GB. A 70B wants 48 GB or a machine with unified memory.

The calculator does this for the whole catalog, and the pages for each memory size say what that size is good for: start with 8 GB or 12 GB and move up from there. If you are still deciding what to buy, the buying tool starts from the model and works back to the cheapest card that runs it properly.