Quantization explained: which GGUF level to download

Quantization stores each weight in fewer bits so a model fits in less memory, and the names on a download page (Q4_K_M, IQ3_XXS, Q8_0) look like noise but are almost systematic. Choosing the right one matters more than most hardware upgrades: dropping from Q8_0 to Q4_K_M frees nearly half the weights' memory for a quality loss you will struggle to measure.

Reading the names

Take Q4_K_M apart. Q4 is the headline bit count. K means a k-quant, which stores weights in small blocks that each carry their own scale, a big accuracy gain for almost no extra size. M is medium, meaning a few important tensors stay at higher precision than they would in the small (S) variant, and L keeps even more. IQ4_XS is an importance-matrix quant: the publisher runs sample text through the model first, finds which weights matter most, and spends the bit budget there. An IQ file is usually a little smaller than a K file of the same bit count and about as accurate. Q8_0 is plain 8-bit with none of that machinery, close enough to the original that the difference does not show up in normal use.

The names also hide that they are not quite what they say. A Q4_K_M file is about 4.83 bits per weight, because block scales and the higher-precision tensors have to live somewhere. That is why a "4-bit 8B model" downloads at about 4.5 GB and not 4. Here is what each level costs for one model.

Quantization	Weights	KV cache	Total	Quality
Q8_0	7.43 GB	1.00 GB	8.83 GB	99%
Q6_K	5.73 GB	1.00 GB	7.14 GB	98%
Q5_K_M	4.97 GB	1.00 GB	6.38 GB	96%
Q4_K_M	4.22 GB	1.00 GB	5.63 GB	93%
IQ4_XS	3.71 GB	1.00 GB	5.12 GB	91%
Q3_K_M	3.42 GB	1.00 GB	4.82 GB	86%
IQ3_XXS	2.67 GB	1.00 GB	4.08 GB	79%

The quality column is our own coarse scale, informed by published perplexity comparisons, not a benchmark we ran. Read it as a ranking.

What each level costs you

Q8_0 and Q6_K are indistinguishable from the original in everything we have seen people do with them, so they are only worth choosing when memory is plentiful. Q5_K_M gives up a little on word choice. Q4_K_M is the standard trade, and the point where most users stop noticing any loss while the memory bill is about a third of the 16-bit original.

Below 4 bits the picture changes. Q3_K_M is where degradation starts to show, with instruction following getting sloppier. IQ3_XXS is clearly worse on anything under about 30B. Q2_K is a curiosity: larger than IQ3_XXS and worse, so it only wins when no IQ3 file exists for the model you want. IQ2_XXS produces fluent, confident, subtly wrong output, which is worse than a smaller model that knows less. When the calculator cannot find anything better it will still list these levels, but it marks them with a warning and never recommends them.

A big model squeezed, or a small one kept intact?

For a long time the community answer was "always take the bigger model", and that holds down to about 4 bits. A 30B at Q4_K_M really does beat a 13B at Q8_0. It stops holding below roughly 3.5 bits, where the model starts losing the ability to follow its own reasoning, and the damage falls hardest on exactly what large models are for: multi-step reasoning, code and recall over long context.

Here is a concrete case. Qwen3 32B at IQ2_XXS needs about 10.1 GB at an 8k window, and Qwen3 14B at Q6_K needs about 12.4 GB. They land within a couple of gigabytes of each other, so they compete for the same card. I would take the 14B every time. It follows instructions, it holds a conversation together, and it is the model you can actually trust the output of. That judgment is baked into the calculator: it will not offer a 2-bit quantization as an answer unless nothing else fits, and it says so when it does.

Quantizing the cache as well

The weights are not the only thing you can compress. The KV cache can be stored at 8 or 4 bits instead of 16, and at long context it is often the better lever. In Ollama it is the OLLAMA_KV_CACHE_TYPE setting, with q8_0 using about half the memory of f16 and q4_0 about a quarter. In llama.cpp it is -ctk and -ctv. Quantizing the V half of the cache needs flash attention switched on, which is the default on the setups that support it.

At 8 bits the quality cost is close to nothing in what we have seen and read, so q8_0 is a safe first move when you are a gigabyte or two short of the context you want. It is usually a better trade than dropping the weights from Q5 to Q4, because the cache is where the memory goes at long context and it tolerates compression well. At 4 bits, recall over very long windows starts to suffer, so keep that for when nothing else works. The reasons the cache gets so large in the first place are in our KV cache guide.

What to download

For most people, most of the time: Q4_K_M. If you have memory to spare, take Q5_K_M or Q6_K, because there is no virtue in leaving VRAM empty. Take Q8_0 only when the model is small enough that it costs you nothing. Go below Q4 when the alternative is a model a whole size class smaller, and stop at Q3_K_M unless you are above 30B. If your runtime is vLLM rather than llama.cpp, the formats change and so does the advice, which the formats guide covers.

To see what your own card allows at each level, put it into the calculator. Every model page lists the memory cost of every quantization side by side, so you can see where the next step down saves you the most.