Quantisation explained: which one to download

A model is published as a pile of numbers, sixteen bits each. Quantisation replaces those with shorter numbers — four bits, three, sometimes two — so the pile fits in less memory. Everything interesting about it is in what gets lost.

Reading the names

The filenames look cryptic and are almost entirely systematic.

Q4_K_M breaks into three parts. Q4 is four bits per weight. K means a k-quant, which stores weights in small blocks with their own scaling factor rather than one scale for the whole tensor — a large accuracy win for almost no size. M is medium, meaning some tensors are kept at higher precision than others; S for small keeps fewer, L for large keeps more.

IQ4_XS is an importance-matrix quant. The publisher runs sample text through the model first, works out which weights matter most, and spends the precision budget on those. An IQ quant is typically a little smaller than a K quant of the same bit count and about as accurate.

Q8_0 is eight bits with no k-quant machinery, and is close enough to the original that the difference does not show up in normal use.

The one thing the names hide is that the numbers lie slightly. A Q4_K_M file is not four bits per weight, it is about 4.83, because the block scales and the higher-precision tensors have to live somewhere. That is why a "4-bit 8B model" downloads at 4.8 GB and not 4.0 GB — and why the tables on this site use measured bits per weight rather than the name.

What each level costs

Roughly, and in the order you should prefer them:

Level	Bits per weight	What you lose
Q8_0	8.5	Nothing you can measure in normal use
Q6_K	6.56	Nothing you will notice
Q5_K_M	5.69	Occasional worse word choice
Q4_K_M	4.83	The standard trade. Slight loss on hard reasoning
IQ4_XS	4.25	Same as Q4_K_S, smaller file
Q3_K_M	3.91	Noticeable. Instruction-following gets sloppier
IQ3_XXS	3.06	Clearly degraded below about 30B
Q2_K	3.35	Damaged. Only defensible on very large models
IQ2_XXS	2.06	Expect broken output on anything under 30B

Q2_K sits below IQ3_XXS in that list despite being the larger file, which is not a typo: it is both bigger and worse, and only wins when nobody has published IQ3 weights for the model you want.

The question that actually matters

Given a fixed amount of memory, is it better to run a large model squeezed hard, or a smaller model kept intact?

For a long time the community answer was "always take the bigger model". That is true down to about four bits and stops being true below it. A 30B at Q4_K_M genuinely beats a 13B at Q8_0. A 30B at IQ2_XXS does not beat a 13B at Q6_K — it produces confident, fluent, subtly wrong output, which is worse than a smaller model that knows less.

The dividing line sits around three and a half bits. Above it you are trading a little precision for a lot of capability. Below it the model starts losing the ability to follow the thread of its own reasoning, and the damage falls hardest on exactly the tasks people run large models for: multi-step reasoning, code, and long-context recall.

This is why the recommendations here refuse to hand you a two-bit quantisation as an answer. If a model only fits at IQ2, the tables mark it with a warning and suggest something smaller instead. It is not that the file will not load. It is that we would be recommending a worse experience than the alternative sitting one row down.

The cache is quantisable too

Less discussed, and often the better lever. The KV cache can be stored at eight or four bits instead of sixteen, halving or quartering it, and at eight bits the quality cost is close to nothing.

If you are three gigabytes short of a long context window, quantising the cache to Q8 is almost always a better trade than dropping the weights from Q5 to Q4. The cache is where the memory goes at long context, and it is the part that tolerates compression best.

What to download

For most people, most of the time: Q4_K_M. It is the default for a reason.

Take Q5_K_M or Q6_K if you have the memory spare — there is no reason to leave VRAM unused. Take Q8_0 only if the model is small enough that it costs you nothing. Go below Q4 only when the alternative is a model a whole size class smaller, and stop at Q3_K_M unless you are working with something above 30B.

To see what your own card allows at each level, put it into the calculator — every model page lists the memory cost of every quantisation side by side.