llama.cpp flags that decide memory and speed: -ngl, -c, -fa and --n-cpu-moe

llama.cpp has more than a hundred command-line options, and about six of them decide whether a model loads and how quickly it talks. The rest tune things you will not notice. This page is the short list, with what each one costs, and the numbers come from the sweep we ran on an RTX 3050 6 GB, not from theory.

How many layers go on the GPU

-ngl (long form --n-gpu-layers) is the number of layers placed on the GPU. A 7B model has around 28 to 32 layers, and each one you move to the card makes the card hold more weights and the CPU do less. Recent builds accept -ngl auto and -ngl all, and older advice to write -ngl 99 still works because it means "all of them".

The effect on speed is not linear, and this is the thing worth seeing once. We forced Qwen2.5 7B at Q4_K_M onto the RTX 3050 one step at a time through Ollama's num_gpu setting, which is the same lever:

Layers on the GPU	Tokens per second
0 of 28	4.8
12	7.2
23	14.7
28 and the output layer	29.5

The first dozen layers bought 2.5 tokens per second. The last five bought 14.8. Every layer left in RAM puts a slow read in every token, so the final step to a full GPU is worth far more than the first steps, and a model that is a few percent too big for the card is a bad case to be in. If you are close, a smaller quantization or a shorter window that lets everything fit beats a partial split.

Context, cache precision and flash attention

-c sets the context window, in tokens, and llama.cpp reserves the KV cache for all of it when the model loads. This is the cheapest memory you can get back. Dropping from 32k to 8k on a model with a heavy cache returns gigabytes, as the KV cache guide shows. If you do not need a long window, do not ask for one.

-ctk and -ctv set the precision of the cache's keys and values, with f16 as the default. Setting both to q8_0 roughly halves the cache with a small quality cost, and q4_0 quarters it with a larger one. The value cache needs flash attention to be quantized, which brings up the next option.

-fa turns flash attention on, off or to auto (-fa on, -fa off, -fa auto). It avoids building the large table of attention scores, so scratch memory stays flat as the window grows, and on most hardware it is also faster. Current builds default to auto, which enables it where the model and backend support it. Our estimates assume it is on, since that matches what most people run. If you turn it off, expect the compute buffer to grow with the window, noticeably above 16k.

Mixture-of-experts models

A mixture-of-experts model does not have to be split by layer. --cpu-moe keeps all the expert weights in system RAM and --n-cpu-moe N does so for the first N layers, while everything dense stays on the GPU through a large -ngl. The card then reads attention and cache at full speed and only the few experts picked for each token come from RAM. For a model such as gpt-oss-120b that is the difference between not running at all and running at a usable speed on a 12 GB card, provided you have 64 GB of fast system RAM. The MoE guide covers when it pays.

For a finer split there is -ot (--override-tensor), which places tensors by a pattern you write. It is powerful and easy to get wrong, so reach for it only when --n-cpu-moe is not enough.

Extra GPUs, threads and a starting command

With more than one card, -sm picks the split mode (none, layer, row, or tensor in recent builds) and -ts sets the proportion each card takes, for example -ts 3,1 for a 3 to 1 split. Layer split is the safe default and works with mismatched cards. -t sets CPU threads. For the CPU share of a split model it matters less than people expect, because that half of the work waits on RAM bandwidth, not on the cores. Setting it to the number of physical cores is a fine start.

--mlock pins the weights in RAM so the operating system cannot page them out, and --no-mmap loads the whole file instead of mapping it. Both use more memory and neither makes a healthy setup faster. Reach for them only when generation stutters because the model keeps being evicted.

For a 7B to 9B model on an 8 GB card:

llama-server -m model.gguf -ngl all -c 8192 -fa on -ctk q8_0 -ctv q8_0

That puts every layer on the card, keeps a moderate window, and halves the cache. If the load fails, drop -c first, then switch the cache to q4_0, then pick a smaller quantization, in that order, because that is the order of how much quality each one costs you. For a MoE model that does not fit, add --n-cpu-moe and raise it until the load succeeds.

Then compare the result with the calculator: it models the same split, so you can see the speed you should expect before you spend half an hour on a download. On a model that fits the card, the estimate for an 8B at Q4_K_M is about 5.63 GB of memory and 31.4 tokens per second on an RTX 3060.