Run a 120B model on one GPU: expert offload with llama.cpp, step by step

A mixture-of-experts model has the memory bill of a very large model and the compute of a small one, and that mismatch is the opportunity. Most of its weight sits in experts that are used a few at a time, so they can live in ordinary system RAM while the part used on every token stays on the GPU. This is the way to run a model like gpt-oss-120b on a single consumer card, and it works better than most people expect.

What you need

A GPU with 12 GB or more, 64 GB of system RAM, and about 60 GB of free disk space. The model is gpt-oss-120b in its native MXFP4 format, a single file of about 63 GB that you can download from Hugging Face in the ggml-org/gpt-oss-120b-GGUF repository, and a recent build of llama.cpp. The MoE guide explains why the arrangement works. Here we only cover how to set it up.

The command

Start with the experts all on the CPU and everything else on the GPU:

llama-server -m gpt-oss-120b-MXFP4.gguf -ngl 99 --n-cpu-moe 99 -c 16384 -fa on --jinja

-ngl 99 puts every layer on the GPU, which for this purpose means the attention and the shared parts. --n-cpu-moe 99 then moves the experts of that many layers back to system RAM, and since the model has only 36 layers, 99 means all of them. -c 16384 sets the window, -fa on turns on flash attention, and --jinja applies the model's chat template. This loads on almost any card, because only the dense part and the cache are in VRAM, and it is the slowest arrangement.

Finding the right number

Now spend your spare VRAM. Lower --n-cpu-moe and reload. Every unit you remove puts one more layer's experts on the GPU, which means fewer bytes read from RAM for every token, and the speed rises. Keep going until the load fails with an out-of-memory error, then go back up by one. That value is the best for your card at that window. A longer window leaves less room, so if you raise -c later, raise the number a little.

Our engine estimates the result, so you know where to aim. These are the values for gpt-oss-120b at an 8k window with 64 GB of DDR4-3200 behind each card:

Card	Experts kept on the CPU	Share on the GPU	Estimated speed
GeForce RTX 3060 12GB	first 30 of 36 layers	17%	16.9 tokens/s
GeForce RTX 4070 Ti SUPER	first 27 of 36 layers	24%	19.5 tokens/s
GeForce RTX 3090	first 22 of 36 layers	38%	23.7 tokens/s
GeForce RTX 5090	first 17 of 36 layers	51%	31.2 tokens/s

Treat the speeds as estimates good to about 30%. We have not measured a mixture-of-experts model ourselves, and the factor behind these figures comes from published llama.cpp results, which the methodology page states plainly. The shape of the table is the reliable part: a larger card keeps more of the experts and answers faster, and even the smallest card makes the model usable.

RAM speed is the lever

The experts left in RAM are read from it for every token, so generation speed follows system memory bandwidth almost directly. On the same RTX 3090 and the same model, the estimate is 23.7 tokens per second with DDR4-3200 and 38.8 with DDR5-5600. Two things to check on the machine you have: that the memory runs at its rated speed, since many boards use a slow default until you enable XMP or EXPO, and that both channels are populated. The RAM and CPU guide goes through both.

When it goes wrong

If the load fails immediately, you did not leave enough room for the cache, so raise --n-cpu-moe or lower -c. If it loads and generates slowly, check that the RAM is the speed you think it is, and that -t is set to your physical core count, not your thread count. If the whole machine becomes sluggish, the model plus your other programs are using all of RAM and the operating system is paging, so close things or add memory. --mlock can keep the model resident, and it will not help if there is no room for it.

Recent builds of llama.cpp include an automatic fitting option, --fit, that tries to place layers and experts for you. It is a good place to start, and the manual route above is what to reach for when you want to control the result or it picks a poor split.

Everything here applies to smaller mixture-of-experts models too. Qwen3 30B A3B has 128 experts with 8 active, and on a 12 GB card with 32 GB of RAM it runs with a good share of its experts on the GPU at a speed that few dense models of its quality can match. The flags guide lists the options used here and the rest of the ones that matter.