Mixture-of-experts models: big memory bill, small speed bill

Some of the most popular local models have two sizes in their name. Qwen3 30B-A3B is a 30B model that generates like a 3B one, and gpt-oss-120b generates like a model under 6B. They are mixture-of-experts models, and the trade they make is strange at first: you pay the memory of the big number and get the speed of the small one.

Where the speed comes from

A dense model uses every parameter for every token. A mixture-of-experts model splits most of its parameters into many small experts and adds a router that picks a few of them for each token. Qwen3 30B-A3B has 128 experts and uses 8 per token. gpt-oss-120b has 128 and uses 4. Attention layers, shared experts and the embedding are always active, and everything else sits idle until the router calls it.

Generation speed follows the bytes read per token, as the bandwidth guide shows, and a MoE model reads only the active part. That is why Qwen3 30B-A3B on an RTX 4090 runs at about 146 tokens per second while Qwen3 32B, a dense model of nearly the same size, runs at about 37.1. The memory bill does not move, because every expert must be loaded before the router can choose from it.

Model	Weights	Cache at 8k	Total at 8k	Cache at 32k	Total at 32k
Qwen3 30B A3B 30.53B, 3.34B active	17.2 GB	0.75 GB	18.1 GB	3.00 GB	20.3 GB
Qwen3 32B 32.76B	18.4 GB	2.00 GB	20.4 GB	8.00 GB	26.4 GB
GPT OSS 20B 20.91B, 4.18B active	11.3 GB	0.19 GB	11.5 GB	0.75 GB	12.1 GB
GPT OSS 120B 116.83B, 5.7B active	59.0 GB	0.29 GB	59.4 GB	1.13 GB	60.2 GB

When it is a bargain

It is a good deal whenever the whole model fits in fast memory. A 16 GB card holds all of gpt-oss-20b, a 21B model with about 4B active, and generates at about 51.5 tokens per second, which no dense 21B could approach on that card. Its published file is 11.3 GiB, so on a 12 GB card there is no room left for the cache and a few layers of experts stay in system RAM, and the estimate is still about 65.7. Quality is not free either, and MoE models of a given total size tend to land between a dense model of their active size and one of their total size. For a chat model, that middle ground at that speed is attractive.

It is a poor deal when memory is the thing you lack. Nothing about a MoE design shrinks the weights, so on an 8 GB card a 30B MoE is as impossible as a 30B dense, and the usual levers apply: a lower quantization, or offload.

Running a MoE model that does not fit

Offload works very differently here. For a dense model, moving layers to the CPU makes every token touch slow memory for the layers you moved. For a MoE model there is a better split, and llama.cpp exposes it: keep the experts in system RAM and everything dense on the GPU, with --cpu-moe for all of them or --n-cpu-moe N for the first N layers. The GPU then reads the attention weights and the cache at full speed, and only the handful of experts chosen for each token come from RAM.

Our engine models that split, using the same arithmetic we validated on a real offload sweep, and it changes the answer a lot. With 64 GB of DDR4-3200, it estimates gpt-oss-120b at about 16.9 tokens per second on a 12 GB RTX 3060 when it keeps the experts in RAM, where moving whole layers would give about 11. We have not measured a MoE model ourselves, because the machine we test on has 32 GB of slow DDR4-2133, so treat the figure as an estimate good to about 30% and expect system RAM speed to dominate it. A model like this wants 64 GB of RAM and fast memory, since the RAM is doing most of the reading.

Device	Memory	Bandwidth	GPT OSS 20B	Qwen3 30B A3B
GeForce RTX 3060 12GB	12 GB	360 GB/s	does not fit	does not fit
GeForce RTX 4070	12 GB	504 GB/s	does not fit	does not fit
GeForce RTX 3090	24 GB	936 GB/s	168 t/s	153 t/s
GeForce RTX 4090	24 GB	1008 GB/s	181 t/s	165 t/s

Choosing between dense and MoE

If it all fits in VRAM, take the MoE model for speed and judge it by its quality on your task. If it does not fit and you have 64 GB or more of system RAM, a MoE model with the experts offloaded often runs better than a dense model of any size that spills over the same way. If memory is tight and RAM is small, a smaller dense model at a decent quantization is the safer choice. Context behaves normally in all cases: the KV cache of a MoE model follows its attention design, and the hybrid and sliding-window designs in the KV cache guide apply here too. At 128k, gpt-oss-120b holds 4.50 GB of cache because half its layers use a small window.

The calculator marks which MoE results come from keeping experts in RAM, and the model pages show both numbers so you can see what the technique is worth for the model you care about.