System RAM and CPU for local LLMs: what matters when there is no big GPU

When a model does not fit on a graphics card, or you have no card at all, the CPU and system memory do the work. That path is slow compared with a GPU, and it is worth knowing exactly how slow, because the parts that decide it are not the ones the spec sheet puts first.

The number that matters is memory bandwidth

Generating a token on a CPU works like it does on a GPU. Every weight is read once, so speed follows how fast memory can be read, and the cores spend most of their time waiting. System memory bandwidth is the transfer rate multiplied by 8 bytes and by the number of channels. DDR4-3200 in dual channel is 3,200 x 8 x 2, which is 51.2 GB/s.

We measured it on a machine we own, which has dual-channel DDR4-2133 and a Xeon E3-1245 v5. Running Qwen2.5 7B at Q4_K_M on the CPU alone it generated 4.8 tokens per second. The memory's paper bandwidth is 34.1 GB/s, and the model reads about 4.5 GB per token, so the machine achieved about 63% of the theoretical figure. That 0.63 is the CPU efficiency constant the calculator uses.

Applying the same 63% to an 8B model at 4 bits, which is a file of about 4.9 GB, gives a table you can check against your own hardware:

Memory	Bandwidth	Estimated tokens per second
DDR4-2133, dual channel	34 GB/s	4.3
DDR4-3200, dual channel	51 GB/s	6.5
DDR5-5600, dual channel	90 GB/s	11.4
DDR5-6400, dual channel	102 GB/s	13.0
DDR5, quad channel at 5600	179 GB/s	22.8
12 channels of DDR5-4800 (server)	461 GB/s	58.7

Everything in that table sits at or below what a mid-range graphics card does, and the server row shows why people build large EPYC machines for the biggest models: it is the only way to get GPU-class bandwidth with hundreds of gigabytes of capacity.

Three things that matter more than the CPU

Enable the memory profile. Most boards run DIMMs at a conservative default until you turn on XMP or EXPO in the BIOS, and the difference between 2133 and 3200 MT/s is a 50% jump in generation speed, for free. Our own test machine was stuck at the default, so the figure above is a floor for that CPU.

Use both channels. A single stick, or two sticks in the wrong slots, runs single-channel at half the bandwidth. Check that the pair sits in the slots the manual names. A fourth DIMM often pulls the speed down, since four sticks per channel pair stress the controller, so two larger sticks beat four smaller ones.

Mind the capacity. 16 GB runs a 7B or 8B model and little else. 32 GB is the comfortable level for partial offload, where the operating system, the model and the cache all need room. 64 GB lets you keep the experts of a large mixture-of-experts model in RAM while a card handles everything else, which is how a 120B model becomes usable on a 12 GB card, as the MoE guide explains.

What the CPU itself changes

Core count matters less than people think for generation. Six to eight real cores saturate the memory bus on a desktop, and extra cores sit idle. Where the CPU does count is prompt processing, which is compute-bound and benefits from more cores and from wide vector units, AVX2 at least and AVX-512 where available. If you push long documents through a CPU-only setup, a chip with more cores gets you to the first token sooner.

Hyper-threading rarely helps. Setting -t to the number of physical cores, not logical ones, is a sound default, and the flags guide covers the rest.

Prompt processing is where a CPU hurts most, and we measured it. On our Xeon the CPU-only run read prompts at 19.4 tokens per second, so a 4,000-token prompt, roughly a long email, waits about 3.4 minutes before the first word appears. The same card-assisted setup, with the whole model on the GPU, reads that prompt in about 4 seconds. If you paste long documents into a local model, the GPU's contribution is almost entirely here, and it is the reason a small card transforms a machine that felt unusable on the CPU alone.

Three example machines

An older office PC with dual-channel DDR4-2666 reaches about 5.5 tokens per second on an 8B model at 4 bits, which reads as a slow typist. A current desktop with dual-channel DDR5-6000 reaches about 12.2, which is comfortable for a private assistant. A mini PC built on AMD's Ryzen AI Max+ 395 has 256 GB/s of LPDDR5X and gets about 32.6 on the same model by the same arithmetic, though its integrated GPU does the work in practice and the figure is a ceiling. Each of them runs a 14B model at roughly half those speeds, and a 30B dense model at a quarter.

None of those numbers need a new processor to improve. Faster memory in the same machine is the cheapest upgrade, and a mixture-of-experts model is the cheapest way to turn the memory you have into speed.

When CPU-only makes sense

For a 7B or 8B model at 4 bits, DDR5 gets you to a readable pace, around 10 tokens per second and above. That is fine for a private assistant that you query occasionally, or for batch jobs that run overnight. It is not a good place to run a 30B or larger dense model, where the same arithmetic gives a few tokens per second. For anything bigger, a GPU, or a mixture-of-experts model that reads only a fraction of its weights, is the way out. The calculator uses the speed of your system memory for whatever part of a model spills over from the card, so set it to the RAM you actually have.