Memory bandwidth, not TFLOPS, sets your tokens per second

Cards with the same amount of VRAM can differ by a factor of 3 in how fast they answer, and the spec that explains it is rarely printed in large type. Generating one token means reading every active weight out of memory once, so speed is set by how fast the memory can be read, which is its bandwidth.

The formula, and a real measurement

Per generated token the card streams the model's weights, plus the cache so far, from memory to the compute units. That makes decoding almost purely a memory-reading job, and a good estimate is:

tokens per second = bandwidth (GB/s) x efficiency / bytes read per token (GB)

We tested the efficiency term on hardware we own rather than assuming it. An RTX 3050 has 168 GB/s of memory bandwidth. With every layer of Qwen2.5 7B on the card it generated 29.5 tokens per second while reading about 4.5 GB per token, which works out to about 133 GB/s, or 79% of the peak. Run on the CPU alone, the same machine's dual-channel DDR4-2133 (34.1 GB/s on paper) gave 4.76 tokens per second, which is about 21 GB/s, or 63%. Those two figures are the efficiency constants the calculator uses for NVIDIA cards and for system RAM. Other platforms use values we have not been able to measure ourselves, and we say so on the methodology page.

Device	Memory	Bandwidth	Qwen3 8B	Qwen3 14B	Qwen3 32B
GeForce RTX 3050 6GB	6 GB	168 GB/s	does not fit	does not fit	does not fit
GeForce RTX 3060 12GB	12 GB	360 GB/s	49.2 t/s	29.1 t/s	does not fit
GeForce RTX 4060	8 GB	272 GB/s	37.2 t/s	does not fit	does not fit
GeForce RTX 4060 Ti 16GB	16 GB	288 GB/s	39.4 t/s	23.2 t/s	does not fit
GeForce RTX 4070	12 GB	504 GB/s	68.9 t/s	40.7 t/s	does not fit
GeForce RTX 3090	24 GB	936 GB/s	128 t/s	75.5 t/s	34.5 t/s
GeForce RTX 4090	24 GB	1008 GB/s	138 t/s	81.3 t/s	37.1 t/s
GeForce RTX 5090	32 GB	1792 GB/s	245 t/s	145 t/s	66.0 t/s
Apple M4 Max 64GB	64 GB	546 GB/s	69.0 t/s	40.7 t/s	18.6 t/s

What the formula explains

It explains why a 12 GB RTX 3060 beats a 16 GB RTX 4060 Ti on any model that fits both. The 3060 has 360 GB/s and the 4060 Ti has 288, so on Qwen3 8B the older card answers at about 30.8 tokens per second and the newer one at about 24.6. The 16 GB card earns its place by loading models the 12 GB one cannot, not by running the shared ones faster. Memory capacity decides what loads, and bandwidth decides how quickly it talks.

It explains Apple Silicon too. A MacBook with an M4 Max has 546 GB/s and can hold a 70B model that no single consumer GPU can, but a used RTX 3090 has 936 GB/s, so a model that fits both answers faster on the old graphics card. And it explains why quantization speeds things up as well as shrinking them: a Q4 file is about 3.3 times fewer bytes to read than 16-bit, so a bandwidth-bound model runs roughly that much faster.

Mixture-of-experts models break the rule in a useful way. They read only the experts the router picks, so a 30B model with 3B active reads about a tenth of the weights per token. It does not run 10 times faster, because gathering small expert matrices from all over memory wastes some of the bandwidth, but it generates several times faster than a dense 30B on the same card, as long as every expert fits in memory. The MoE guide goes into what that costs.

Where compute matters after all

Reading your prompt is a different job. Prompt processing, called prefill, handles many tokens at once, which keeps the compute units busy and makes the job compute-bound, not bandwidth-bound. On our RTX 3050 test it ran at 928.2 tokens per second, roughly 50 times faster than generation, so a 4,000-token prompt waits about 4 to 5 seconds before the first word appears.

That is why 2 cards with similar bandwidth can feel different. One starts answering almost at once and the other makes you wait, and Apple chips in particular tend to have strong bandwidth but weaker prefill than NVIDIA cards of the same price. Image and video generation sit on the compute side too, which is why the media pages use a separate engine. Batched serving, with many users at once, drifts compute-bound as well, because each weight read is shared across many tokens.

Reading a spec sheet

Look for the bandwidth figure in GB/s. If the sheet only gives a memory speed and a bus width, multiply them: a 128-bit bus is 16 bytes per transfer, so 18 Gbps memory gives 288 GB/s. A narrow bus is the usual reason a card with plenty of memory feels slow, and it is the number to check before you buy anything for local models.

Then do the sum yourself. Take the file size of the model you want, divide your bandwidth by it, and multiply by 0.75 to 0.8. If the answer is under about 10 tokens per second, chat will feel slow however much memory the card has. If it is over 30, you are past the point where the reader, not the GPU, is the bottleneck. The calculator does this for every model and device pair, and flags where the whole model fits in the card's own memory so the number means what you expect.