We tested local LLMs on an RTX 3050 6 GB: what the measurements showed

Most advice about small GPUs comes from people who do not own one. We own this one. The RTX 3050 6 GB is the machine behind every measurement on this site, and running a model on it taught us five things that a spreadsheet would not have, some of which changed how the calculator works.

How we tested, and what the desktop takes

The machine is an ordinary desktop: an RTX 3050 6 GB, an Intel Xeon E3-1245 v5, 32 GB of DDR4-2133 in dual channel and Windows 10 with the desktop running, which is how most people really use a card. The runtime is Ollama 0.35.0 and the model is Qwen2.5 7B Instruct at Q4_K_M, the most downloaded 7B model in our catalog. Every point is the median of 3 warm runs of 96 generated tokens at temperature 0, after one discarded run that loads the model. We ran two sweeps: the context window from 2k to 32k with Ollama placing the layers, and the number of layers on the GPU from none to all at a fixed 4k window.

Before we loaded anything, Windows was already using 1.13 GB of the card's 6 GB for the desktop and a few open programs. Our first version of the calculator assumed 0.8 GB, which is the figure usually quoted. The real one on this machine was larger, and it is the reason a "6 GB card" is a 5 GB card for the purposes of fitting a model. With every layer of Qwen2.5 7B on the GPU, the driver still showed 0.63 GB free, so the model fit with room to spare, though less than the specification suggests.

If your processor has integrated graphics and you plug the monitor into the motherboard instead of the card, the card's memory is free for the model. That is the cheapest 1 GB upgrade there is.

Ollama chose to leave layers on the CPU

We let Ollama place the model by itself and watched where it went. At a 2k window it put 84% of the model on the GPU and the rest in system RAM, though all of it would have fit. As we raised the window, the share fell, to 61% at 32k, and the speed fell with it, from 19.0 to 10.1 tokens per second.

Then we forced every layer onto the card with the num_gpu setting. Generation went to 29.5 tokens per second, which is 52% faster than Ollama's own choice at the same 4k window. Ollama's caution is reasonable: it keeps a margin so that another program cannot crash the model. But on a small card the margin costs a third of your speed, and it is worth knowing how to turn it off. The Ollama guide shows the setting.

The speed cliff is at the end

The most useful chart we made is how speed changes as layers move to the GPU. With none on the card the model ran at 4.8 tokens per second, entirely from DDR4-2133. With 12 of 28 layers moved, that rose to 7.2. With 23, it was 14.7. And with all of them, 29.5.

Read that again: the first 12 layers bought 2.5 tokens per second, and the last 5 bought 14.8. Every token has to wait for the slowest part, so a model that is a little too big for the card runs at the speed of the part that is in RAM, and most of the benefit arrives only when nothing is. For a 6 GB card that means one thing: pick a model and a quantization that fit entirely, and treat partial offload as a last resort, not a plan.

Prompt processing, heat and power

Generation is limited by memory bandwidth, and reading your prompt is not. With the whole model on the GPU the card processed prompts at about 1127 tokens per second, roughly 40 times its writing speed. A 4,000-token prompt waits about 3.5 seconds before the first word appears, and the same prompt on the CPU alone would take far longer. The CPU-only run managed 19 tokens per second of prompt processing.

That is why a small GPU feels fine for chat and why running the CPU alone feels so much worse than the generation numbers suggest. If you are doing retrieval work or pasting long documents, the GPU's contribution is mostly here.

At full load the card drew 69 W against a 70 W rating and sat at 63 degrees Celsius with its fans at a normal speed. Nothing dramatic, and nothing that needed attention. When the model was partly on the CPU the card idled most of the time, and with 17 layers or fewer on the card the draw fell to between 9 and 25 W. The running cost guide puts those figures into dollars.

What we would tell someone with this card

Treat it as a 7B machine and stop looking for tricks. A 7B or 8B model at Q4_K_M with every layer on the GPU and a window of 4k to 8k is the sweet spot. Force num_gpu if your runtime leaves layers on the CPU, move your monitor to the motherboard if you can, and keep the window short. For anything larger, the right answer is another card, not more settings. The 6 GB tier guide lists what fits, and the calculator lets you check a specific model before you download it.

We will add other machines to the validation page as we get them. If you have measured something on a different card, send it to us.