Ollama, context length and VRAM: why your model is slower than it should be

Ollama is the easiest way to run a model and the one that makes the most decisions on your behalf. It picks a context window, it decides how many layers go on the GPU, and neither choice is announced when you type ollama run. On a small card those two choices can cost you half your speed, and the fix is one setting.

What Ollama decides for you

The context window defaults to a value chosen from your VRAM. Ollama's documentation gives 4k tokens for cards under 24 GiB, 32k from 24 to 48 GiB, and 256k from 48 GiB up. For most readers of this page that means 4,096 tokens, which is short: a long document or a chat that has been going for a while will silently lose its beginning. You can raise it with the OLLAMA_CONTEXT_LENGTH environment variable, with PARAMETER num_ctx in a Modelfile, or per request through num_ctx in the API options.

Then it places layers. Ollama estimates what fits and keeps a margin of free VRAM, and if it decides the model does not fit, it puts some layers on the CPU. You can see what it did while a model is loaded:

ollama ps

The processor column reads 100% GPU when everything is on the card. Anything like 16%/84% CPU/GPU means a share of the model is in system RAM, and every token pays for it.

What it cost us on a 6 GB card

We ran Qwen2.5 7B at Q4_K_M on an RTX 3050 with 6 GB, DDR4-2133 and Ollama 0.35, and let it place layers itself at five window sizes:

Context window	Share on the GPU	Tokens per second
2,048	84%	19.0
4,096	84%	19.4
8,192	79%	16.0
16,384	73%	13.7
32,768	61%	10.1

Two things stand out. Ollama never put the whole model on the card, even at the shortest window where there was room, and the share it kept on the GPU shrank as the window grew, because the cache needs the memory it would have given to layers. Speed fell from 19.0 to 10.1 tokens per second over the sweep, a loss of about 47%.

Then we forced the placement with num_gpu, the number of layers on the GPU, and set it above the layer count so that everything went on the card. At a 4k window the same model ran at 29.5 tokens per second, which is 52% faster than Ollama's own choice, and it held 0.63 GB of VRAM free on top. The auto placement was cautious, not wrong: a margin is what keeps a second application from crashing a model. But if you know the card is yours, the margin is speed you are leaving on the table.

How to force it, and when not to

In a Modelfile:

FROM qwen2.5:7b-instruct-q4_K_M
PARAMETER num_ctx 4096
PARAMETER num_gpu 99

Build it with ollama create and run the new name. num_gpu can also go in the API options, or be set interactively with /set parameter num_gpu 99. A value larger than the layer count means all layers.

Do this only after you have checked that the total fits. For this model the calculator predicts about 4.59 GB at a 4k window, which is inside a 6 GB card with room left. Forcing a model that needs 6.5 GB onto a 6 GB card gives you an out-of-memory error or, worse, a card that spills into shared system memory and crawls. If the margin is under about 0.3 GB, take the smaller quantization or the shorter window instead.

Settings that change the memory bill

OLLAMA_FLASH_ATTENTION=1 turns on flash attention, which keeps scratch memory flat as the window grows. OLLAMA_KV_CACHE_TYPE sets the cache precision to f16, q8_0 or q4_0, and the quantized types need flash attention on. At 8 bits the cache takes about half the memory, which for a model like Llama 3.1 8B at 32k brings 4.00 GB down to about half that. On Windows and macOS you set these in the environment and restart the Ollama app, and on Linux with the systemd service you add them with systemctl edit ollama.service.

OLLAMA_NUM_PARALLEL is the one that surprises people. Ollama's documentation says the memory a model needs scales with the number of parallel requests times the context length, so setting it to 4 for a household server quadruples the cache. If you are the only user, leave it at its default.

When a model feels slow

Work through it in this order, because each step is cheaper than the next. First, run ollama ps while the model is loaded and read the processor column. A split such as 16%/84% CPU/GPU on a model that the calculator says fits is the usual culprit, and setting num_gpu as above fixes it. Second, check the context window: if it is far larger than your conversations need, lower it, because the cache it reserves is memory the layers could have used. Third, check that nothing else is holding VRAM, since a browser with hardware acceleration and a game launcher can each take a gigabyte, and on Windows the desktop itself takes more than most people expect.

If all three are clean and the speed is still below the calculator's estimate, compare it with the bandwidth formula in the bandwidth guide. On our card the estimate and the measurement agreed within 7% once every layer was on the GPU, so a larger gap points at the setup and not at the hardware. The sweep above is the pattern to remember: the same weights, the same card, and a speed that depended entirely on where the layers landed.