What can you run with 24 GB of VRAM? 30B models, long context and the 70B question

24 GB is the amount most people mean when they say a card is good for local AI. It holds the 27B to 32B class of dense models at a quality that is hard to tell from the full-precision version in daily use, it leaves a real context window, and it runs the fast mixture-of-experts models with room to spare. It is also where the next question starts, which is whether the 70B models are within reach. They are, with caveats.

What fits

Usable memory is about 23.2 GB. That takes a 27B model at Q6, a 32B at Q4_K_M, or a 24B at Q8, and still leaves room for a window of 16k or more. The strongest pairings the engine finds on an RTX 3090 are below.

Pick	Model	Download	Memory	Speed
Best all-rounder	Qwen3.5 27B	Q6_K	21.3 GB	33 t/s
Best for code	Qwen3 32B	Q4_K_M	20.4 GB	34.5 t/s
Best for reasoning	Qwen3.5 27B	Q6_K	21.3 GB	33 t/s
Largest that still runs well	Mixtral 8x7B Instruct V0.1	Q3_K_M	22.6 GB	58.3 t/s

Model	Parameters	Download	Memory	Speed	Longest context
Qwen3.5 27B	27.78B	Q6_K	21.3 GB	33 t/s	32k
Gemma 4 31B Instruct	31.27B	Q4_K_M	21.3 GB	33 t/s	8k
Qwen3 32B	32.76B	Q4_K_M	20.4 GB	34.5 t/s	16k
Granite 4.1 30B	28.87B	Q5_K_M	21.5 GB	32.6 t/s	8k
Gemma 3 27B Instruct	27.43B	Q6_K	22.5 GB	31.3 t/s	16k
Qwen2.5 32B Instruct	32.76B	Q4_K_M	20.4 GB	34.5 t/s	16k
Gemma 4 26B A4B Instruct	25.81B (3.74B active)	Q6_K	20.9 GB	108.6 t/s	32k
Qwen2.5 Coder 32B Instruct	32.76B	Q4_K_M	20.4 GB	34.5 t/s	16k
GLM 4.7 Flash	31.22B (3.66B active)	Q5_K_M	21.2 GB	142 t/s	32k
Gemma 4 12B Instruct	11.96B	Q8_0	13.6 GB	52.4 t/s	64k
Mistral Small 24B Instruct 2501	23.57B	Q6_K	19.2 GB	36.8 t/s	32k
Qwen3 14B	14.77B	Q8_0	15.5 GB	45.6 t/s	32k

Choosing between a 27B at Q6 and a 32B at Q4 is a matter of taste and workload. The 32B has more parameters and a lower-quality quantization, and the 27B is smaller and closer to lossless. For code and reasoning we lean to the 32B. For chat and writing, the difference is small enough that the 27B's extra headroom for context decides it.

The mixture-of-experts models change the experience more than any of the dense ones. GLM 4.7 Flash generates at about 142 tokens per second on this card, and gpt-oss-20b fits whole with a long window and answers at about 167. Against roughly 34.5 for a dense 32B, that is a change you feel in every response. We would keep one of each installed: a dense model for the hard questions and a MoE for everything else.

Context at this size

At 24 GB the cache starts to matter, because the weights already take most of the card. Here is a 32B model at Q4_K_M as the window grows:

Context	KV cache	Total memory
4k	1.00 GB	19.4 GB
8k	2.00 GB	20.4 GB
16k	4.00 GB	22.4 GB
32k	8.00 GB	26.4 GB

On a 24 GB card the 32B at Q4_K_M reaches about 16k before it runs out, and a 27B at Q6 reaches about 32k, since its attention design keeps the cache small. If you need very long windows, the choice of model decides it. A hybrid-attention model gets you to 64k or more where a classic 32B stops at 16k.

The 70B question

A 70B model at Q4_K_M needs about 42.1 GB, which is nearly twice what the card holds. With the rest in system RAM, our engine estimates about 1.5 tokens per second on an RTX 3090 with 64 GB of RAM. That is a number you can use for an occasional hard question and not for a conversation. At 2 bits it fits, and the quality loss at that precision on a model this size is something we would not recommend.

The realistic ways to run a 70B are two 24 GB cards, which is 48 GB and holds it at 4 bits, or a Mac with 64 GB or more of unified memory. Our multi-GPU guide covers the first, and the Mac and PC comparison the second. For a single 24 GB card, the better question is whether a very good 32B does the job, and for most work it does.

Which cards have 24 GB

Device	Memory	Bandwidth	Price
GeForce RTX 4090 Desktop GPU	24 GB	1008 GB/s	$1,900
GeForce RTX 3090 Ti Desktop GPU, used	24 GB	1008 GB/s	$800
Radeon RX 7900 XTX Desktop GPU	24 GB	960 GB/s	$850
GeForce RTX 3090 Desktop GPU, used	24 GB	936 GB/s	$700
GeForce RTX 5090 Laptop Laptop GPU	24 GB	896 GB/s	n/a
RTX A5000 Workstation GPU, used	24 GB	768 GB/s	$1,200
Apple M4 Pro 24GB Unified memory	24 GB	273 GB/s	$1,399
Apple M5 24GB Unified memory	24 GB	153 GB/s	$1,799
Ryzen AI 9 HX 370 24GB Unified memory	24 GB	128 GB/s	n/a
Apple M4 24GB Unified memory	24 GB	120 GB/s	$799

The RTX 3090 and 3090 Ti, the RTX 4090, the RX 7900 XTX and the workstation RTX A5000 all have 24 GB. Compared on the same models:

Device	Memory	Bandwidth	Qwen3 32B	Qwen3.5 27B	GPT OSS 20B
GeForce RTX 3090	24 GB	936 GB/s	34.5 t/s	44.3 t/s	168 t/s
GeForce RTX 3090 Ti	24 GB	1008 GB/s	37.1 t/s	47.7 t/s	181 t/s
GeForce RTX 4090	24 GB	1008 GB/s	37.1 t/s	47.7 t/s	181 t/s
Radeon RX 7900 XTX	24 GB	960 GB/s	30.0 t/s	38.5 t/s	146 t/s
RTX A5000	24 GB	768 GB/s	28.3 t/s	36.3 t/s	138 t/s

The 4090's lead over a 3090 on generation is under 10%, because memory bandwidth went from 936 to 1008 GB/s and bandwidth is what sets the speed. The 4090 is far faster at reading a long prompt and at image generation, where compute matters. For chat with a local model, a used 3090 at a third of the price is the sensible purchase, as our used GPU guide explains.

Our advice at this tier

Install a dense model in the 27B to 32B range and a fast MoE, set the window to what you use, and leave a gigabyte free. If you want the 70B class, plan for 48 GB and not for squeezing it onto one card. And if you are still deciding whether to buy, 24 GB is the point where the extra money stops buying capability you will miss and starts buying capability you will not.