Desktop GPU · NVIDIA

What AI models can a GeForce RTX 4060 run?

8 GB is the awkward size. It runs the 7 to 9 billion parameter class comfortably and hits a wall immediately above it, so most of the decisions here are about context length rather than model choice. Bandwidth is the weak point at 272 GB/s. Models fit, then generate slowly, because every token means reading the whole active model out of memory.

Efficient and quiet, and starved of memory for this job. 8 GB on a 128-bit bus means 272 GB/s, so it loads 7B to 9B models and generates them at a modest pace. It is a good gaming card that is hard to recommend for local AI when 12 GB cards cost about the same used.

Memory8 GB

Bandwidth272 GB/s

Usable for a model7.2 GB

Runtime backendCUDA

ArchitectureAda Lovelace

Power115 W

The short answer

Assuming an 8k context window and default settings, these are the models worth downloading first.

Best all-rounder

Qwen3.5 9B

Q5_K_M · 6.42 GB · about 33.3 tokens/s

Best for code

Qwen2.5 Coder 7B Instruct

Q6_K · 6.23 GB · about 34.3 tokens/s

Largest that still runs well

Mistral Nemo Instruct 2407

12.25B parameters · Q3_K_M · about 30.7 tokens/s

Fastest useful answer

LFM2.5 8B A1B

about 89.8 tokens/s · 6.9 GB

See how fast it feels

GeForce RTX 4060 running Qwen3.5 9B at Q5_K_M

Wait for the first word426 ms

Then writes at33.3 tok/s

Whole answer5.6 s

YouWhy does my model use more memory when the conversation gets longer?

Model

Simulated from our estimate at a 512-token question, not a recording. Assumes nothing else is competing for the GPU.

Where it sits

With 8 GB it belongs to the 8 GB tier. The 8 GB guide explains what that tier runs and where it runs out, and the bandwidth guide explains why 272 GB/s decides how fast it answers.

Our indicative price for this card is $280 new, reviewed 2026-09-01. That is 2.9 GB of memory and 97 GB/s of bandwidth for every $100. Prices move, so treat these as a place to start a search and not a quote.

Devices in a similar price band, fastest memory first:

Device	Price	Memory	Bandwidth	Best all-round pick
GeForce RTX 3080 10GB	$350 used	10 GB	760 GB/s	Qwen3.5 9B, 81.1 t/s
GeForce RTX 2080 Ti	$250 used	11 GB	616 GB/s	Qwen3.5 9B, 51.3 t/s
GeForce RTX 3070 Ti	$300 used	8 GB	608 GB/s	Qwen3.5 9B, 74.4 t/s
Arc A770 16GB	$280 used	16 GB	560 GB/s	Qwen3.5 27B, 23.1 t/s

Can an RTX 4060 run it?

A direct answer for the models people ask about most, each with the settings we would use.

Qwen2.5 7B Instruct Llama 3.1 8B Instruct Qwen3 8B Qwen3.5 9B Gemma 4 12B Instruct Qwen3 14B Phi 4 GPT OSS 20B Qwen3.5 27B Qwen3 30B A3B GPT OSS 120B

Every model, scored on this device

Each row uses the highest-quality quantization that both fits and stays conversational. Speed is a single-stream estimate at 8k context.

Model	Params	Verdict	Download	Memory used	Speed	Max context
Qwen3.5 9B Qwen	9.65B	Just fits	Q5_K_M	6.42 GB	33.3 t/s	32k
Gemma 4 E4B Instruct Gemma	8B	Just fits	Q6_K	6.61 GB	32 t/s	32k
Qwen2.5 VL 7B Instruct Qwen	8.29B	Just fits	Q6_K	6.74 GB	31.5 t/s	16k
Qwen3 8B Qwen	8.19B	Just fits	Q5_K_M	6.54 GB	32.6 t/s	8k
Qwen2.5 7B Instruct Qwen	7.62B	Just fits	Q6_K	6.23 GB	34.3 t/s	16k
Llama 3.1 8B Instruct Llama	8.03B	Just fits	Q6_K	7.14 GB	29.7 t/s	8k
Gemma 4 12B Instruct Gemma	11.96B	Just fits	Q3_K_M	7.16 GB	29.6 t/s	8k
Granite 4.1 8B Granite	8.79B	Just fits	Q4_K_M	6.6 GB	32.3 t/s	8k
Qwen2.5 Coder 7B Instruct Qwen	7.62B	Just fits	Q6_K	6.23 GB	34.3 t/s	16k
Internlm3 8B Instruct InternLM	8.8B	Just fits	Q6_K	7.1 GB	29.9 t/s	8k
Mistral 7B Instruct V0.3 Mistral	7.25B	Just fits	Q6_K	6.84 GB	31.1 t/s	8k
Gemma 4 E2B Instruct Gemma	5.12B	Runs great	Q8_0	5.44 GB	39.1 t/s	128k
Qwen3.5 4B Qwen	4.66B	Runs great	Q8_0	5.27 GB	40.8 t/s	64k
Gemma 3 12B Instruct Gemma	12.19B	Just fits	Q3_K_M	6.76 GB	31.5 t/s	8k
Mistral 7B Instruct V0.2 Mistral	7.24B	Just fits	Q6_K	6.83 GB	31.1 t/s	8k
OLMo 3 7B Instruct OLMo	7.3B	Just fits	Q4_K_M	6.78 GB	31.4 t/s	8k
Gemma 2 9B Instruct Gemma	9.24B	Just fits	IQ4_XS	6.93 GB	30.6 t/s	8k
Qwen3 14B Qwen	14.77B	Fits only at 2 to 3 bits	IQ3_XXS !	6.67 GB	32.1 t/s	8k
Mistral Nemo Instruct 2407 Mistral	12.25B	Just fits	Q3_K_M	6.96 GB	30.7 t/s	8k
Qwen2.5 14B Instruct Qwen	14.77B	Fits only at 2 to 3 bits	IQ3_XXS !	6.92 GB	30.9 t/s	8k
LFM2.5 8B A1B Liquid	8.47B (1.57B active)	Just fits	Q6_K	6.9 GB	89.8 t/s	32k
Qwen2.5 Coder 14B Instruct Qwen	14.77B	Fits only at 2 to 3 bits	IQ3_XXS !	6.92 GB	30.9 t/s	8k
Qwen3 4B Qwen	4.02B	Runs great	Q8_0	5.46 GB	39.2 t/s	16k
Phi 4 Phi	14.66B	Fits only at 2 to 3 bits	IQ3_XXS !	7.04 GB	30.3 t/s	8k
Gemma 3 4B Instruct Gemma	4.3B	Runs great	Q8_0	4.91 GB	44 t/s	64k
Phi 4 Mini Instruct Phi	3.84B	Runs great	Q8_0	5.17 GB	41.7 t/s	16k
Granite 4.1 3B Granite	3.4B	Runs great	Q8_0	4.34 GB	50.2 t/s	32k
Qwen2.5 3B Instruct Qwen	3.09B	Runs great	Q8_0	3.68 GB	59.9 t/s	32k
Phi 3 Mini 4k Instruct Phi	3.82B	Runs great	Q8_0	4.8 GB	45.2 t/s	4k
DeepSeek Coder 7B Instruct V1.5 DeepSeek	6.91B	Just fits	Q3_K_M	7.11 GB	29.8 t/s	4k

Show the other 56 models

Llama 3.2 3B Instruct	3.21B	Runs great	Q8_0	4.42 GB	49.4 t/s	32k
LFM2.5 2.6B	2.7B	Runs great	Q8_0	3.14 GB	71.6 t/s	128k
Qwen3.5 2B	2.27B	Runs great	Q8_0	2.7 GB	84.9 t/s	256k
Gemma 2 2B Instruct	2.61B	Runs great	Q8_0	3.54 GB	62.7 t/s	8k
Starcoder2 3B	3.03B	Runs great	Q8_0	3.34 GB	67.5 t/s	16k
Qwen3 1.7B	2.03B	Runs great	Q8_0	3.22 GB	69.4 t/s	32k
DeepSeek Coder 6.7B Instruct	6.74B	Fits only at 2 to 3 bits	IQ3_XXS	6.76 GB	31.5 t/s	8k
Mistral Small 24B Instruct 2501	23.57B	Fits only at 2 to 3 bits	IQ2_XXS	7.18 GB	29.7 t/s	8k
DeepSeek R1 Distill Qwen 1.5B	1.78B	Runs great	Q8_0	2.07 GB	114.4 t/s	128k
Qwen2.5 1.5B Instruct	1.54B	Runs great	Q8_0	2.06 GB	114.8 t/s	32k
SmolLM2 1.7B	1.71B	Runs great	Q8_0	3.53 GB	62.7 t/s	8k
OLMo 2 0425 1B	1.48B	Runs great	Q8_0	2.6 GB	88.5 t/s	4k
Llama 3.2 1B Instruct	1.24B	Runs great	Q8_0	1.82 GB	135.5 t/s	128k
LFM2.5 1.2B Instruct	1.17B	Runs great	Q8_0	1.46 GB	178.9 t/s	64k
Gemma 3 1B Instruct	1B	Runs great	Q8_0	1.35 GB	192.6 t/s	32k
Qwen3.5 0.8B	0.87B	Runs great	Q8_0	1.28 GB	205.7 t/s	256k
Qwen3 0.6B	0.75B	Runs great	Q8_0	1.92 GB	123.8 t/s	32k
Qwen2.5 0.5B Instruct	0.49B	Runs great	Q8_0	0.88 GB	345.9 t/s	32k
LFM2.5 350M	0.35B	Runs great	Q8_0	0.68 GB	535.5 t/s	64k
DeepSeek Coder V2 Lite Instruct	15.71B	Partial offload	n/a	9.29 GB	52.2 t/s	n/a
Qwen3.5 35B A3B	35.95B	Partial offload	n/a	20.48 GB	42.1 t/s	n/a
GPT OSS 20B	20.91B	Partial offload	n/a	11.55 GB	31.2 t/s	n/a
Qwen3 Coder Next	79.67B	Partial offload	n/a	45.22 GB	28.9 t/s	n/a
Qwen3 Next 80B A3B Instruct	81.32B	Partial offload	n/a	46.15 GB	28.5 t/s	n/a
GLM 4.7 Flash	31.22B	Partial offload	n/a	18.13 GB	27.3 t/s	n/a
Qwen3 30B A3B	30.53B	Partial offload	n/a	18.08 GB	26.3 t/s	n/a
Qwen3 Coder 30B A3B Instruct	30.53B	Partial offload	n/a	18.08 GB	26.3 t/s	n/a
Gemma 4 26B A4B Instruct	25.81B	Partial offload	n/a	15.7 GB	26.1 t/s	n/a
GPT OSS 120B	116.83B	Partial offload	n/a	59.4 GB	15.3 t/s	n/a
Qwen3.5 122B A10B	125.09B	Partial offload	n/a	70.61 GB	11.4 t/s	n/a
Phi 3.5 MoE Instruct	41.87B	Partial offload	n/a	24.87 GB	10.5 t/s	n/a
MiniMax M2.5	228.7B	Partial offload	n/a	130.56 GB	6.2 t/s	n/a
MiniMax M2.7	228.69B	Partial offload	n/a	130.55 GB	6.2 t/s	n/a
GLM 4.5 Air	110.47B	Partial offload	n/a	63.61 GB	6.2 t/s	n/a
Mixtral 8x7B Instruct V0.1	46.7B	Partial offload	n/a	27.59 GB	5.1 t/s	n/a
Codestral 22B V0.1	22.25B	Partial offload	n/a	14.62 GB	3.5 t/s	n/a
Qwen3 235B A22B	235.09B	Partial offload	n/a	133.71 GB	3.2 t/s	n/a
Qwen3.5 27B	27.78B	Partial offload	n/a	15.99 GB	3 t/s	n/a
Gemma 3 27B Instruct	27.43B	Partial offload	n/a	16.96 GB	2.8 t/s	n/a
Granite 4.1 30B	28.87B	Partial offload	n/a	18.64 GB	2.4 t/s	n/a
Qwen3 32B	32.76B	Partial offload	n/a	20.42 GB	2.1 t/s	n/a
Qwen2.5 32B Instruct	32.76B	Partial offload	n/a	20.42 GB	2.1 t/s	n/a
Qwen2.5 Coder 32B Instruct	32.76B	Partial offload	n/a	20.42 GB	2.1 t/s	n/a
MiniMax M3	427.04B	Partial offload	n/a	240.84 GB	2 t/s	n/a
Gemma 4 31B Instruct	31.27B	Partial offload	n/a	21.31 GB	2 t/s	n/a
GLM 5.1	753.86B	Partial offload	n/a	424.51 GB	1.4 t/s	n/a
DeepSeek V3.2	685.4B	Partial offload	n/a	385.91 GB	1.4 t/s	n/a
DeepSeek R1	684.53B	Partial offload	n/a	385.42 GB	1.4 t/s	n/a
GLM 4.5	358.34B	Partial offload	n/a	204.37 GB	1.4 t/s	n/a
GLM 5.2	753.33B	Partial offload	n/a	424.21 GB	1 t/s	n/a
Qwen2.5 VL 72B Instruct	73.41B	Partial offload	n/a	43.62 GB	0.8 t/s	n/a
Qwen2.5 72B Instruct	72.71B	Partial offload	n/a	43.22 GB	0.8 t/s	n/a
Llama 3.3 70B Instruct	70.55B	Partial offload	n/a	42.12 GB	0.8 t/s	n/a
Llama 3 3 Nemotron Super 49B V1	49.87B	Partial offload	n/a	47.99 GB	0.7 t/s	n/a
Kimi K2.6	1026.88B	Will not run	n/a	577.78 GB	1.3 t/s	n/a
Kimi K2 Instruct	1026.41B	Will not run	n/a	577.52 GB	1.3 t/s	n/a

What it will not run

2 of the 86 architectures we track are out of reach here, even at 2-bit precision. Another 35 run by splitting the model between the card and system RAM, which works but drops generation to single digits for most dense models.

If you want more headroom

The next steps up in memory, in order. More memory changes which models load at all; more bandwidth changes how fast they answer, so the two are worth weighing separately.

Device	Memory	Bandwidth	Models that fit
GeForce RTX 3080 10GB	10 GB	760 GB/s	53	Check price
Arc B570	10 GB	380 GB/s	53	Check price
GeForce RTX 2080 Ti	11 GB	616 GB/s	57	Check price
GeForce GTX 1080 Ti	11 GB	484 GB/s	57	Check price

Price links go to an Amazon search for the model name and are affiliate links: if you buy through one, we earn a commission at no cost to you. We do not take payment for placement, and the ordering above is by memory capacity alone.