Which GPU should you buy for local AI?
Memory decides what loads, bandwidth decides how fast it answers, and the two are separate purchases. How to pick, and why the obvious card is often the wrong one.
Published 2026-09-01
Most buying advice for local AI is gaming advice with the word "AI" pasted on. It ranks cards by how fast they render frames, which is very nearly irrelevant here, and it treats memory as a tiebreaker rather than the whole question.
Two numbers decide what a card can do with a language model.
Memory decides what loads
A model either fits or it does not. Below the line it will still run, split between the card and system RAM, at a fraction of the speed. Above it, everything works.
That makes memory capacity a cliff rather than a slope, and it means the sensible way to shop is to pick the model class you want first and buy the smallest card that clears it:
| You want to run | You need | Which means |
|---|---|---|
| 7B to 9B, short context | 8 GB | Almost any current card |
| 12B to 14B | 12 GB | The 3060 12GB floor |
| 20B to 24B | 16 GB | 4060 Ti 16GB, 7800 XT, Arc A770 |
| 30B to 32B | 20 to 24 GB | 3090, 7900 XT, 7900 XTX |
| 70B | 48 GB, or unified memory | Two cards, a workstation part, or a Mac |
The right column is where the surprises are. A used RTX 3090 has as much memory as a new RTX 4090 and costs a third as much. A 4060 Ti 16GB was written off by every gaming reviewer and is one of the better value cards on this list, because 16 GB at 165 W is exactly what this workload wants.
Bandwidth decides how fast
Generating one token means reading every active parameter out of memory. Not computing on them cleverly — reading them, all of them, once per token. That makes generation speed almost purely a function of memory bandwidth.
The consequence is that two cards with the same memory can differ by three times in speed. A 4060 Ti 16GB has 288 GB/s; a 4070 Ti SUPER has 672 GB/s with the same 16 GB. They load the same models. One answers roughly twice as fast.
It also explains the most common disappointment: a card with plenty of memory and slow memory feels sluggish no matter how modern it is. If you are choosing between two cards that both fit your model, take the faster memory. If you are choosing between one that fits and one that is faster but does not, take the one that fits — a slow answer beats an answer that never loads.
The cases worth knowing
Used 24 GB cards are the value play. An RTX 3090 is three generations old, has 24 GB and 936 GB/s, and goes for a fraction of a current 24 GB part. For inference specifically it is close to unbeatable per dollar. It also draws 350 W and needs a power supply that expects it.
Apple Silicon buys capacity, not speed. Unified memory means a laptop can hold a 70B model, which no consumer graphics card can do. But the bandwidth is workstation-class at best, so the experience is "it runs, at ten to twenty tokens a second" rather than "it flies". macOS also only lets the GPU wire down about three quarters of the installed memory by default, so a 48 GB machine really has around 36 GB for a model. That limit is raisable.
Pascal cards are a trap. A GTX 1080 Ti or a Tesla P40 gives you a lot of memory for very little money, and generates tokens at a respectable rate. What they cannot do is process a prompt: those chips have no usable half-precision path, so feeding the model a long document takes minutes rather than seconds. They are fine for short back-and-forth and miserable for anything else.
AMD and Intel work, with effort. ROCm and Vulkan both run language models properly now, and a 7900 XTX gives you 24 GB for well under the price of the NVIDIA equivalent. Expect to spend an evening on drivers, and expect less community help when something breaks. If tinkering is the point, this is good value. If you want it working in twenty minutes, pay the NVIDIA tax.
More cards is a real option, and not a simple one. Two 3090s give you 48 GB and will run a 70B properly. They also need a motherboard with the lanes, a power supply that can take 700 W of GPU, and a runtime configured to split the model. It works well and it is a project.
The question to ask instead
"What is the best GPU for AI" has no answer. "What is the cheapest thing that runs the model I want, at a speed I will tolerate" has a precise one, and it is usually cheaper than expected.
That is what the buying tool does: pick your model, say how well you want it to run and how long a context you need, and it works out the cheapest hardware that gets there — across current cards, used cards and whole machines. If you already own something, the calculator answers the same question from the other direction.