30 guides
Local AI guides: the parts worth understanding
The calculator gives you a number. These explain where it comes from, so you can judge it, and so you know which setting to change when it is not the number you wanted. Where we ran something ourselves we say so, and where we are estimating we say that too.
Newest and most recently updated
How to run a local LLM on Windows, Mac or Linux: a first-time walkthrough
From checking your GPU memory to a working chat in about 15 minutes: install Ollama, pick a model that fits, confirm it uses your GPU, and fix the usual problems.
How much VRAM do you need for a local LLM?
An 8B model at 4-bit needs about 6 GB, not 8. The formula behind that number, and why it moves with the model and the context window.
llama.cpp flags that decide memory and speed: -ngl, -c, -fa and --n-cpu-moe
The handful of llama.cpp options that move VRAM and tokens per second, what each one trades, and a sensible starting command for a small GPU.
The basics
What decides whether a model fits, and what decides how fast it runs.
How much VRAM do you need for a local LLM?
An 8B model at 4-bit needs about 6 GB, not 8. The formula behind that number, and why it moves with the model and the context window.
Quantization explained: which GGUF level to download
What Q4_K_M and IQ3_XXS actually mean, what each level costs in quality, and how to choose between a big model squeezed small and a small one kept intact.
KV cache explained: why a long context window eats your VRAM
The KV cache is the memory nobody budgets for. What it holds, why it grows with every token, and the four designs that keep it small.
Memory bandwidth, not TFLOPS, sets your tokens per second
Cards with the same VRAM can answer at very different speeds. The formula that explains it, checked against a real RTX 3050 run.
Mixture-of-experts models: big memory bill, small speed bill
A 30B MoE model needs the memory of a 30B and generates like a 3B. Why, when that is a bargain, and how to run a 120B on a mid-range GPU.
GGUF, AWQ, GPTQ, EXL2 and MLX: which model format to download
Model files come in half a dozen formats and they are not interchangeable. What each one is for, which runtime reads it, and which to pick.
Local LLM glossary: VRAM, KV cache, quantization and the rest
Plain definitions of the terms you hit when you run models yourself, from VRAM and tokens per second to prefill, offload and mixture of experts.
Choosing hardware
What to buy, what to avoid, and what a used card really costs you.
Best GPU for local LLMs by budget: what to buy at each price
Which graphics card to buy for running models at home, ranked by memory and bandwidth rather than by gaming scores, with the strongest model each one runs.
Buying a used GPU for local AI: what to check before you pay
A used RTX 3090 is the best memory per dollar for local models, and also where buyers get burned. Which cards make sense, which are traps, how to test one.
Mac or PC for local LLMs: unified memory against a graphics card
A Mac can hold a 70B model no single GPU can, and a GPU answers faster on anything that fits. Where each wins, with speeds from our calculator.
Running a local LLM on multiple GPUs: what you gain and what you do not
A second graphics card adds memory, not speed. How llama.cpp splits a model across cards, when two cheap cards beat one expensive one, and when they do not.
System RAM and CPU for local LLMs: what matters when there is no big GPU
Memory speed and channel count set how fast a CPU generates tokens, and the core count barely does. Real numbers from a DDR4 machine and a table for faster RAM.
What it costs to run a local LLM: electricity, heat and noise
A 350 W GPU costs pennies an hour to run a model, and the bill that surprises people is the idle one. A small calculator and the numbers behind it.
Running models
The settings in Ollama and llama.cpp that move memory and speed the most.
How to run a local LLM on Windows, Mac or Linux: a first-time walkthrough
From checking your GPU memory to a working chat in about 15 minutes: install Ollama, pick a model that fits, confirm it uses your GPU, and fix the usual problems.
llama.cpp flags that decide memory and speed: -ngl, -c, -fa and --n-cpu-moe
The handful of llama.cpp options that move VRAM and tokens per second, what each one trades, and a sensible starting command for a small GPU.
Run a 120B model on one GPU: expert offload with llama.cpp, step by step
A 120B mixture-of-experts model does not need 60 GB of VRAM. Keep its experts in system RAM, find your --n-cpu-moe value, and see what speed to expect.
Ollama, context length and VRAM: why your model is slower than it should be
Ollama quietly picks a context window and a GPU split for you. What it chooses, how to check it with ollama ps, and what forcing it did on our RTX 3050.
Best local LLM for coding: what fits your GPU, and how much context you need
A coding assistant needs a long context window and fast prompt reading more than size. The strongest code model for each amount of VRAM, and what a repo costs.
By amount of VRAM
What actually runs on a card with a given amount of memory.
What can you run with 6 GB of VRAM? Tested on an RTX 3050
A 6 GB card runs 7B to 9B models with the right quantization and a short window. What fits, what to avoid, and how it held up on a real RTX 3050.
What can you run with 8 GB of VRAM? Models, quantizations and limits
An 8 GB card is the most common setup and the first to get cramped. Which 7B to 9B models run well, where long context breaks, and why 8 GB cards differ 3x.
What can you run with 12 GB of VRAM? 14B models, 20B experts and long context
12 GB is the first tier where local models feel roomy: 14B models at Q5, a 20B mixture-of-experts at the edge, and a used card that doubles the speed.
What can you run with 16 GB of VRAM? 27B models, big windows and the bandwidth trap
16 GB loads 24B to 27B models at 4 bits and 14B models at Q8 with a long window. Which pairing to pick, and why one 16 GB card can be 3 times faster than another.
What can you run with 24 GB of VRAM? 30B models, long context and the 70B question
24 GB is the tier where local models stop being a compromise. 27B to 32B models at Q4 to Q6, a fast MoE at long context, and what a 70B really costs on one card.
What can you run with 32 GB of VRAM? The RTX 5090 tier
32 GB holds a 30B model at Q6 and a 27B at Q8 with a long window. Where it falls short of 70B, and what a Mac with 32 GB really offers.
What can you run with 48 GB of VRAM? 70B models, one card or two
48 GB holds a 70B model at 4 bits with a usable window. One workstation card or two used 3090s, what context costs, and which large MoE models fit.
Image, video and audio
Why diffusion and video models behave nothing like language models.
Image generation VRAM, explained: why Flux needs more than SDXL
A diffusion model is a pipeline of parts, and its memory is not the size of one file. Where the gigabytes go in SDXL and Flux, and the 3 settings that decide.
Running Flux and SDXL on a low-VRAM card: 6, 8 and 12 GB
SDXL runs on 6 GB and Flux on 12 GB with the right precision. The diffusers, ComfyUI and A1111 settings that make each fit, and what each one costs you.
Video generation on your own GPU: VRAM, render time and what is realistic
Video models make one very long sequence, and attention cost explodes with it. Why a 5 second clip is hard, which models are practical, and how long they take.
Local text-to-speech and transcription: what hardware you need
Speech models are small, and what matters is seconds of audio per second of compute. Kokoro, XTTS, F5-TTS and Whisper compared across GPUs.
Measured by us
Real runs on real hardware, set against our own estimates.