How to run a local LLM on Windows, Mac or Linux: a first-time walkthrough

Running a language model on your own computer takes less than most people expect, and the first time is mostly about not downloading the wrong thing. This walkthrough gets you from nothing to a working chat, and spends its effort on the three decisions that matter: which model fits, how to tell it is using your GPU, and what to change when it is slow.

Step 1: Find out how much memory you have

The number that decides everything is the memory on your graphics card. On Windows, open Task Manager, go to the Performance tab and select your GPU, and read "Dedicated GPU memory". On a machine with an NVIDIA card, the command nvidia-smi shows the same figure in its first table. On a Mac, the number that counts is the total unified memory, under "About This Mac", and roughly two thirds to three quarters of it is available to the GPU. On a PC with no dedicated card, it is your system RAM that decides, and the models you can run are smaller and slower.

Then put that number into the calculator and see what fits. As a rule of thumb, an 8B model at 4 bits needs about 5.63 GB with an 8k window, a 14B needs 10 GB or so, and a 32B needs 20 GB or more. If you are unsure which model to try first, a recent 7B to 9B model is a safe choice on almost any machine.

Step 2: Install a runtime

A runtime is the program that loads the model and runs it. There are two good beginner options, and they run the same model files. Ollama is a small program with a command-line interface and a background server. Download the installer for your system from its website, or on Linux run the one-line install script it publishes. LM Studio is a desktop application with a model browser and a chat window, and nothing to type. If you like a graphical interface, start there.

Both detect NVIDIA, AMD and Apple hardware without configuration. Support for a brand-new card sometimes lags, so if the program says it is using the CPU on a card that should work, check the release notes of the runtime and update your graphics driver first.

Step 3: Run a model that fits

With Ollama, open a terminal and run a model by name. The first run downloads it, and later runs start in seconds:

ollama run qwen3:8b

That downloads about 5 GB and opens a chat. Type a question, and press Ctrl+D or type /bye to leave. Other commands worth knowing are ollama list, which shows what you have downloaded, and ollama rm followed by a name, which deletes one. In LM Studio, search for a model in its browser, pick a file whose size is comfortably below your memory, and press download.

The file you pick has a quantization in its name, such as Q4_K_M. For a first try, take Q4_K_M: it is the default for good reason, as our quantization guide explains. If the model is too big for your card, choose a smaller one before you choose a lower quantization.

Step 4: Check it is using your GPU

This is the step most people skip and the one that explains most complaints about speed. While the model is loaded, run ollama ps in a second terminal and read the processor column. It should say 100% GPU. A split such as 30%/70% CPU/GPU means part of the model is running from system RAM, which is several times slower, and it happens when the model plus its context window do not quite fit. In LM Studio, the same information is in the model's loading panel and in its GPU offload slider.

On our own test card, a model that fit entirely sat at only 84% on the GPU until we forced it, and generation ran 52% faster once everything was there. The Ollama guide shows the setting. If you see a split on a model that should fit, lower the context window first.

Step 5: Fix the three problems everyone hits

The first is slowness, and it is almost always the split from Step 4 or a model that is simply too big. The second is the conversation forgetting its beginning, which happens because the default window is short, 4k tokens on cards under 24 GiB in Ollama. Raise it with the OLLAMA_CONTEXT_LENGTH environment variable, bearing in mind that a longer window costs memory. A 32k window on an 8B model adds about 4.00 GB of cache.

The third is the out-of-memory error at load or in the middle of a long prompt. Close other programs that use the GPU, since a browser with hardware acceleration can hold a gigabyte. Then shorten the window, then choose a smaller quantization, then a smaller model, in that order. Each step costs less quality than the one after it.

Once a chat works, the next questions are usually which model is best for your job and what to buy if nothing runs well. The VRAM guide explains the arithmetic behind the calculator, and the GPU buying guide covers hardware. If you are curious about speed, the bandwidth guide shows why the same model runs at very different speeds on cards with the same memory.