Local LLM glossary: VRAM, KV cache, quantization and the rest

You do not need most of these words to use a local model, but you will meet all of them within a week of trying, usually on a forum thread where everyone assumes you know them. We wrote this page for ourselves first. The definitions are short and each one points to the guide that goes further.

Memory terms

VRAM is the memory on a graphics card, where a model's weights and cache have to sit for the GPU to use them quickly. Unified memory is the Apple Silicon and some AMD arrangement in which the CPU and GPU share a single pool, so a model can use most of the machine's RAM, though the system keeps a share for itself.

Weights are the learned numbers that make up the model, and parameters is the count of them, which is where "8B" and "70B" come from. Quantization stores each weight in fewer bits, and bits per weight is the resulting average, about 4.83 for Q4_K_M. GGUF is the file format llama.cpp, Ollama and LM Studio read. The quantization guide explains the names.

The context window is how many tokens the model can hold in view at once, and the KV cache is the memory that holds the keys and values for all of them. It grows with the window and is the part most people forget to budget. Grouped-query attention (GQA) shrinks it by sharing keys and values across heads, sliding-window attention caps it for most layers, and linear attention replaces it with a small fixed state in some layers. All of that is in the KV cache guide. Flash attention is a method that avoids building the full attention table and keeps scratch memory flat as the window grows. Ollama and llama.cpp turn it on by default where supported.

Speed terms

A token is a chunk of text, often a short word or a piece of a longer one. Tokens per second measures generation speed, and decode or generation is the phase where the model writes its answer one token at a time. It is limited by memory bandwidth, the rate at which the memory can be read, which is why GB/s matters more than TFLOPS for this job. See the bandwidth guide.

Prefill is the phase where the model reads your prompt, and time to first token is how long you wait for the answer to start. Prefill is compute-bound, so it follows a different spec from decode. Offload means running part of the model somewhere slower than the GPU, usually system RAM. A layer split moves whole layers, and for mixture-of-experts models you can instead keep the experts in RAM and the dense part on the GPU. Offload is not all or nothing, and the cost is small at first and steep at the end.

Batch size is how many requests a runtime handles at once. It does little for a single user and a great deal for a server. Speculative decoding uses a small draft model to guess several tokens that the large model then checks together, which can speed up generation when memory bandwidth is the limit.

Model terms

A dense model uses all its parameters for every token. A mixture-of-experts (MoE) model splits most of its parameters into many experts and activates a few per token, so its active parameters are far fewer than its total. Qwen3 30B-A3B has 30B parameters and 3B active. The MoE guide explains the memory and speed consequences.

An instruct or chat model has been trained to follow instructions, and a base model has not. A distill is a smaller model trained to imitate a larger one. Perplexity is a measure of how well a model predicts text, used to compare quantizations, where lower is better. An imatrix is the importance data used to build IQ quantizations. A LoRA is a small add-on that changes a model's behavior without replacing its weights, and it adds a little memory on top of the model.

Runtime and tooling terms

llama.cpp is the C++ inference engine behind most local tools, and the format it reads is GGUF. Ollama wraps it in a command-line tool and a local server with a library of ready-made models. LM Studio is a desktop application built on the same engine. vLLM is a serving framework for GPUs that handles many users at once, and MLX is Apple's array framework for its own chips. A backend is the layer that talks to the hardware: CUDA for NVIDIA, ROCm for AMD, Metal for Apple, Vulkan and SYCL for the rest.

Temperature controls how random the output is, with 0 giving the same answer every time and higher values giving more variety. Top-p and top-k limit the choices considered for each token. A system prompt is the standing instruction given to the model before your message. Tool calling lets a model ask a program to do something, such as run a search, and read the result. RAG, retrieval-augmented generation, feeds the model passages from your own documents so it can answer from them, and it needs an embedding model, a small model that turns text into vectors for searching. A fine-tune is a model trained further on specific data, and it shares its base model's architecture, so it needs the same memory.

Tensor parallelism splits each layer across several GPUs so they work together on every token, and it is what serving frameworks use. Layer split divides the layers between cards so they take turns, which is what llama.cpp does by default and why a second card adds memory and not speed. NVLink is a fast link between certain NVIDIA cards, useful for tensor parallelism and unnecessary for a layer split. mmap loads a model file by mapping it into memory, so the operating system decides what stays resident, and it is why a large model can seem to load instantly and then run slowly until its pages arrive.

If a term you met is missing from this page, tell us and we will add it.