GGUF, AWQ, GPTQ, EXL2 and MLX: which model format to download

A model repository on Hugging Face often lists the same weights five ways, and picking the wrong one means a file your runtime cannot open or one that cannot run on your hardware. The formats are not better or worse than each other. Each is built for a runtime and a kind of machine, and the quickest way to choose is to start from the runtime you use.

The formats and what reads them

Format	Read by	Runs on	Typical use
GGUF	llama.cpp, Ollama, LM Studio	CPU, GPU, or both	Local use, and anything that spills into system RAM
Safetensors (BF16 or FP16)	transformers, vLLM, diffusers	GPU	The original weights, and the base for every conversion
AWQ	vLLM, transformers	GPU only	4-bit serving on a GPU
GPTQ	vLLM, transformers	GPU only	4-bit serving, older and widely available
EXL2 and EXL3	ExLlama	NVIDIA GPU only	Fast single-user inference at a chosen bit rate
MLX	mlx-lm	Apple Silicon	Native speed on a Mac
FP8, NVFP4, MXFP4	vLLM, recent runtimes	GPUs with hardware support	Native low-precision weights

GGUF is the one most readers need. It is a single file that carries the weights, the tokenizer and the metadata, and the llama.cpp family that reads it can split a model between the GPU and system RAM, which no GPU-only format can do. It also has the widest range of quantization levels, from Q8_0 down to the 2-bit IQ variants, which the quantization guide covers. If you run Ollama or LM Studio, you are already using GGUF.

When a GPU-only format makes sense

AWQ and GPTQ are 4-bit weight-only methods built for GPU kernels, and their main home is vLLM, where many users share one server and the whole model is in VRAM. They land near 4.15 bits per weight against Q4_K_M's 4.83, so the files are a little smaller and fast when the model fits. If you serve a model to several people from one card, this is the family to look at. If you are alone at the keyboard and your model does not quite fit, none of them help, because they have no way to put the overflow in RAM.

EXL2 and its newer sibling EXL3 are ExLlama's formats. They let the publisher pick an exact average bit rate, such as 4.65 bits per weight, so you can fill a card precisely, and they are fast for one user on an NVIDIA GPU. They are less common than GGUF and tied to one runtime.

Apple Silicon and the native low-precision formats

On a Mac, MLX is Apple's own array framework and the models converted for it, usually under mlx-community, run natively on the GPU and the shared memory. It is usually competitive with llama.cpp on Apple Silicon and sometimes faster, and it brings a separate set of 4-bit and 8-bit files. llama.cpp with GGUF also runs well there, and Ollama uses it, so you are not forced to choose. Our Mac and PC comparison goes through the trade-offs.

FP8, NVFP4 and MXFP4 are different: they are floating-point formats with hardware support in recent GPUs, not integer quantization after the fact. FP8 runs natively on Ada and newer. MXFP4 is a 4-bit float with a shared scale per block, and some current models ship their expert weights in it, which is why gpt-oss-120b fits in about 60 GB rather than the 230 GB its parameter count suggests at 16 bits. NVFP4 is NVIDIA's variant, supported on Blackwell. If your card and runtime support them, they give the best quality for the bits. If not, the files do nothing for you.

Reading a repository

A Hugging Face page for a popular model often lists a dozen GGUF files, and the names carry the information. In Qwen3-8B-Q4_K_M.gguf the first part is the model and the last is the quantization, covered in our quantization guide. A name with imatrix or an IQ prefix was built with an importance matrix, which keeps the weights that matter most at higher precision and usually gives better quality at the same size. A name ending in -00001-of-00003.gguf is one piece of a split file, and you need all of the pieces in the same folder, so download them together. Some publishers add their own tags for variants tuned for a specific use, and the model card says what they mean.

If the model you want has no GGUF, you can make one. llama.cpp ships a script, convert_hf_to_gguf.py, that turns the original safetensors into a 16-bit GGUF, and a tool called llama-quantize that shrinks it to the level you choose. It takes a few minutes for a small model and needs enough memory to hold the full-precision weights while it works, which is why most people download a ready-made file instead. We have not found a reason to prefer a home-made quantization over a well-known publisher's, except when the model is too new to have one.

How to choose

Pick the runtime first. Ollama, llama.cpp or LM Studio means GGUF. vLLM on a GPU means safetensors with AWQ, GPTQ or FP8. An Apple laptop means GGUF through Ollama, or MLX if you want to stay native. Our calculator assumes GGUF-style bit rates because that is what most people on this site run, and it lists a 4-bit GPU format separately where it matters. An 8B model at Q4_K_M needs about 5.63 GB at an 8k window, and an AWQ file of the same model is within a few percent of that.

When two formats both work for you, the smaller file at the same quality is the better download. When one needs the CPU and the other cannot use it, the choice has already been made for you.