H100 SXM 80GB: What Local LLMs It Can Run

The short answer

Assuming an 8k context window and default settings, these are the models worth downloading first.

Best all-rounder

GPT OSS 120B

MXFP4 · 59.4 GB · about 453.6 tokens/s

Best for code

Llama 3.3 70B Instruct

Q8_0 · 71.81 GB · about 34.6 tokens/s

Largest that still runs well

Qwen3.5 122B A10B

125.09B parameters · Q4_K_M · about 292 tokens/s

Fastest useful answer

LFM2.5 8B A1B

about 867.8 tokens/s · 8.81 GB

See how fast it feels

H100 SXM 80GB running GPT OSS 120B at MXFP4

Wait for the first word16 ms

Then writes at453.6 tok/s

Whole answer397 ms

YouWhy does my model use more memory when the conversation gets longer?

Model

Simulated from our estimate at a 512-token question, not a recording. Assumes nothing else is competing for the GPU.

Where it sits

With 80 GB it belongs to the 48 GB tier. The 48 GB guide explains what that tier runs and where it runs out, and the bandwidth guide explains why 3350 GB/s decides how fast it answers.

We do not track a price for this device, so it does not appear in the budget tools.

Every model, scored on this device

Each row uses the highest-quality quantization that both fits and stays conversational. Speed is a single-stream estimate at 8k context.

Model	Params	Verdict	Download	Memory used	Speed	Max context
GPT OSS 120B GPT-OSS	116.83B (5.7B active)	Runs great	MXFP4	59.4 GB	453.6 t/s	128k
Llama 3.3 70B Instruct Llama	70.55B	Runs great	Q8_0	71.81 GB	34.6 t/s	16k
Qwen2.5 VL 72B Instruct Qwen	73.41B	Runs great	Q8_0	74.45 GB	33.3 t/s	16k
Qwen2.5 72B Instruct Qwen	72.71B	Runs great	Q8_0	73.76 GB	33.7 t/s	16k
Gemma 4 31B Instruct Gemma	31.27B	Runs great	Q8_0	34.67 GB	72 t/s	128k
Qwen3.5 122B A10B Qwen	125.09B (8.17B active)	Runs great	Q4_K_M	70.61 GB	292 t/s	256k
Qwen3 32B Qwen	32.76B	Runs great	Q8_0	34.09 GB	73.3 t/s	32k
Qwen3.5 27B Qwen	27.78B	Runs great	Q8_0	27.32 GB	91.7 t/s	256k
Llama 3 3 Nemotron Super 49B V1 Llama	49.87B	Runs great	Q8_0	68.85 GB	36.1 t/s	8k
GLM 4.5 Air GLM	110.47B (13.4B active)	Runs great	Q5_K_M	74.61 GB	139.3 t/s	32k
Qwen3 Coder Next Qwen	79.67B (3.19B active)	Just fits	Q8_0	79.13 GB	420 t/s	8k
Granite 4.1 30B Granite	28.87B	Runs great	Q8_0	30.97 GB	80.6 t/s	128k
Qwen2.5 32B Instruct Qwen	32.76B	Runs great	Q8_0	34.09 GB	73.3 t/s	32k
Qwen2.5 Coder 32B Instruct Qwen	32.76B	Runs great	Q8_0	34.09 GB	73.3 t/s	32k
Qwen3 Next 80B A3B Instruct Qwen	81.32B (3.19B active)	Runs great	Q6_K	62.46 GB	532.2 t/s	256k
Gemma 3 27B Instruct Gemma	27.43B	Runs great	Q8_0	28.68 GB	87.3 t/s	128k
Qwen3.5 35B A3B Qwen	35.95B (2.9B active)	Runs great	Q8_0	35.63 GB	469.4 t/s	256k
Gemma 4 26B A4B Instruct Gemma	25.81B (3.74B active)	Runs great	Q8_0	26.72 GB	316.2 t/s	256k
GLM 4.7 Flash GLM	31.22B (3.66B active)	Runs great	Q8_0	31.33 GB	357.6 t/s	128k
Mixtral 8x7B Instruct V0.1 Mistral	46.7B (12.88B active)	Runs great	Q8_0	47.49 GB	104.3 t/s	32k
Qwen3 30B A3B Qwen	30.53B (3.34B active)	Runs great	Q8_0	30.99 GB	355.5 t/s	32k
Mistral Small 24B Instruct 2501 Mistral	23.57B	Runs great	Q8_0	24.35 GB	103.1 t/s	32k
Phi 3.5 MoE Instruct Phi	41.87B (6.64B active)	Runs great	Q8_0	42.71 GB	189.3 t/s	128k
Gemma 4 12B Instruct Gemma	11.96B	Runs great	Q8_0	13.55 GB	187.5 t/s	256k
Qwen3 14B Qwen	14.77B	Runs great	Q8_0	15.54 GB	163.3 t/s	32k
GPT OSS 20B GPT-OSS	20.91B (4.18B active)	Runs great	MXFP4	11.55 GB	598.6 t/s	128k
Qwen3 Coder 30B A3B Instruct Qwen	30.53B (3.34B active)	Runs great	Q8_0	30.99 GB	355.5 t/s	32k
Qwen2.5 14B Instruct Qwen	14.77B	Runs great	Q8_0	15.79 GB	160.6 t/s	32k
Qwen2.5 Coder 14B Instruct Qwen	14.77B	Runs great	Q8_0	15.79 GB	160.6 t/s	32k
Codestral 22B V0.1 Mistral	22.25B	Runs great	Q8_0	24.04 GB	104.6 t/s	32k

Show the other 56 models

Qwen3.5 9B	9.65B	Runs great	Q8_0	9.25 GB	278.8 t/s	256k
Phi 4	14.66B	Runs great	Q8_0	16 GB	158.4 t/s	16k
Gemma 3 12B Instruct	12.19B	Runs great	Q8_0	13.27 GB	191.4 t/s	128k
Gemma 4 E4B Instruct	8B	Runs great	Q8_0	8.42 GB	305.8 t/s	128k
Granite 4.1 8B	8.79B	Runs great	Q8_0	10.35 GB	247.8 t/s	128k
Qwen3 8B	8.19B	Runs great	Q8_0	9.02 GB	286.2 t/s	32k
Mistral Nemo Instruct 2407	12.25B	Runs great	Q8_0	13.15 GB	194 t/s	128k
Qwen2.5 VL 7B Instruct	8.29B	Runs great	Q8_0	8.49 GB	304.2 t/s	64k
Qwen2.5 7B Instruct	7.62B	Runs great	Q8_0	7.83 GB	331.4 t/s	32k
Llama 3.1 8B Instruct	8.03B	Runs great	Q8_0	8.83 GB	292.5 t/s	128k
DeepSeek Coder V2 Lite Instruct	15.71B	Runs great	Q8_0	15.91 GB	490.8 t/s	128k
Gemma 2 9B Instruct	9.24B	Runs great	Q8_0	11.5 GB	221.8 t/s	8k
Qwen2.5 Coder 7B Instruct	7.62B	Runs great	Q8_0	7.83 GB	331.4 t/s	32k
OLMo 3 7B Instruct	7.3B	Runs great	Q8_0	9.72 GB	264.5 t/s	64k
Internlm3 8B Instruct	8.8B	Runs great	Q8_0	8.97 GB	287.9 t/s	32k
Mistral 7B Instruct V0.3	7.25B	Runs great	Q8_0	8.45 GB	306.5 t/s	32k
Gemma 4 E2B Instruct	5.12B	Runs great	Q8_0	5.44 GB	481.5 t/s	128k
Qwen3.5 4B	4.66B	Runs great	Q8_0	5.27 GB	502.1 t/s	256k
Mistral 7B Instruct V0.2	7.24B	Runs great	Q8_0	8.44 GB	306.8 t/s	32k
DeepSeek Coder 7B Instruct V1.5	6.91B	Runs great	Q8_0	10.58 GB	242.3 t/s	4k
DeepSeek Coder 6.7B Instruct	6.74B	Runs great	Q8_0	10.94 GB	233.9 t/s	16k
LFM2.5 8B A1B	8.47B	Runs great	Q8_0	8.81 GB	867.8 t/s	64k
Qwen3 4B	4.02B	Runs great	Q8_0	5.46 GB	483 t/s	32k
Gemma 3 4B Instruct	4.3B	Runs great	Q8_0	4.91 GB	541.5 t/s	128k
Phi 4 Mini Instruct	3.84B	Runs great	Q8_0	5.17 GB	513.5 t/s	128k
Granite 4.1 3B	3.4B	Runs great	Q8_0	4.34 GB	617.8 t/s	128k
Qwen2.5 3B Instruct	3.09B	Runs great	Q8_0	3.68 GB	738.2 t/s	32k
Phi 3 Mini 4k Instruct	3.82B	Runs great	Q8_0	4.8 GB	556.1 t/s	4k
Llama 3.2 3B Instruct	3.21B	Runs great	Q8_0	4.42 GB	608.4 t/s	128k
LFM2.5 2.6B	2.7B	Runs great	Q8_0	3.14 GB	881.3 t/s	128k
Qwen3.5 2B	2.27B	Runs great	Q8_0	2.7 GB	1045.2 t/s	256k
Gemma 2 2B Instruct	2.61B	Runs great	Q8_0	3.54 GB	772.2 t/s	8k
Starcoder2 3B	3.03B	Runs great	Q8_0	3.34 GB	831 t/s	16k
Qwen3 1.7B	2.03B	Runs great	Q8_0	3.22 GB	854.7 t/s	32k
DeepSeek R1 Distill Qwen 1.5B	1.78B	Runs great	Q8_0	2.07 GB	1409.1 t/s	128k
Qwen2.5 1.5B Instruct	1.54B	Runs great	Q8_0	2.06 GB	1414.4 t/s	32k
SmolLM2 1.7B	1.71B	Runs great	Q8_0	3.53 GB	772.1 t/s	8k
OLMo 2 0425 1B	1.48B	Runs great	Q8_0	2.6 GB	1090 t/s	4k
Llama 3.2 1B Instruct	1.24B	Runs great	Q8_0	1.82 GB	1668.7 t/s	128k
LFM2.5 1.2B Instruct	1.17B	Runs great	Q8_0	1.46 GB	2203.2 t/s	64k
Gemma 3 1B Instruct	1B	Runs great	Q8_0	1.35 GB	2372.6 t/s	32k
Qwen3.5 0.8B	0.87B	Runs great	Q8_0	1.28 GB	2533.6 t/s	256k
Qwen3 0.6B	0.75B	Runs great	Q8_0	1.92 GB	1524.1 t/s	32k
Qwen2.5 0.5B Instruct	0.49B	Runs great	Q8_0	0.88 GB	4259.7 t/s	32k
LFM2.5 350M	0.35B	Runs great	Q8_0	0.68 GB	6595.9 t/s	64k
MiniMax M2.5	228.7B	Partial offload	n/a	130.56 GB	17.3 t/s	n/a
MiniMax M2.7	228.69B	Partial offload	n/a	130.55 GB	17.3 t/s	n/a
Qwen3 235B A22B	235.09B	Partial offload	n/a	117.88 GB	11.4 t/s	n/a
MiniMax M3	427.04B	Partial offload	n/a	240.84 GB	5.7 t/s	n/a
GLM 4.5	358.34B	Partial offload	n/a	204.37 GB	4.6 t/s	n/a
DeepSeek V3.2	685.4B	Partial offload	n/a	385.91 GB	3 t/s	n/a
DeepSeek R1	684.53B	Partial offload	n/a	385.42 GB	3 t/s	n/a
Kimi K2.6	1026.88B	Partial offload	n/a	577.78 GB	2.8 t/s	n/a
Kimi K2 Instruct	1026.41B	Partial offload	n/a	577.52 GB	2.8 t/s	n/a
GLM 5.1	753.86B	Partial offload	n/a	424.51 GB	2.7 t/s	n/a
GLM 5.2	753.33B	Partial offload	n/a	424.21 GB	2.7 t/s	n/a

What it will not run

0 of the 86 architectures we track are out of reach here, even at 2-bit precision. Another 11 run by splitting the model between the card and system RAM, which works but drops generation to single digits for most dense models.

If you want more headroom

The next steps up in memory, in order. More memory changes which models load at all; more bandwidth changes how fast they answer, so the two are worth weighing separately.

Device	Memory	Bandwidth	Models that fit
RTX PRO 6000 Blackwell	96 GB	1792 GB/s	76	Check price
Ryzen AI Max+ 395 96GB	96 GB	256 GB/s	69	Check price
NVIDIA DGX Spark 128GB	128 GB	273 GB/s	75	Check price
Ryzen AI Max+ 395 128GB	128 GB	256 GB/s	75	Check price

Price links go to an Amazon search for the model name and are affiliate links: if you buy through one, we earn a commission at no cost to you. We do not take payment for placement, and the ordering above is by memory capacity alone.

Questions people ask about an H100 SXM 80GB

How much VRAM does an H100 SXM 80GB have?

The H100 SXM 80GB has 80 GB of VRAM, of which roughly 79.2 GB is available to a model once the operating system takes its share.

Can an H100 SXM 80GB run a 70B model?

Yes. At Q8_0 a 70B model fits in 71.81 GB and generates around 34.6 tokens per second.

What is the best local model for an H100 SXM 80GB?

GPT OSS 120B at MXFP4 is the strongest all-round fit: 59.4 GB used, around 453.6 tokens per second at 8k context.