Pro / workstation · GDDR7 ECC

RTX PRO 6000 Blackwell 96 GB: which LLMs can it run?

On its own, it loads up to Mistral Large 123B in Q4. With several cards, memory adds up and opens bigger models.

Memory96 GB

Bandwidth1,792 GB/s

FP16 compute~250 TF

Power600 W

Indicative price€9,000

Cloud rental€1.40/h

Model	Q4	Q8	FP16
Qwen3 4B	✓ ~387 tok/s	✓ ~225 tok/s	✓ ~121 tok/s
Llama 3.1 8B	✓ ~199 tok/s	✓ ~114 tok/s	✓ ~61 tok/s
Qwen3 8B	✓ ~194 tok/s	✓ ~111 tok/s	✓ ~60 tok/s
Gemma 3 12B	✓ ~134 tok/s	✓ ~77 tok/s	✓ ~41 tok/s
Mistral Nemo 12B	✓ ~132 tok/s	✓ ~75 tok/s	✓ ~40 tok/s
Phi-4 14B	✓ ~110 tok/s	✓ ~63 tok/s	✓ ~33 tok/s
Qwen3 14B	✓ ~109 tok/s	✓ ~62 tok/s	✓ ~33 tok/s
gpt-oss-20b	✓ ~427 tok/s	✓ ~249 tok/s	✓ ~134 tok/s
Mistral Small 24B	✓ ~68 tok/s	✓ ~39 tok/s	✓ ~20 tok/s
Gemma 3 27B	✓ ~60 tok/s	✓ ~34 tok/s	✓ ~18 tok/s
Qwen3.8 27B	✓ ~58 tok/s	✓ ~33 tok/s	✓ ~18 tok/s
Qwen3 30B-A3B	✓ ~463 tok/s	✓ ~270 tok/s	✓ ~146 tok/s
Qwen3 32B	✓ ~50 tok/s	✓ ~28 tok/s	✓ ~15 tok/s
Qwen2.5 Coder 32B	✓ ~50 tok/s	✓ ~28 tok/s	✓ ~15 tok/s
DeepSeek R1 Distill 32B	✓ ~50 tok/s	✓ ~28 tok/s	✓ ~15 tok/s
Mixtral 8x7B	✓ ~125 tok/s	✓ ~71 tok/s	2 × · ~57 tok/s
Llama 3.3 70B	✓ ~23 tok/s	✓ ~13 tok/s	2 × · ~10 tok/s
GLM-4.5-Air 106B	✓ ~134 tok/s	2 × · ~115 tok/s	3 × · ~92 tok/s
gpt-oss-120b	✓ ~307 tok/s	2 × · ~266 tok/s	3 × · ~214 tok/s
Mistral Large 123B	✓ ~13 tok/s	2 × · ~11 tok/s	3 × · ~9.0 tok/s
Qwen3 235B-A22B	2 × · ~111 tok/s	3 × · ~94 tok/s	✗
GLM-4.5 355B	3 × · ~115 tok/s	✗	✗
Llama 3.1 405B	3 × · ~9.1 tok/s	✗	✗
DeepSeek R1 671B	✗	✗	✗
DeepSeek V3.1 671B	✗	✗	✗
Kimi K2 1T	✗	✗	✗

8k tokens, one user. “2 ×” = number of cards needed, speed with tensor parallelism (vLLM). ✗ = doesn’t fit, even with 4 cards.