Pro / workstation · GDDR6 ECC
RTX 6000 Ada 48 GB: which LLMs can it run?
On its own, it loads up to Llama 3.3 70B in Q4. With several cards, memory adds up and opens bigger models.
Specifications
Memory48 GB
Bandwidth960 GB/s
FP16 compute~182 TF
Power300 W
Indicative price€7,000
Cloud rental€0.90/h
Against cards at the same price
Models and expected speed
| Model | Q4 | Q8 | FP16 |
|---|---|---|---|
| Qwen3 4B | ✓ ~207 tok/s | ✓ ~120 tok/s | ✓ ~65 tok/s |
| Llama 3.1 8B | ✓ ~107 tok/s | ✓ ~61 tok/s | ✓ ~33 tok/s |
| Qwen3 8B | ✓ ~104 tok/s | ✓ ~60 tok/s | ✓ ~32 tok/s |
| Gemma 3 12B | ✓ ~72 tok/s | ✓ ~41 tok/s | ✓ ~22 tok/s |
| Mistral Nemo 12B | ✓ ~71 tok/s | ✓ ~40 tok/s | ✓ ~22 tok/s |
| Phi-4 14B | ✓ ~59 tok/s | ✓ ~34 tok/s | ✓ ~18 tok/s |
| Qwen3 14B | ✓ ~58 tok/s | ✓ ~33 tok/s | ✓ ~18 tok/s |
| gpt-oss-20b | ✓ ~229 tok/s | ✓ ~133 tok/s | ✓ ~72 tok/s |
| Mistral Small 24B | ✓ ~36 tok/s | ✓ ~21 tok/s | 2 × · ~16 tok/s |
| Gemma 3 27B | ✓ ~32 tok/s | ✓ ~18 tok/s | 2 × · ~15 tok/s |
| Qwen3.8 27B | ✓ ~31 tok/s | ✓ ~18 tok/s | 2 × · ~14 tok/s |
| Qwen3 30B-A3B | ✓ ~248 tok/s | ✓ ~145 tok/s | 2 × · ~117 tok/s |
| Qwen3 32B | ✓ ~27 tok/s | ✓ ~15 tok/s | 2 × · ~12 tok/s |
| Qwen2.5 Coder 32B | ✓ ~27 tok/s | ✓ ~15 tok/s | 2 × · ~12 tok/s |
| DeepSeek R1 Distill 32B | ✓ ~27 tok/s | ✓ ~15 tok/s | 2 × · ~12 tok/s |
| Mixtral 8x7B | ✓ ~67 tok/s | 2 × · ~57 tok/s | 3 × · ~46 tok/s |
| Llama 3.3 70B | ✓ ~12 tok/s | 2 × · ~11 tok/s | 4 × · ~11 tok/s |
| GLM-4.5-Air 106B | 2 × · ~108 tok/s | 3 × · ~92 tok/s | ✗ |
| gpt-oss-120b | 2 × · ~247 tok/s | 3 × · ~214 tok/s | ✗ |
| Mistral Large 123B | 2 × · ~11 tok/s | 4 × · ~12 tok/s | ✗ |
| Qwen3 235B-A22B | 4 × · ~119 tok/s | ✗ | ✗ |
| GLM-4.5 355B | ✗ | ✗ | ✗ |
| Llama 3.1 405B | ✗ | ✗ | ✗ |
| DeepSeek R1 671B | ✗ | ✗ | ✗ |
| DeepSeek V3.1 671B | ✗ | ✗ | ✗ |
| Kimi K2 1T | ✗ | ✗ | ✗ |
8k tokens, one user. “2 ×” = number of cards needed, speed with tensor parallelism (vLLM). ✗ = doesn’t fit, even with 4 cards.