Consumer · GDDR6X
RTX 4090 24 GB: which LLMs can it run?
On its own, it loads up to DeepSeek R1 Distill 32B in Q4. With several cards, memory adds up and opens bigger models.
Specifications
Memory24 GB
Bandwidth1,008 GB/s
FP16 compute~165 TF
Power450 W
Indicative price€2,000
Cloud rental€0.45/h
Against cards at the same price
Models and expected speed
| Model | Q4 | Q8 | FP16 |
|---|---|---|---|
| Qwen3 4B | ✓ ~217 tok/s | ✓ ~126 tok/s | ✓ ~68 tok/s |
| Llama 3.1 8B | ✓ ~112 tok/s | ✓ ~64 tok/s | ✓ ~34 tok/s |
| Qwen3 8B | ✓ ~109 tok/s | ✓ ~63 tok/s | ✓ ~33 tok/s |
| Gemma 3 12B | ✓ ~75 tok/s | ✓ ~43 tok/s | 2 × · ~34 tok/s |
| Mistral Nemo 12B | ✓ ~74 tok/s | ✓ ~42 tok/s | 2 × · ~34 tok/s |
| Phi-4 14B | ✓ ~62 tok/s | ✓ ~35 tok/s | 2 × · ~28 tok/s |
| Qwen3 14B | ✓ ~61 tok/s | ✓ ~35 tok/s | 2 × · ~28 tok/s |
| gpt-oss-20b | ✓ ~240 tok/s | 2 × · ~210 tok/s | 2 × · ~113 tok/s |
| Mistral Small 24B | ✓ ~38 tok/s | 2 × · ~32 tok/s | 3 × · ~26 tok/s |
| Gemma 3 27B | ✓ ~34 tok/s | 2 × · ~29 tok/s | 3 × · ~23 tok/s |
| Qwen3.8 27B | ✓ ~33 tok/s | 2 × · ~28 tok/s | 3 × · ~22 tok/s |
| Qwen3 30B-A3B | ✓ ~260 tok/s | 2 × · ~228 tok/s | 3 × · ~185 tok/s |
| Qwen3 32B | ✓ ~28 tok/s | 2 × · ~24 tok/s | 4 × · ~25 tok/s |
| Qwen2.5 Coder 32B | ✓ ~28 tok/s | 2 × · ~24 tok/s | 4 × · ~25 tok/s |
| DeepSeek R1 Distill 32B | ✓ ~28 tok/s | 2 × · ~24 tok/s | 4 × · ~25 tok/s |
| Mixtral 8x7B | 2 × · ~105 tok/s | 3 × · ~90 tok/s | ✗ |
| Llama 3.3 70B | 3 × · ~29 tok/s | 4 × · ~22 tok/s | ✗ |
| GLM-4.5-Air 106B | 3 × · ~170 tok/s | ✗ | ✗ |
| gpt-oss-120b | 4 × · ~518 tok/s | ✗ | ✗ |
| Mistral Large 123B | 4 × · ~22 tok/s | ✗ | ✗ |
| Qwen3 235B-A22B | ✗ | ✗ | ✗ |
| GLM-4.5 355B | ✗ | ✗ | ✗ |
| Llama 3.1 405B | ✗ | ✗ | ✗ |
| DeepSeek R1 671B | ✗ | ✗ | ✗ |
| DeepSeek V3.1 671B | ✗ | ✗ | ✗ |
| Kimi K2 1T | ✗ | ✗ | ✗ |
8k tokens, one user. “2 ×” = number of cards needed, speed with tensor parallelism (vLLM). ✗ = doesn’t fit, even with 4 cards.