Mistral · Dense model · 2025-01

Mistral Small 24B: how much VRAM?

In Q4_K_M with 8k tokens of context, Mistral Small 24B needs about 17 GB of memory. The cheapest option: 2 × RTX 3060 12 GB (€1,460 for the complete machine).

Model sheet

Parameters24 B

Active / token24 B

Cache / 1k tok.164 MB

ollama run mistral-small:24b

Weights on Hugging Face: mistralai/Mistral-Small-24B-Instruct-2501

Configure the cheapest server →Compare GPUs

VRAM by precision and context

Precision	4k	8k	32k	128k
FP16	51 GB	52 GB	56 GB	71 GB
FP8	26 GB	27 GB	31 GB	46 GB
Q8_0	28 GB	28 GB	32 GB	48 GB
Q6_K	22 GB	22 GB	26 GB	42 GB
Q4_K_M	16 GB	17 GB	21 GB	36 GB
Q3_K_M	13 GB	14 GB	18 GB	33 GB

One user. Each simultaneous user adds their own context cache.

Frequently asked questions

How much VRAM does Mistral Small 24B need?

With an 8k-token context and one user, Mistral Small 24B needs about 17 GB in Q4, 28 GB in Q8 and 52 GB in FP16.

What is the cheapest hardware to run Mistral Small 24B?

In Q4, the cheapest option in our catalogue to run Mistral Small 24B is 2 × RTX 3060 12 GB, about €1,460 for the complete machine.

How fast does Mistral Small 24B run?

Expect around 20 tokens per second for one user on 2 × RTX 3060 12 GB in Q4 (estimate based on memory bandwidth).

On which hardware?

Hardware	Cards	Speed	Machine
RTX 3060 12 GB	2 cards	~20 tok/s	€1,460
RTX 3090 24 GB (used)	1 is enough	~35 tok/s	€1,650
RTX 4060 Ti 16 GB	2 cards	~16 tok/s	€1,800
Radeon RX 7900 XTX 24 GB	1 is enough	~36 tok/s	€1,850
Radeon RX 7800 XT 16 GB	2 cards	~35 tok/s	€1,900
RTX 4070 12 GB	2 cards	~29 tok/s	€2,000
Mini-PC Ryzen AI Max+ 395 128 GB	1 is enough	~9.7 tok/s	€2,300
RTX 4070 Ti SUPER 16 GB	2 cards	~38 tok/s	€2,600
RTX 4090 24 GB	1 is enough	~38 tok/s	€2,900
RTX 4080 SUPER 16 GB	2 cards	~42 tok/s	€3,000
RTX 5080 16 GB	2 cards	~54 tok/s	€3,300
RTX 5090 32 GB	1 is enough	~68 tok/s	€3,500
NVIDIA DGX Spark 128 GB	1 is enough	~10 tok/s	€4,000
Mac Studio M4 Max 128 GB	1 is enough	~21 tok/s	€4,400
RTX 6000 Ada 48 GB	1 is enough	~36 tok/s	€7,900
RTX PRO 6000 Blackwell 96 GB	1 is enough	~68 tok/s	€9,900
Mac Studio M3 Ultra 512 GB	1 is enough	~31 tok/s	€11,500
L40S 48 GB	1 is enough	~33 tok/s	€13,500
A100 80 GB	1 is enough	~77 tok/s	€17,000
H100 80 GB	1 is enough	~127 tok/s	€34,000
H200 141 GB	1 is enough	~181 tok/s	€39,000
B200 192 GB	1 is enough	~302 tok/s	€48,000

Estimated complete machine: cards + host PC (€900 for 1–2 consumer cards, €2,800 beyond, a €6,000 server for datacenter cards). Macs and mini-PCs are sold complete.