Mistral · Mixture of Experts · 2023-12

Mixtral 8x7B: how much VRAM?

In Q4_K_M with 8k tokens of context, Mixtral 8x7B needs about 31 GB of memory. The cheapest option: 2 × RTX 4060 Ti 16 GB (€1,800 for the complete machine).

Model sheet

Parameters47 B

Active / token13 B

Cache / 1k tok.128 MB

ollama run mixtral:8x7b

Weights on Hugging Face: mistralai/Mixtral-8x7B-Instruct-v0.1

Configure the cheapest server →Compare GPUs

VRAM by precision and context

Precision	4k	8k	32k	128k
FP16	99 GB	99 GB	102 GB	114 GB
FP8	50 GB	50 GB	53 GB	65 GB
Q8_0	53 GB	53 GB	56 GB	68 GB
Q6_K	41 GB	42 GB	45 GB	57 GB
Q4_K_M	30 GB	31 GB	34 GB	46 GB
Q3_K_M	25 GB	25 GB	28 GB	40 GB

One user. Each simultaneous user adds their own context cache.

Frequently asked questions

How much VRAM does Mixtral 8x7B need?

With an 8k-token context and one user, Mixtral 8x7B needs about 31 GB in Q4, 53 GB in Q8 and 99 GB in FP16.

What is the cheapest hardware to run Mixtral 8x7B?

In Q4, the cheapest option in our catalogue to run Mixtral 8x7B is 2 × RTX 4060 Ti 16 GB, about €1,800 for the complete machine.

How fast does Mixtral 8x7B run?

Expect around 30 tokens per second for one user on 2 × RTX 4060 Ti 16 GB in Q4 (estimate based on memory bandwidth).

On which hardware?

Hardware	Cards	Speed	Machine
RTX 4060 Ti 16 GB	2 cards	~30 tok/s	€1,800
Radeon RX 7800 XT 16 GB	2 cards	~65 tok/s	€1,900
Mini-PC Ryzen AI Max+ 395 128 GB	1 is enough	~18 tok/s	€2,300
RTX 3090 24 GB (used)	2 cards	~98 tok/s	€2,400
RTX 4070 Ti SUPER 16 GB	2 cards	~70 tok/s	€2,600
Radeon RX 7900 XTX 24 GB	2 cards	~100 tok/s	€2,800
RTX 4080 SUPER 16 GB	2 cards	~77 tok/s	€3,000
RTX 5080 16 GB	2 cards	~100 tok/s	€3,300
RTX 5090 32 GB	1 is enough	~125 tok/s	€3,500
RTX 3060 12 GB	3 cards	~56 tok/s	€3,640
NVIDIA DGX Spark 128 GB	1 is enough	~19 tok/s	€4,000
Mac Studio M4 Max 128 GB	1 is enough	~38 tok/s	€4,400
RTX 4070 12 GB	3 cards	~79 tok/s	€4,450
RTX 4090 24 GB	2 cards	~105 tok/s	€4,900
RTX 6000 Ada 48 GB	1 is enough	~67 tok/s	€7,900
RTX PRO 6000 Blackwell 96 GB	1 is enough	~125 tok/s	€9,900
Mac Studio M3 Ultra 512 GB	1 is enough	~57 tok/s	€11,500
L40S 48 GB	1 is enough	~60 tok/s	€13,500
A100 80 GB	1 is enough	~142 tok/s	€17,000
H100 80 GB	1 is enough	~234 tok/s	€34,000
H200 141 GB	1 is enough	~335 tok/s	€39,000
B200 192 GB	1 is enough	~558 tok/s	€48,000

Estimated complete machine: cards + host PC (€900 for 1–2 consumer cards, €2,800 beyond, a €6,000 server for datacenter cards). Macs and mini-PCs are sold complete.