Mixtral 8x7B and Open Sparse Mixture of Experts

By Satwik · May 7, 2026

Mistral released Mixtral 8x7B in December 2023, and it defined much of the open-model conversation into 2024. It is a sparse mixture-of-experts model: eight expert feed-forward blocks per layer, with a router selecting two per token. That gives it roughly 47 billion total parameters but only about 13 billion active per token, so it runs at the cost of a much smaller model while matching or beating far larger dense models like Llama 2 70B on many benchmarks.

Why it mattered: Mixtral demonstrated that sparse MoE, long a proprietary technique, could be shipped as open weights under a permissive Apache 2.0 license. It made high-quality inference cheaper and put a strong multilingual, code-capable model in the hands of anyone with modest hardware. The efficiency argument, more capability per active FLOP, became a template that later open models and DeepSeek pushed further.

The security and governance angle is the open-weights dilemma in miniature. Permissively licensed weights cannot be recalled, guardrails can be fine-tuned away, and safety mitigations are optional for downstream users. That same openness is what enables independent research, on-premises deployment for sensitive data, and freedom from vendor lock-in, all of which have real security value. Mixtral sits squarely in this tradeoff.

For practitioners, Mixtral was also a reminder that architecture matters as much as scale. A well-designed sparse model can undercut a larger dense one on cost while meeting its quality, which reshapes deployment economics for anyone self-hosting. It remains a reference point for what open, efficient inference looks like.