Description
Mixtral 8x7B introduces a Sparse Mixture of Experts (SMoE) language model built on the Mistral 7B architecture, but enhanced with eight feedforward “experts” per layer. A lightweight router activates just two of these experts per token during inference, giving each token access to a massive 47 billion parameters while only actively using 13 billion—striking an effective balance between capacity and efficiency (47B vs 13B). Trained with a context size of 32K tokens, Mixtral outperforms or matches the performance of Llama 2 70B and GPT-3.5 across benchmarks—especially excelling in math, code generation, and multilingual tasks. Its instruction-tuned variant, Mixtral 8x7B – Instruct, surpasses leading models like GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B-chat on human evaluation tests. Both models are released under an Apache 2.0 license and accompanied by open-source tooling integrations.
