Mixtral of Experts
Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088)
Subscriber deck — free during early access
12 cards
1 Introduction
- What is Mixtral 8x7B?An open-weights (Apache 2.0) sparse mixture-of-experts language model with Mistral 7B’s architecture, except that each layer’s feed-forward block is replaced by 8 experts, of which a router picks 2 per token; trained with a 32K-token context.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §1
- What does “47B total, 13B active parameters” mean for Mixtral?Each token can be routed through any experts, so the model holds 47B parameters, but a given token passes through only 2 of 8 feed-forward blocks per layer, using about 13B: inference compute is that of a 13B model.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §1
- How did Mixtral compare with Llama 2 70B and GPT-3.5?It matched or beat both on almost all benchmarks, with about 5× fewer active parameters than Llama 2 70B, and was far stronger in mathematics, code generation and multilingual tasks.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §1
2.1 Sparse Mixture of Experts
- How does Mixtral’s MoE layer compute its output?\(y = \sum_{i=0}^{n-1} \mathrm{Softmax}(\mathrm{Top2}(x \cdot W_g))_i \cdot \mathrm{SwiGLU}_i(x)\): a linear router scores the 8 experts, keeps the two highest logits (the others are set to \(-\infty\)), normalizes them with a softmax and sums the two experts’ outputs with those weights.
Each token uses 2 of 8 experts per layer: 47B parameters, 13B active (invented weights) - Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
- What do \(n\) and \(K\) control in a sparse MoE layer?\(K\), the number of experts per token, sets compute per token (active parameters); \(n\), the number of experts, sets total (sparse) parameters. Raising \(n\) with \(K\) fixed adds capacity at roughly constant compute.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
- What practical costs do sparse MoE models carry despite low active parameters?Memory scales with all 47B parameters, routing adds overhead, and loading several experts per device lowers hardware efficiency; MoE suits batched workloads. With expert parallelism, uneven routing can overload some GPUs.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
- How does Mixtral’s MoE design differ from GShard?Mixtral replaces every feed-forward block with an MoE layer (GShard every other one) and uses a simple top-2 softmax gate rather than GShard’s more elaborate rule for choosing the second expert.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
3.1 Multilingual benchmarks
- Why is Mixtral much stronger than Llama 2 70B in French, German, Spanish and Italian?Its pretraining substantially upsampled multilingual data compared with Mistral 7B, and the extra capacity of the experts let it do well in these languages while keeping high English accuracy.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §3.1
3.2 Long range performance
- How well does Mixtral use its 32K context?On passkey retrieval it found the inserted key 100% of the time regardless of context length or position, and its perplexity on proof-pile fell steadily as context grew.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §3.2
4 Instruction Fine-tuning
- How was Mixtral 8x7B – Instruct trained, and how good was it?With supervised fine-tuning on instructions followed by Direct Preference Optimization on paired feedback. It scored 8.30 on MT-Bench and an Arena Elo above GPT-3.5-Turbo, Claude-2.1, Gemini Pro and Llama-2-70B-chat: the best open-weights model as of December 2023.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §4
5 Routing analysis
- Do Mixtral’s experts specialize in topics like biology or philosophy?No clear topical specialization: expert usage was very similar for arXiv, PubMed abstracts and philosophy texts (DM Mathematics differed slightly). Routing looked more syntactic, e.g. Python’s “self” and indentation tokens went consistently to the same experts.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §5
- What is temporal locality in Mixtral’s routing, and why does it matter?Consecutive tokens are assigned the same expert far more often than chance in middle and late layers. This can overload particular experts under expert parallelism, but can also be exploited for caching.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §5
Source
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William El Sayed. 2024. Mixtral of Experts. In arXiv preprint arXiv:2401.04088. https://arxiv.org/abs/2401.04088