Mixtral of Experts
Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088)
Subscriber deck — free during early access
3 cards in “1 Introduction” · whole deck
1 Introduction
- What is Mixtral 8x7B?An open-weights (Apache 2.0) sparse mixture-of-experts language model with Mistral 7B’s architecture, except that each layer’s feed-forward block is replaced by 8 experts, of which a router picks 2 per token; trained with a 32K-token context.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §1
- What does “47B total, 13B active parameters” mean for Mixtral?Each token can be routed through any experts, so the model holds 47B parameters, but a given token passes through only 2 of 8 feed-forward blocks per layer, using about 13B: inference compute is that of a 13B model.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §1
- How did Mixtral compare with Llama 2 70B and GPT-3.5?It matched or beat both on almost all benchmarks, with about 5× fewer active parameters than Llama 2 70B, and was far stronger in mathematics, code generation and multilingual tasks.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §1
Source
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William El Sayed. 2024. Mixtral of Experts. In arXiv preprint arXiv:2401.04088. https://arxiv.org/abs/2401.04088