Mixtral of Experts
Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088)
Subscriber deck — free during early access
1 cards in “3.1 Multilingual benchmarks” · whole deck
3.1 Multilingual benchmarks
- Why is Mixtral much stronger than Llama 2 70B in French, German, Spanish and Italian?Its pretraining substantially upsampled multilingual data compared with Mistral 7B, and the extra capacity of the experts let it do well in these languages while keeping high English accuracy.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §3.1
Source
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William El Sayed. 2024. Mixtral of Experts. In arXiv preprint arXiv:2401.04088. https://arxiv.org/abs/2401.04088