Mixtral of Experts
Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088)
Subscriber deck — free during early access
2 cards in “5 Routing analysis” · whole deck
5 Routing analysis
- Do Mixtral’s experts specialize in topics like biology or philosophy?No clear topical specialization: expert usage was very similar for arXiv, PubMed abstracts and philosophy texts (DM Mathematics differed slightly). Routing looked more syntactic, e.g. Python’s “self” and indentation tokens went consistently to the same experts.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §5
- What is temporal locality in Mixtral’s routing, and why does it matter?Consecutive tokens are assigned the same expert far more often than chance in middle and late layers. This can overload particular experts under expert parallelism, but can also be exploited for caching.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §5
Source
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William El Sayed. 2024. Mixtral of Experts. In arXiv preprint arXiv:2401.04088. https://arxiv.org/abs/2401.04088