Mixtral of Experts
Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088)
Subscriber deck — free during early access
1 cards in “4 Instruction Fine-tuning” · whole deck
4 Instruction Fine-tuning
- How was Mixtral 8x7B – Instruct trained, and how good was it?With supervised fine-tuning on instructions followed by Direct Preference Optimization on paired feedback. It scored 8.30 on MT-Bench and an Arena Elo above GPT-3.5-Turbo, Claude-2.1, Gemini Pro and Llama-2-70B-chat: the best open-weights model as of December 2023.
- Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §4
Source
- Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix and William El Sayed. 2024. Mixtral of Experts. In arXiv preprint arXiv:2401.04088. https://arxiv.org/abs/2401.04088