Mixtral of Experts

2.1 Sparse Mixture of Experts

1. How does Mixtral’s MoE layer compute its output?
   \(y = \sum_{i=0}^{n-1} \mathrm{Softmax}(\mathrm{Top2}(x \cdot W_g))_i \cdot \mathrm{SwiGLU}_i(x)\): a linear router scores the 8 experts, keeps the two highest logits (the others are set to \(-\infty\)), normalizes them with a softmax and sums the two experts’ outputs with those weights.

   • Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
2. What do \(n\) and \(K\) control in a sparse MoE layer?
   \(K\), the number of experts per token, sets compute per token (active parameters); \(n\), the number of experts, sets total (sparse) parameters. Raising \(n\) with \(K\) fixed adds capacity at roughly constant compute.
   • Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
3. What practical costs do sparse MoE models carry despite low active parameters?
   Memory scales with all 47B parameters, routing adds overhead, and loading several experts per device lowers hardware efficiency; MoE suits batched workloads. With expert parallelism, uneven routing can overload some GPUs.
   • Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1
4. How does Mixtral’s MoE design differ from GShard?
   Mixtral replaces every feed-forward block with an MoE layer (GShard every other one) and uses a simple top-2 softmax gate rather than GShard’s more elaborate rule for choosing the second expert.
   • Jiang et al., 2024, Mixtral of Experts (arXiv preprint arXiv:2401.04088), §2.1