Are Emergent Abilities of Large Language Models a Mirage?

2 Alternative Explanation for Emergent Abilities

1. In Schaeffer et al.’s toy model, why does exact-match accuracy look emergent when per-token loss falls smoothly?
   If per-token cross-entropy falls as a power law in model size \(N\), the probability of a correct token \(p_N\) rises smoothly. Requiring all \(L\) tokens correct gives accuracy \(\approx p_N^{\,L}\), which stays near zero and then rises steeply on a log-scale plot, more so for longer targets.

   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
2. How does the picture change if a linear metric like token edit distance is used?
   Token edit distance is about \(L(1 - p_N)\), linear in the per-token error, so the same models show smooth, continuous, predictable improvement with scale: the “emergence” disappears.
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
3. What three factors can fully explain sharp, unpredictable jumps with scale, according to Schaeffer et al.?
   (1) A metric that nonlinearly or discontinuously scales the per-token error rate; (2) insufficient resolution to estimate small models’ performance (resolution is set by 1/test-set size); (3) too few model sizes sampled in the large-parameter regime.
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2