Are Emergent Abilities of Large Language Models a Mirage?

4 Meta-Analysis of Claimed Emergent Abilities

1. What did the meta-analysis of BIG-Bench show about where emergence appears?
   Emergence is tied to metrics, not to task–model pairs: of 39 BIG-Bench metrics at most 5 showed it; human-annotated emergent abilities appeared under only 4 metrics. Since BIG-Bench often scores a task with several metrics, the absence under the others is telling.
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
2. What happened to LaMDA’s emergent abilities under a continuous metric?
   Tasks where LaMDA looked emergent under the discontinuous Multiple Choice Grade showed smooth improvement under the continuous Brier score (squared error of the probability given to the right answer).
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4