Are Emergent Abilities of Large Language Models a Mirage?

8 Discussion

1. What practical lessons for benchmarks do Schaeffer et al. draw?
   Task and metric are separate, meaningful choices; continuous or linear metrics are better for forecasting scaling, while nonlinear ones need much more test data; claims need controls for multiple comparisons (BIG-Bench has about \(10^6\) task–metric–model triples); and models and outputs should be open to independent study.
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §8