Are Emergent Abilities of Large Language Models a Mirage?

3 Analyzing InstructGPT/GPT-3’s Emergent Arithmetic Abilities

1. What happened to GPT-3’s “emergent” arithmetic abilities when the metric changed?
   On 2-digit multiplication and 4-digit addition, accuracy looked emergent for 4–5-digit targets, but scoring the same outputs with token edit distance showed smooth, predictable improvement with scale, degrading quasi-linearly as target length grew.
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
2. What did a larger test set reveal about small models’ accuracy on arithmetic?
   With more test data (higher resolution), every InstructGPT/GPT-3 model had above-chance accuracy, improving smoothly with scale and falling roughly geometrically with target length, as the toy model predicts; zero accuracy had been an artifact of small test sets.
   • Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3