Are Emergent Abilities of Large Language Models a Mirage?
Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “5 Inducing Emergent Abilities in Networks on Vision Tasks” · whole deck
5 Inducing Emergent Abilities in Networks on Vision Tasks
- How did Schaeffer et al. create “emergent abilities” in vision models?By choosing sharp metrics: shallow autoencoders on CIFAR-100 improved reconstruction error smoothly, but a metric counting images reconstructed below an error cutoff jumped abruptly; Transformers classifying Omniglot characters looked emergent when scored as getting all images in a sequence right.
- Schaeffer et al., 2023, Are Emergent Abilities of Large Language Models a Mirage? (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
Source
- Rylan Schaeffer, Brando Miranda and Sanmi Koyejo. 2023. Are Emergent Abilities of Large Language Models a Mirage?. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/adc98a266f45005c403b8311ca7e8bd7-Abstract-Conference.html