On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

2 Background

1. How do Bender et al. characterize the trend in language models up to 2021?
   From large n-gram LMs to pretrained word vectors to pretrained Transformer LMs, the models and their data grew by orders of magnitude (BERT 340M parameters, GPT-3 175B, Switch-C 1.6T), and their uses grew from rescoring outputs to apparently meaning-manipulating tasks, bringing new kinds of risk.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §2, pp. 611–612
2. Do multilingual models solve the inclusion problem, according to Bender et al.?
   No: models like mBERT perform unevenly across languages, and over 90% of the world’s languages, used by more than a billion people, still have little or no language technology support.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §2, pp. 611–612