On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

3 Environmental and Financial Cost

1. What environmental costs of large LMs did Bender et al. highlight?
   Strubell et al. estimated that training a Transformer with neural architecture search emitted about 284 t CO₂ (an average person emits about 5 t a year), and that a 0.1 BLEU gain from NAS for English–German cost about $150,000 in compute; training BERT-base once used about as much energy as a trans-American flight.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §3, pp. 612–613
2. Why do Bender et al. say the costs of large LMs are unfairly distributed?
   The negative effects of climate change hit the most marginalized communities first, while the benefits of English-centred LMs and devices like smart speakers mostly go to the already privileged; e.g. people in the Maldives or flood-hit Sudan pay environmental costs of models not built for Dhivehi or Sudanese Arabic.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §3, pp. 612–613
3. What practices did Bender et al. recommend for environmental and financial costs?
   Report training time, energy and carbon, and sensitivity to hyperparameters; evaluate efficiency, not only accuracy (green AI); run experiments in carbon-friendly regions; consider inference costs over a model’s lifetime; and weigh costs before starting a project.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §3, pp. 612–613