On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

4.1 Size Doesn’t Guarantee Diversity

1. Why doesn’t a huge web corpus represent humanity’s views, according to Bender et al.?
   Internet access is uneven (younger users, developed countries); particular sources skew further (Reddit users mostly young men; few Wikipedians are women); moderation makes some platforms less welcoming to marginalized people; and crawling favours well-linked sites, so at each step hegemonic viewpoints are retained.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §4.1, pp. 613–614
2. How can data filtering further silence marginalized voices?
   C4 drops any page containing a word from a list of about 400 “bad words”, mostly sexual terms, which removes pornography but also suppresses LGBTQ community discourse that uses or reclaims such words; GPT-3’s similarity filter towards Reddit-linked text removed unknown content beyond the “unintelligible”.
   • Bender et al., 2021, On the Dangers of Stochastic Parrots (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21)), §4.1, pp. 613–614