On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

How big is too big for language models, what risks come with ever larger LMs, and how can they be mitigated? It recommends weighing environmental and financial costs first, curating and documenting data sets instead of ingesting the web, pre-development exercises on how a project fits research goals and stakeholder values, and research beyond ever larger LMs.

Environmental and financial costs (§3); unfathomable training data that encodes hegemonic views and biases (§4); misdirected research effort, mistaking form manipulation for understanding (§5); and harms from people taking seemingly coherent synthetic text as meaningful (§6).

From large n-gram LMs to pretrained word vectors to pretrained Transformer LMs, the models and their data grew by orders of magnitude (BERT 340M parameters, GPT-3 175B, Switch-C 1.6T), and their uses grew from rescoring outputs to apparently meaning-manipulating tasks, bringing new kinds of risk.

No: models like mBERT perform unevenly across languages, and over 90% of the world’s languages, used by more than a billion people, still have little or no language technology support.

Strubell et al. estimated that training a Transformer with neural architecture search emitted about 284 t CO₂ (an average person emits about 5 t a year), and that a 0.1 BLEU gain from NAS for English–German cost about $150,000 in compute; training BERT-base once used about as much energy as a trans-American flight.

The negative effects of climate change hit the most marginalized communities first, while the benefits of English-centred LMs and devices like smart speakers mostly go to the already privileged; e.g. people in the Maldives or flood-hit Sudan pay environmental costs of models not built for Dhivehi or Sudanese Arabic.

Report training time, energy and carbon, and sensitivity to hyperparameters; evaluate efficiency, not only accuracy (green AI); run experiments in carbon-friendly regions; consider inference costs over a model’s lifetime; and weigh costs before starting a project.

Internet access is uneven (younger users, developed countries); particular sources skew further (Reddit users mostly young men; few Wikipedians are women); moderation makes some platforms less welcoming to marginalized people; and crawling favours well-linked sites, so at each step hegemonic viewpoints are retained.

C4 drops any page containing a word from a list of about 400 “bad words”, mostly sexual terms, which removes pornography but also suppresses LGBTQ community discourse that uses or reclaims such words; GPT-3’s similarity filter towards Reddit-linked text removed unknown content beyond the “unintelligible”.

Social movements create new language and reframe narratives (e.g. Black Lives Matter influenced Wikipedia coverage of police shootings), but LMs trained on static data reify older, less inclusive understandings, and retraining huge models often enough to keep up is infeasible.

Audits rely on automated measures (sentiment, toxicity) that are themselves biased (e.g. toxicity scores rise with identity terms); they need a priori knowledge of which social categories matter, which is culture-bound; and certifying “safety” requires engaging with the systems of power behind harms, which is inherently political.

Building data sets so large and undocumented that they can’t be documented post hoc, so harms persist without recourse. Budget for curation and documentation from the start of a project and collect only as much data as can be thoroughly documented.

Research effort flows into beating NLU benchmarks with LMs that manipulate form well enough to pass tests without understanding (shown by removing spurious cues), an opportunity cost against meaning-capturing approaches and against technology built from carefully curated data for more languages.

Human communication is jointly constructed: listeners model the speaker’s beliefs, intentions and shared ground. LM text has no communicative intent, world model or model of the reader; the coherence we perceive comes from our own linguistic competence and predisposition to find meaning.

Bender et al.’s description of an LM: a system that haphazardly stitches together sequences of linguistic forms observed in its training data, according to probabilistic information about how they combine, but without any reference to meaning.

Generated text can spread and amplify stereotypes and abusive language, causing psychological harm to people targeted (microaggressions, stereotype threat), reinforcing others’ prejudices and discrimination, and feeding biased text into future training data; inside classifiers or search, biases cause allocational harms largely invisible to those affected.

Cheap on-demand text for extremist recruitment (GPT-3 imitating a conspiracy theorist), term-paper mills and bots; fluent but wrong machine translation taken as the author’s intent (a Palestinian man was arrested after “good morning” in Arabic was translated as “attack them”); and extraction of personal information memorized by larger models.

Synthetic text can enter conversations with no person or organization accountable for its truth, yet people interpret it as if someone meant it; accountability is part of how meaning is situated.

Plan before building: weigh environmental and financial costs; curate and document data with frameworks like datasheets and model cards; identify direct and indirect stakeholders with value sensitive design; run pre-mortems imagining failures; consider dual use (e.g. watermarking generated text); and evaluate how models work, not just leaderboard scores.

As a dual-use case: improved ASR helps Deaf and hard of hearing people, but large LMs aren’t the only path to better ASR (and can’t rely on terabytes of data for most languages); where LMs are critical, mitigate their harms as stochastic parrots while preserving their uses.

Work on synthetic human behaviour: applications that aim to believably mimic humans bring risk of extreme harm, so their downstream effects must be understood and modelled, with target tasks grounded in concrete use cases designed with affected communities.