Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

A system trained only on linguistic form has a priori no way to learn meaning, so claims that large language models “understand” language or capture “meaning” are overclaims; clarity about form versus meaning will guide the field towards better science of natural language understanding.

Any system trained only on string prediction, over characters, words or sentences, sequentially or not.

Terms like “understands sentence relationships”, “machine comprehension” or “recall factual knowledge” are gross overclaims if read as human-analogous abilities, and otherwise should be defined as technical terms; imprudent terminology feeds AI hype in the popular press.

Niven and Kao (2019): BERT’s strong argument reasoning results fall to chance once adversarial examples negate cues so label-specific lexical cues are balanced. McCoy et al. (2019): BERT’s MNLI success relies on overlap heuristics, and on data built to defeat them it falls well below chance.

Form is any observable realization of language (marks on a page, bytes, articulator movements). Meaning is the relation \(M \subseteq E \times I\) between expressions \(e\) and the communicative intents \(i\) they can be used to evoke; understanding is retrieving \(i\) from \(e\).

Standing meaning is what an expression means constantly across contexts: a linguistic system provides \(C \subseteq E \times S\), with meanings that have interpretations (e.g. can be tested for truth against a model of the world). A speaker picks \(e\) whose standing meaning fits their intent; the listener reconstructs \(s\) and uses context and guesses about the speaker to infer \(i\).

Listeners must use context and hypotheses about the speaker’s mind to infer intent: in “The photographer asked me to do a Napoleon for the camera”, Napoleon means a particular pose; in “Never ask two China trips to the same party”, China trip means a person who recently visited China.

The Chinese Room shows form manipulation can look like understanding from outside; Harnad notes no one can learn Chinese word meanings from a Chinese dictionary alone. Learning a language requires grounding symbols, and modern LMs are trained (not just run) on form alone.

Two English speakers, A and B, stranded on islands, chat by underwater telegraph. A hyper-intelligent octopus O taps the cable, learns the statistics of their messages, then cuts B off and impersonates B. Can O fool A, having seen only form?

Social small talk, which needs only internally coherent text, may fool A. With A’s coconut catapult instructions, O can only reply generically (“Cool idea, great job!”), and A does the work of attributing meaning. When A urgently asks how to fight a bear with sticks, O fails: that needs mapping words to real-world entities.

It isn’t that O’s utterances make sense, but that A can make sense of them: A assumes the sender has communicative intents (as English speakers usually do) and attributes intent from conventional meaning and her model of B.

Train any LM on all well-formed Java on GitHub, with no compiler, bytecode or input–output examples, then ask it to execute a program. Or train it on English text and, separately, unlabelled photos, then ask “How many dogs in the picture are jumping?”. Both tests are absurd because the training data lacks what meaning requires: execution or reference.

No: children don’t learn languages from passive exposure such as TV or radio (e.g. infants learned Mandarin phoneme contrasts from live interaction but not from recordings). Learning depends on interaction and joint attention, where child and caregiver attend to the same thing and both know it.

Not distribution in a text corpus, but language used in the real world to convey communicative intents to real people. Grounding distributional models needs more than text: perceptual data or interaction data with success signals.

Bottom-up, NLP is climbing fast with steady state-of-the-art gains; top-down, the question is whether this hill leads to human-analogous NLU. Past paradigms (grammar-based, statistical) also felt like constant progress before their shortcomings showed.

Cultivate humility towards language and ask top-down questions; be aware of task limitations (artificial or crowdsourced data sets; superhuman scores suggest artifacts); value careful creation of new tasks; evaluate meaning across tasks; and thoroughly analyse both errors and successes.

Not directly: such data sets specify semantic relations between texts, a training signal beyond pure form, so a model might learn some aspects of meaning from them, though McCoy et al. and Niven and Kao warn that gains often come from data idiosyncrasies.

Quantity doesn’t matter to the principle: the octopus may watch forever. People keep generating new intents about evolving worlds, so producing apt responses without meaning would require memorizing infinitely many stimulus–response pairs.

It has probably learned something about meaning, as syntax and semantic similarity capture something about it: a useful but incomplete reflection of meaning in form. LMs learn that reflection, not meaning itself.