Natural Language Processing (Almost) from Scratch

A single neural architecture and learning algorithm for several NLP tasks (POS tagging, chunking, NER, SRL) that avoids task-specific feature engineering and instead learns internal representations, largely from vast unlabeled data.

Most state-of-the-art systems applied linear models to hand-engineered, task-specific features, often built from outputs of other systems. Optimizing for one benchmark teaches little about the broader goals of language understanding; Collobert et al. treat benchmarks as indirect measures of the quality of learned intermediate representations.

Some a priori NLP knowledge remained: lowercasing words, a capitalization feature, replacing digit sequences, and tokenization. A truly from-scratch system would learn from characters (or speech).

POS tagging (WSJ, per-word accuracy), chunking (CoNLL 2000, F1), named entity recognition (CoNLL 2003, F1) and semantic role labeling (CoNLL 2005, F1). The benchmark systems they compared with: 97.24%, 94.29, 89.31 and 77.92 respectively.

The harder the task (lower accuracies), the more engineered features the best systems used: POS needed few, while SRL systems relied on many parse-tree features (paths, voice, head words, subcategorization, multiple parse trees).

A first layer maps each word to a feature vector; a second layer extracts features from a window of words or the whole sentence (as a sequence, not a bag of words); then standard layers, trained end to end by backpropagation, output one score per tag.

Words are indices into a dictionary \(\mathcal{D}\); the layer maps word \(w\) to the \(w\)th column of a trainable matrix \(W \in \mathbb{R}^{d_{wrd} \times |\mathcal{D}|}\). Other discrete features (e.g. capitalization) get their own lookup tables, and their vectors are concatenated.

To tag a word, take a fixed window of \(d_{win}\) words around it (padding at sentence edges), concatenate their lookup vectors, then apply Linear → HardTanh → Linear, giving one score per tag. HardTanh clips its input to \([-1, 1]\), a cheaper tanh.

A word’s role depends on a chosen predicate, which may lie outside the window. The sentence approach looks at the whole sentence, adding to each word features for its relative distance to the predicate and to the word being tagged.

Convolutional (time-delay) layers apply the same linear map to every window, giving local features per position; a max over positions then keeps, for each feature, its strongest value anywhere in the sentence. An average would make little sense, since most words don’t affect a given word’s role.

IOBES for all tasks, the most expressive: B-X begins a segment, I-X is inside, E-X ends it, S-X is a single-word segment, O is outside. For chunking, a one-word NP is S-NP; longer NPs are B-NP … I-NP … E-NP. Output was converted back to IOB for evaluation.

Each word is classified independently: a softmax over the network’s tag scores gives \(p(i \mid x, \theta)\), and \(\log p(y \mid x, \theta) = [f_\theta]_y - \text{logadd}_j [f_\theta]_j\), where \(\text{logadd}_i z_i = \log \sum_i e^{z_i}\). It ignores dependencies between neighbouring tags.

A path of tags gets score \(s = \sum_t ([A]_{[i]_{t-1}, [i]_t} + [f_\theta]_{[i]_t, t})\), adding trained transition scores to network scores; the probability of the true path is its score normalized over all paths with logadd, computed in linear time by a forward recursion. Decoding uses Viterbi.

It maximizes the same likelihood as a linear-chain CRF, but with network scores instead of a linear model. The scores aren’t normalized per position, so they act like CRF potentials and avoid the label-bias problem; the non-linear network learns features for each task.

Behind the benchmarks: with SLL, POS 96.37 (vs 97.24), chunking 90.33 (94.29), NER 81.47 (89.31), SRL 70.99 (77.92). SLL beat WLL, mostly on chunking, NER and SRL.

Nearest neighbours in the embedding space weren’t semantically related. Most parameters are in the 100,000 × 50 lookup table, and most words occur too rarely in labeled data to be trained well.

A pairwise ranking criterion: \(\sum_{x} \sum_{w \in \mathcal{D}} \max\{0, 1 - f_\theta(x) + f_\theta(x^{(w)})\}\), where \(x^{(w)}\) is the text window \(x\) with its centre word replaced by \(w\). The network should score real windows at least 1 above corrupted ones. It avoids normalizing over the vocabulary, and unlike entropy is not dominated by frequent phrases: rare but legal phrases matter for learning syntax.

On Wikipedia (631M words) and then Wikipedia plus Reuters RCV1 (852M words), with window size 11 and 100 hidden units, using a curriculum of growing dictionaries (5k up to 100k words), each initialized from the previous network. LM1 took about four weeks, LM2 three more. Hyperparameters were chosen by “breeding”.

Without enough compute for grid search, train \(k\) parameter settings in parallel for some days, keep the best by validation ranking loss, and start the next round from slight variations of the winners, initialized with their parents’ embeddings so training continues online.

Nearest neighbours were syntactically and semantically related: France near Austria, Belgium, Germany; XBox near Amiga, PlayStation; reddish near greenish, bluish. Supervised-only embeddings had shown no such structure.

The lookup tables of the supervised networks were initialized with the language model embeddings and then fine-tuned. Results with SLL below. The larger language model helped more, suggesting even more unlabeled data would help.

task	benchmark	NN	NN + LM2
POS	97.24	96.37	97.20
chunking	94.29	90.33	93.63
NER	89.31	81.47	88.67
SRL	77.92	70.99	74.15

It resembles Harris’s (1968) operator grammars, which grade the acceptability of sentences with one word class varied and find sentence forms with the same gradings, revealing structure through acceptability comparisons rather than explicit hierarchy.

Networks for different tasks share the lookup tables and first hidden (or convolution) layer, with task-specific last layers; training alternates examples from each task’s own data set. Gains were marginal; the most important “multi-task” component was the unsupervised learning of word embeddings.

Joint decoding combines separately trained models at test time, so their extra dependency paths can’t involve unobserved variables. Shared hidden representations are unobserved, so they can only be learned by training the models together.

Moderate gains, reaching or passing the benchmarks: two-character suffixes for POS (97.29), the CoNLL gazetteer for NER (89.59), POS tags for chunking (94.32), chunk tags for SRL (74.72). Ensembles of ten networks gave small further gains.

Gazetteers contain multi-word entities; the network only had per-word representations. united and bicycle are probably non-entities alone, but United Bicycle may be one, which needs higher-level information.

Charniak parse levels were fed as IOBES segment tags in extra lookup tables: level 0 (leaf phrases) added almost 1.5 F1, further levels less, up to 76.06, close to the 77.92 benchmark that used six parse trees. A network predicting level-0 segments (PT0) worked as well as the parser and better than chunk tags.

In the non-linear network they did at least as well as Brown cluster features on all tasks, and fine-tuning them helped further. In a convex (linear, CRF-like) setup with fixed representations the results were mixed, and non-convex was usually better for both.

The standalone C implementation of the networks (about 2500 lines): under 150 MB of memory and under a millisecond per word for all tags. With a few features it matched or beat benchmarks (POS 97.29, chunking 94.32, NER 89.59; SRL 75.49), and ran far faster with much less memory than existing taggers, e.g. SRL in 51 s vs 6253 s.

No single NLP task covers the goals of NLP, so engineering that doesn’t generalize across tasks is undesirable, though they acknowledge their debt to task-specific research. They chose neural networks for their ability to discover hidden representations with stochastic learning that scales linearly with data.