Natural Language Processing (Almost) from Scratch

1. Introduction

1. What is the goal of “NLP (Almost) from Scratch”?
   A single neural architecture and learning algorithm for several NLP tasks (POS tagging, chunking, NER, SRL) that avoids task-specific feature engineering and instead learns internal representations, largely from vast unlabeled data.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 1. Introduction, pp. 2493–2494
2. What criticism of the prevailing NLP methodology motivated Collobert et al.?
   Most state-of-the-art systems applied linear models to hand-engineered, task-specific features, often built from outputs of other systems. Optimizing for one benchmark teaches little about the broader goals of language understanding; Collobert et al. treat benchmarks as indirect measures of the quality of learned intermediate representations.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 1. Introduction, pp. 2493–2494
3. Why “almost” from scratch?
   Some a priori NLP knowledge remained: lowercasing words, a capitalization feature, replacing digit sequences, and tokenization. A truly from-scratch system would learn from characters (or speech).
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 1. Introduction, pp. 2493–2494