Natural Language Processing (Almost) from Scratch

2. The Benchmark Tasks

1. Which four benchmark tasks did Collobert et al. use?
   POS tagging (WSJ, per-word accuracy), chunking (CoNLL 2000, F1), named entity recognition (CoNLL 2003, F1) and semantic role labeling (CoNLL 2005, F1). The benchmark systems they compared with: 97.24%, 94.29, 89.31 and 77.92 respectively.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 2. The Benchmark Tasks, pp. 2494–2498
2. What pattern did Collobert et al. observe between task difficulty and feature engineering?
   The harder the task (lower accuracies), the more engineered features the best systems used: POS needed few, while SRL systems relied on many parse-tree features (paths, voice, head words, subcategorization, multiple parse trees).
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 2. The Benchmark Tasks, pp. 2494–2498