Natural Language Processing (Almost) from Scratch

4.2 Ranking Criterion versus Entropy Criterion

1. What language-model training criterion did Collobert et al. use instead of cross-entropy, and why?
   A pairwise ranking criterion: \(\sum_{x} \sum_{w \in \mathcal{D}} \max\{0, 1 - f_\theta(x) + f_\theta(x^{(w)})\}\), where \(x^{(w)}\) is the text window \(x\) with its centre word replaced by \(w\). The network should score real windows at least 1 above corrupted ones. It avoids normalizing over the vocabulary, and unlike entropy is not dominated by frequent phrases: rare but legal phrases matter for learning syntax.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), §4.2, pp. 2511–2512