Logistic Regression and Text Classification

Historical notes

1. Where did logistic regression originate, and how did it enter NLP?
   In statistics (by the 1960s, especially medicine); in linguistics from the late 1970s for studying linguistic variation. NLP adopted it in the 1990s from information retrieval and speech, and at IBM under the name maximum entropy (maxent) modeling.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, Historical notes, p. 125
2. What is feature selection, and what metrics are used for it?
   Removing features unlikely to generalize, by ranking them by how informative they are about the class: information gain (bits of information a word’s presence gives about the class), χ², pointwise mutual information, GINI index.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, Historical notes, p. 125