Lexicons for Sentiment, Affect, and Connotation

23.7 Using lexicons for affect recognition

1. What is the standard approach to detecting emotion and other affective states in text?
   Supervised classification: with large, similar training and test sets, all words or bigrams as features to logistic regression or an SVM is hard to beat; affect detection is treated as document classification.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.7, p. 534
2. How did Schwartz et al. select n-gram features from a huge corpus of social media posts?
   Keep only words and phrases used by at least 1% of subjects, and keep 2- and 3-grams only with high PMI, \(\log\frac{p(phrase)}{\prod_{w \in phrase} p(w)} > 2 \cdot length\); counts were normalized by each subject’s total word use.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.7, p. 534
3. What values can a lexicon feature take?
   An indicator (1 if the text has any word from lexicon \(L\)), the count of tokens in \(L\), \(\sum_{w \in L} count(w)\), or a weighted count \(\sum_{w \in L} \theta^L_w count(w)\); counts may be logged or normalized per writer.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.7, p. 534
4. After training a lexicon-feature classifier, how do you see which lexicons matter?
   Inspect the feature weights, e.g. in logistic regression the weight indicates how associated the lexicon feature is with each class.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.7, p. 535