Lexicons for Sentiment, Affect, and Connotation

23.3 Creating affect lexicons by human labeling

1. How is affect lexicon creation by human labeling usually done today?
   By crowdsourcing: break the task into small pieces and distribute them to many annotators.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.3, pp. 523–524
2. How did EmoLex make sure annotators judged the right word sense?
   Each item first asked a multiple-choice synonym question that primed the intended sense (e.g. which word is closest to startle: automobile, shake, honesty, entertain?), then asked for association with each of the 8 emotions.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.3, p. 524
3. How were EmoLex’s graded ratings turned into labels?
   Ratings on a not/weakly/moderately/strongly scale were filtered for outliers, the majority class taken (ties to the stronger), then mapped to binary: not/weak → 0, moderate/strong → 1.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.3, p. 524
4. What is best–worst scaling?
   Annotators get N items (usually 4) and choose the best (highest) and worst (lowest) on some property. A word’s score is the proportion of times chosen best minus the proportion chosen worst. Used for the NRC VAD lexicon.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.3, p. 524
5. What is split-half reliability?
   A measure of annotation agreement: split the annotations in half and correlate the scores computed from the two halves.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 23, §23.3, p. 525