Thumbs up? Sentiment Classification using Machine Learning Techniques

6.2 Results

1. How did the three classifiers do with unigram features, compared with the baselines and with topic classification?
   All beat the human-word-list baselines (58%, 64%) and, by smaller margins, the 69% baseline. Unigram frequencies gave NB 78.7% and SVM 72.8%; unigram presence gave NB 81.0, MaxEnt 80.4 and SVM 82.9%. Topic classification typically reaches 90%+, even with more classes, suggesting sentiment is harder.

   features	NB	ME	SVM
   unigrams, frequency	78.7	–	72.8
   unigrams, presence	81.0	80.4	82.9
   unigrams + bigrams	80.6	80.8	82.7
   bigrams only	77.3	77.4	77.1
   adjectives only	77.0	77.7	75.1

   • Pang et al., 2002, Thumbs up? Sentiment Classification using Machine Learning Techniques (Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)), §6.2, pp. 83–84
2. Feature presence vs frequency: what did Pang et al. find, and why is it surprising?
   Binary presence features beat counts, much more so for SVMs (82.9% vs 72.8%). This contradicts McCallum and Nigam’s (1998) finding for Naive Bayes topic classification; perhaps topics are signalled by repeated content words, but sentiment isn’t.
   • Pang et al., 2002, Thumbs up? Sentiment Classification using Machine Learning Techniques (Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)), §6.2, pp. 83–84
3. Did bigrams help sentiment classification in Pang et al.?
   No: adding bigrams to unigram presence didn’t improve accuracy (though it didn’t hurt much, even for NB), and bigrams alone were up to 5.8 points worse than unigrams. So bigrams didn’t capture the context the authors expected to matter.
   • Pang et al., 2002, Thumbs up? Sentiment Classification using Machine Learning Techniques (Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)), §6.2, pp. 83–84
4. What did part-of-speech tags and adjective-only features show?
   Appending POS tags (crude word sense disambiguation: love in “I love this movie” vs “a love story”) was a wash. The 2633 adjectives alone did worse than the 2633 most frequent unigrams, which nearly matched all 16165 unigrams, despite earlier work’s focus on adjectives.
   • Pang et al., 2002, Thumbs up? Sentiment Classification using Machine Learning Techniques (Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)), §6.2, pp. 83–84
5. Did word position in a review help?
   Tagging words as in the first quarter, last quarter or middle half (reviews often open and close with the verdict) gave results close to plain unigrams; more refined notions of position might help.
   • Pang et al., 2002, Thumbs up? Sentiment Classification using Machine Learning Techniques (Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)), §6.2, pp. 83–84